Deployable Machine Learning on MIMIC-IV: Leakage-Safe Prediction and Calibration for Incident Diabetes

Abstract

We present a leakage-safe, deployment-oriented pipeline for predicting incident diabetes within 365 days of an index hospital admission, using the MIMIC-IV (Medical Information Mart for Intensive Care) hospital module. The process enforces temporal and patient-level guards, separates model-specific preprocessing, and evaluates models with a calibration-first lens: reliability diagrams and Brier score, alongside discrimination (ROC/PR) and decision-analytic utility via decision curves. Two standard baselines—logistic regression and random forest—are tuned with inner cross-validation and post hoc calibrated (Platt or isotonic) tools. On the held-out test set, calibrated logistic regression attains modest yet consistent performance (e.g., $\mathrm{AUROC}\approx 0.70$, $\mathrm{AUPRC}\approx 0.31$) and competitive net benefit across clinically plausible thresholds; calibrated random forest is lower. Global explanations (permutation importance; SHAP for logistic) yield clinically coherent reliance patterns (age, length of stay, sex, ethnicity). The primary contribution is procedural and implementation-oriented: a transparent, reproducible, leak-proof pipeline that surfaces calibrated risks, threshold-aware decision support, and auditability artifacts suitable for clinical review and site adaptation. We outline a practical path to external validation and lightweight site-specific recalibration, positioning the work as a reference implementation for reliable EHR risk stratification rather than an algorithmic innovation.

1. Introduction

Diabetes mellitus and cardiovascular disease (CVD) are leading contributors to morbidity and mortality and share overlapping pathophysiology and risk profiles [1,2]. Early identification of individuals at elevated risk—particularly at the time of hospital admission—can facilitate timely prevention and targeted follow-up. Electronic Health Records (EHRs) enable such risk stratification at scale. In particular, the MIMIC-IV hospital module provides structured demographics, diagnoses, procedures, laboratory tests, medications, microbiology, and administrative events suitable for building and evaluating prognostic models.

While prior work has explored increasingly complex architectures, clinical deployment often benefits from compact models that are transparent, calibration-aware, and easy to maintain. Moreover, downstream decisions consume probabilities at task-specific thresholds, making probability calibration and threshold-dependent utility analysis as important as raw discrimination. In this study we therefore focus on a leakage-safe, end-to-end pipeline for predicting incident diabetes within 365 days of an index admission, using two strong, interpretable baselines: logistic regression and random forest. The pipeline couples admission-time features with patient-level splitting, post hoc calibration (Platt and isotonic), decision curve analysis, and model explanations.

Our contributions are threefold. First, we develop and release a reproducible pipeline that enforces temporal and patient-level leakage guards, standardizes preprocessing, and attaches post hoc probability calibration and decision-analytic evaluation. Second, we benchmark calibrated logistic regression against a calibrated random forest on MIMIC-IV for the task of predicting incident diabetes at 365 days, reporting ROC/PR discrimination, reliability, and threshold-wise net benefit. Third, we provide transparent explanations via SHAP summaries for logistic regression and permutation importance analyses, highlighting clinically plausible risk factors.

Guided by the data and methods used here, we reformulate the research questions as follows.

RQ1:Within a leakage-safe pipeline using only information available at or before admission, which calibrated baseline—logistic regression or random forest—achieves superior discrimination for predicting incident diabetes at 365 days (AUROC/AUPRC)?

RQ2: To what extent does post hoc calibration improve probability quality and translate into higher net benefit across clinically plausible decision thresholds, as assessed by reliability diagrams and decision curve analysis?

RQ3: Which predictors most strongly influence risk estimates, and are these influences clinically coherent, as quantified by SHAP summaries for logistic regression and permutation importance for both models?

Beyond this introduction, Section 2 reviews relevant background. Section 3 describes the pipeline and presents the empirical results on MIMIC-IV. Section 4 interprets the findings and their clinical implications, and Section 5 presents the related work. Finally, Section 6 outlines limitations and future work.

2. Background

We formalize the leakage-safe prediction problem, discrimination and calibration metrics, decision-analytic evaluation, and the global explanation methods used in Section 3. Throughout, $⊮\{·\}$ denotes the indicator function (it returns 1 when the stated condition holds and 0 otherwise).

2.1. Leakage-Safe Problem Formulation

Let $\mathcal{U}$ be the set of patients. For each $u\in \mathcal{U}$, let ${\tau }_u$ denote the index (admission) time, and let ${\left\{{\mathbf{z}}_u\left(t\right)\right\}}_{t\le {\tau }_u}$ be the multivariate EHR process up to ${\tau }_u$ (events, codes, labs, medications). Fix a prediction horizon $H>0$ and define the incident outcome

$$y_u=⊮\left\{\mathrm{incident}\mathrm{diabetes}\mathrm{occurs}\mathrm{in}({\tau }_u,{\tau }_u+H]\right\}.$$

(1)

A feature map $\Phi$ is temporal-guarded if it ignores any information after ${\tau }_u$. Formally, for any two trajectories with identical histories up to ${\tau }_u$,

$${\left\{{\mathbf{z}}_u\left(t\right)\right\}}_{t\le {\tau }_u}={\left\{{\mathbf{z}}_u^{\prime }\left(t\right)\right\}}_{t\le {\tau }_u}⟹\Phi \left({\left\{{\mathbf{z}}_u\left(t\right)\right\}}_{t\le {\tau }_u}\right)=\Phi \left({\left\{{\mathbf{z}}_u^{\prime }\left(t\right)\right\}}_{t\le {\tau }_u}\right).$$

(2)

We write ${\mathbf{x}}_u=\Phi \left({\left\{{\mathbf{z}}_u\left(t\right)\right\}}_{t\le {\tau }_u}\right)\in {\mathbb{R}}^p$ for the feature vector.

Let ${\mathcal{S}}_{\mathrm{tr}},{\mathcal{S}}_{\mathrm{val}},{\mathcal{S}}_{\mathrm{te}}\subset \mathcal{U}$ be disjoint patient-level splits. Imputation $\mathcal{I}$ and preprocessing $\mathcal{P}$ operators are fit on training only,

$$(\widehat{\mathcal{I}},\widehat{\mathcal{P}})\in arg\underset{\mathcal{I},\mathcal{P}}{min}{\mathcal{L}}_{\mathrm{prep}}\left(\{({\mathbf{x}}_u,y_u):u\in {\mathcal{S}}_{\mathrm{tr}}\}\right),$$

(3)

and applied to validation/test without refitting,

$${\tilde{\mathbf{x}}}_u=\widehat{\mathcal{P}}\left(\widehat{\mathcal{I}}\left({\mathbf{x}}_u\right)\right),u\in {\mathcal{S}}_{\mathrm{val}}\cup {\mathcal{S}}_{\mathrm{te}}.$$

(4)

Equations (2)–(4) jointly prevent temporal leakage (using only $t\le {\tau }_u$) and split leakage (no cross-partition fitting).

2.2. Predictors and Discrimination

A probabilistic predictor is a measurable map $f:{\mathbb{R}}^p\to [0,1]$ that produces raw scores ${\widehat{s}}_u=f\left({\tilde{\mathbf{x}}}_u\right)$. When f natively outputs probabilities we also write ${\widehat{p}}_u={\widehat{s}}_u$. For a threshold $t\in [0,1]$ define the true- and false-positive rates

$$\mathrm{TPR}\left(t\right)=\mathbb{P}({\widehat{s}}_u\ge t\mid y_u=1),$$

(5)

$$\mathrm{FPR}\left(t\right)=\mathbb{P}({\widehat{s}}_u\ge t\mid y_u=0).$$

(6)

The ROC curve is $\left\{\right(\mathrm{FPR}\left(t\right),\mathrm{TPR}\left(t\right)):t\in [0,1\left]\right\}$ and its area is

$$\mathrm{AUROC}={\int }_0^1\mathrm{TPR}\left({\mathrm{FPR}}^{-1}\left(x\right)\right)dx.$$

(7)

Precision and recall at threshold t are

$$\mathrm{Precision}\left(t\right)=\mathbb{P}(y_u=1\mid {\widehat{s}}_u\ge t),$$

(8)

$$\mathrm{Recall}\left(t\right)=\mathrm{TPR}\left(t\right).$$

(9)

The PR curve is $\left\{\right(\mathrm{Recall}\left(t\right),\mathrm{Precision}\left(t\right)\left)\right\}$ with

$$\mathrm{AUPRC}={\int }_0^1\mathrm{Precision}\left({\mathrm{Recall}}^{-1}\left(r\right)\right)dr.$$

(10)

In practice, (7)–(10) are computed via trapezoidal integration (ROC) and average precision (PR).

2.3. Probability Calibration and Reliability

We use post hoc calibration maps $g:[0,1]\to [0,1]$ to transform raw scores into calibrated probabilities ${\tilde{p}}_u=g\left({\widehat{s}}_u\right)$. Perfect calibration requires

$$\mathbb{E}[y_u\mid {\tilde{p}}_u=p]=p\mathrm{for}\mathrm{all}p\in [0,1].$$

(11)

Probability quality is measured by the Brier score on a set $\mathcal{S}$,

$$\mathrm{Brier}\left(\mathcal{S}\right)={\displaystyle \frac{1}{\left|\mathcal{S}\right|}}\sum _{u\in \mathcal{S}}{({\tilde{p}}_u-y_u)}^2,$$

(12)

with the understanding that we may also report the same quantity for raw ${\widehat{p}}_u$ when f is probabilistic. We consider two calibrators fitted on the validation set:

$$g_{\mathrm{platt}}\left(s\right)=\sigma (as+b),\sigma \left(u\right)={\displaystyle \frac{1}{1+e^{-u}}},$$

(13)

where $(a,b)$ minimize the negative log-likelihood on ${({\widehat{s}}_u,y_u)}_{u\in {\mathcal{S}}_{\mathrm{val}}}$, and isotonic regression,

$$g_{\mathrm{iso}}\in arg\underset{g\in {\mathcal{G}}_{\uparrow }}{min}\sum _{u\in {\mathcal{S}}_{\mathrm{val}}}{\left(g\left({\widehat{s}}_u\right)-y_u\right)}^2,$$

(14)

where ${\mathcal{G}}_{\uparrow }$ is the set of nondecreasing step functions. We select

$$g^{★}\in \{g_{\mathrm{platt}},g_{\mathrm{iso}}\}\mathrm{such}\mathrm{that}\mathrm{Brier}\left(g^{★}\circ f;{\mathcal{S}}_{\mathrm{val}}\right)=\underset{g\in \{g_{\mathrm{platt}},g_{\mathrm{iso}}\}}{min}\mathrm{Brier}\left(g\circ f;{\mathcal{S}}_{\mathrm{val}}\right),$$

(15)

and apply ${\tilde{p}}_u=g^{★}\left({\widehat{s}}_u\right)$ to test. Reliability diagrams visualize the empirical calibration function

$$\widehat{r}\left(p\right)=\mathbb{E}\left[y_u|q_u\in \mathrm{bin}\left(p\right)\right],$$

(16)

where $q_u$ denotes either raw probabilities ${\widehat{p}}_u$ or calibrated probabilities ${\tilde{p}}_u$, depending on the plot (we use raw for precalibration and calibrated for postcalibration).

2.4. Decision-Analytic Evaluation

For a decision threshold $t\in (0,1)$ define the classified label

$${\widehat{y}}_{u,t}=⊮\{{\tilde{p}}_u\ge t\}.$$

(17)

With $N=|{\mathcal{S}}_{\mathrm{te}}|$, true positives ${\mathrm{TP}}_t$ and false positives ${\mathrm{FP}}_t$ on test, the net benefit at t ([3]) is

$$\mathrm{NB}\left(t\right)={\displaystyle \frac{{\mathrm{TP}}_t}{N}}-{\displaystyle \frac{{\mathrm{FP}}_t}{N}}·{\displaystyle \frac{t}{1-t}}.$$

(18)

Curves $\mathrm{NB}\left(t\right)$ are compared to treat-all and treat-none references to summarize threshold-dependent clinical utility without fixing a single operating point.

2.5. Global Explanation: Permutation Importance and SHAP

Let $m(f;\mathcal{S})$ be a scalar performance functional (e.g., AUPRC) on a dataset $\mathcal{S}$. For feature j, let ${\Pi }_j$ randomly permute column j of the validation design matrix (holding all else fixed). The permutation importance of feature j is the expected performance drop

$$I_j^{\mathrm{perm}}=m\left(f;{\mathcal{S}}_{\mathrm{val}}\right)-\mathbb{E}\left[m\left(f;{\Pi }_j\left({\mathcal{S}}_{\mathrm{val}}\right)\right)\right],$$

(19)

approximated via Monte Carlo over permutations.

SHAP assigns an attribution vector $({\varphi }_{u,1},\dots ,{\varphi }_{u,p})$ and a base value ${\varphi }_{u,0}$ to each instance u, satisfying the Shapley additivity identity

$$f\left({\tilde{\mathbf{x}}}_u\right)={\varphi }_{u,0}+\sum _{j=1}^p{\varphi }_{u,j}.$$

(20)

Global importance is summarized by the mean absolute attribution

$$G_j={\mathbb{E}}_{u\in {\mathcal{S}}_{\mathrm{val}}}\left[|{\varphi }_{u,j}|\right],j=1,\dots ,p,$$

(21)

and visualized via summary and bar plots indicating magnitude and directionality.

2.6. Reproducible Pipeline as Operator Composition

Let $\widehat{\Theta }=(\widehat{\mathcal{I}},\widehat{\mathcal{P}},\widehat{f},\widehat{g})$ denote the fitted operators. The predictor is learned on training data only,

$$\widehat{f}\in arg\underset{f\in \mathcal{F}}{min}{\mathcal{L}}_{\mathrm{train}}\left(f;\widehat{\mathcal{P}}\left(\widehat{\mathcal{I}}\left({\mathbf{X}}_{\mathrm{tr}}\right)\right),{\mathbf{y}}_{\mathrm{tr}}\right),$$

(22)

the calibrator $\widehat{g}$ is selected on validation as in (15), and test-time probabilities are the composition

$${\tilde{p}}_u=\widehat{g}\left(\widehat{f}\left(\widehat{\mathcal{P}}\left(\widehat{\mathcal{I}}\left({\mathbf{x}}_u\right)\right)\right)\right),u\in {\mathcal{S}}_{\mathrm{te}}.$$

(23)

All evaluations in Section 3 (AUROC/AUPRC, reliability/Brier, and decision curves) are computed on (23), preserving the independence between fitting, model selection, and final assessment.

3. Model and Experiments

This study uses the MIMIC-IV database [4,5,6], a large, freely accessible critical care resource containing anonymized demographics, laboratory results, prescriptions, diagnoses, procedures, microbiology, and administrative events from Beth Israel Deaconess Medical Center. We operated on the hospital module and used the tables patients, admissions, transfers, diagnoses_icd, procedures_icd, pharmacy, prescriptions, microbiologyevents, labevents, and the dictionaries d_icd_diagnoses, d_icd_procedures, d_labitems. The code used in the models and experiments is available at https://github.com/EBarbierato/Deployable-Machine-Learning-on-MIMIC-IV (accessed on 7 November 2025).

• Patient-level split and leakage guards.

All preprocessing and modeling were conducted with strict temporal and subject-level guards. We first created patient-level partitions (${\mathcal{S}}_{\mathrm{tr}}/{\mathcal{S}}_{\mathrm{val}}/{\mathcal{S}}_{\mathrm{te}}$) in proportions $60/20/20\%$ to avoid cross-partition contamination; no subject appears in more than one split. Index (admission) time ${\tau }_u$ was harmonized per patient, and all feature construction used only data with timestamps $t\le {\tau }_u$ (temporal guard), as formalized in Section 2. All learned transformers were fit on training only and applied forward to validation and test without refitting.

3.1. Feature Engineering and Temporal Filtering

We constructed admission-time features covering: demographics; admission characteristics and prior-encounter aggregates (e.g., cumulative length of stay before index); counts of diagnoses/procedures; summaries of common lab measurements; coarse medication-exposure indicators; and microbiology flags. Categorical variables were one-hot encoded (rare levels pooled). For numeric features we applied light outlier handling (winsorization at the 1st/99th percentiles) and $log(1+x)$ transforms for heavy-tailed counts. Missingness was handled by SimpleImputer—mean for numeric features and most-frequent for categoricals—fit on training only and applied to validation/test.

• Model-Specific Preprocessing

Preprocessing consists of distinct pipelines: for logistic regressionwe standardized all numeric features with $\mu /\sigma$ computed on training only, then concatenated with one-hot encoded categoricals. For random forest we used the same imputed one-hot design but no scaling (tree-based models are impurity-based and do not require normalization). This separation avoids unnecessary transformations for RF while preserving well-conditioned inputs for LR.

3.2. Model Selection, Training, and Calibration

We trained two single-task baselines for diab_incident_365d.

• Hyperparameter tuning (nested on training).

To ensure a fair comparison and avoid under/overfitting, each model was selected by an inner cross-validation conducted on training data only (validation and test were held out until after selection):

• Logistic Regression (LR). Search over $\lambda =1/C\in \{{10}^{-4},{10}^{-3},{10}^{-2},{10}^{-1},1,10\}$, penalty = L2, solver = lbfgs/liblinear (chosen by convergence), with class_weight = balanced.
• Random Forest (RF). Search over $n\_\mathrm{estimators}\in \{200,400,600\}$, $\max \_\mathrm{depth}\in \{6,8,10,12,\mathrm{None}\}$, $\min \_\mathrm{samples}\_\mathrm{leaf}\in \{1,2,5\}$, $\max \_\mathrm{features}\in \{\mathtt{sqrt},\mathtt{log}\mathtt{2},0.5\}$, bootstrap$\in \{\mathtt{True},\mathtt{False}\}$, with class_weight=balanced_subsample.

For both models we used stratified 5-fold CV on ${\mathcal{S}}_{\mathrm{tr}}$ with the selection metric set to AUROC (primary), recording AUPRC as a secondary criterion (tie-breaker). After selecting hyperparameters, we refit the model on ${\mathcal{S}}_{\mathrm{tr}}$ and evaluated it on ${\mathcal{S}}_{\mathrm{val}}$ to choose a probability calibrator.

• Post hoc calibration and thresholding.

On ${\mathcal{S}}_{\mathrm{val}}$ we fit Platt scaling and isotonic regression and selected the mapping that minimized the Brier score (Section 2). The chosen calibrator was then frozen and applied to ${\mathcal{S}}_{\mathrm{te}}$. Threshold-dependent analyses (decision curves) are reported directly on calibrated probabilities without committing to a single operating point.

3.3. Evaluation Figures

Test-set discrimination is summarized by the calibrated ROC overlay in Figure 1 and by the calibrated precision–recall overlay in Figure 2.

Figure 1. Calibrated ROC on the test set; LR shows higher TPR across FPR ranges compared to RF.

Figure 2. Calibrated precision–recall on the test set; LR maintains higher precision over most recall values in this imbalanced setting.

Decision-curve analysis indicates modest, threshold-dependent utility for both calibrated models. To avoid duplicating nearly identical curves, Figure 3 reports the delta net benefit

$$\Delta \mathrm{NB}\left(t\right)=\mathrm{NB}\times \mathrm{LR}\left(t\right)-\mathrm{NB}\times \mathrm{RF}\left(t\right).$$

Figure 3. Delta net benefit on the test set, defined as $\Delta \mathrm{NB}\left(t\right)={\mathrm{NB}}_{\mathrm{LR}}\left(t\right)-{\mathrm{NB}}_{\mathrm{RF}}\left(t\right)$. Positive values indicate thresholds where calibrated logistic regression offers higher clinical utility than calibrated random forest; negative values indicate the opposite. The panel is zoomed to $t\in [0,0.30]$, the range most relevant for screening.

The delta stays close to zero across clinically plausible thresholds, with RF slightly higher at very low and mid thresholds (e.g., (t = 0.05) and ($t\in [0.20,0.30]$)), and LR competitive elsewhere. The small absolute magnitudes are consistent with a compact admission-time feature set and underscore that threshold selection should be guided by local costs and prevalence rather than assuming a uniform winner.

Probability quality appears in the validation reliability diagram before calibration (Figure 4) and in the test reliability diagrams after calibration (Figure 5 and Figure 6). Global feature reliance is reported as permutation importance for LR and RF (Figure 7 and Figure 8); model-agnostic explainability for the logistic model is shown via SHAP bar and summary plots (Figure 9 and Figure 10).

Figure 4. Validation reliability before calibration (LR); overconfidence at higher bins motivates post hoc calibration.

Figure 5. Test reliability after calibration (LR); closer adherence to the diagonal, especially in mid-probability bins.

Figure 6. Test reliability after calibration (RF); under-prediction improves, though dispersion remains at low probabilities.

Figure 7. Permutation importance on validation for LR (drop in average precision). Age and white ethnicity dominate, with smaller contributions from length of stay and sex.

Figure 8. Permutation importance on validation for RF; age and index length of stay lead, followed by ethnicity indicators.

Figure 9. SHAP (LR): mean absolute attributions; age carries the largest global impact, followed by white ethnicity and sex.

Figure 10. SHAP summary (LR): higher age values push risk upward; other features exert smaller, bidirectional effects consistent with permutation ranking.

3.4. Pipeline Overview

• Operational Summary

Raw CSV extracts are converted to Parquet; timestamps define index admissions. A patient-level cohort is assembled and labels for diab_incident_365d are generated with a look-ahead window, enforcing temporal guards at cohorting and feature extraction so that only pre-index information enters the design matrix. Features are imputed and (for LR only) standardized; the dataset is split into disjoint train/val/test partitions at the patient level. Models are tuned via inner CV on training only, calibrated on validation, and evaluated on the untouched test set via ROC/PR, decision curves, and reliability diagrams. Model reliance combines permutation importance (both models) and SHAP (LR) on validation samples. The pipeline overview is denoted by Figure 11.

Figure 11. End-to-end pipeline with temporal guards before cohorting and feature aggregation, and a patient-level split guard prior to partitioning. Calibrated probabilities feed ROC/PR metrics and decision-curve analysis; permutation importance and SHAP summarize reliance.

This is a single-center retrospective analysis; we therefore emphasize calibration quality and decision utility alongside discrimination and provide a deployment-oriented baseline under strict leakage control. External validation and richer temporal representations are left to future work (Section 6).

4. Discussion

Taken together, the figures provide a consistent view across discrimination, calibration, clinical utility, and model reliance under a leakage-safe pipeline with tuned baselines. After inner cross-validation on training data and post hoc calibration on validation, the calibrated logistic regression dominates the calibrated random forest on the held-out test set in both ROC and precision–recall space (Figure 1 and Figure 2). In an imbalanced endpoint such as incident diabetes at one year, the precision–recall overlay is especially informative and shows a persistent precision advantage for logistic regression over most recall levels.

Calibration materially improves probability quality. Before calibration, the logistic model exhibits overconfidence at higher bins on validation (Figure 4). Applying the selected mapping to test increases alignment with the identity line (Figure 5). The calibrated random forest also improves, but its predicted probabilities are concentrated in a narrower range (Figure 6), limiting granularity for downstream risk stratification. Because clinical workflows consume probabilities rather than ranks, these gains are consequential for thresholding and risk communication.

Decision-curve analysis integrates discrimination, prevalence, and threshold choice into net benefit as a function of the decision threshold. The per-model curves show that calibrated logistic regression achieves consistently higher net benefit across most clinically plausible thresholds, while calibrated random forest remains positive but generally lower.

Model understanding is concordant across permutation importance and SHAP for the logistic model. Age is the dominant predictor in both views, with additional reliance on index length of stay and selected demographic covariates (Figure 7, Figure 8, Figure 9 and Figure 10). The SHAP summary shows a monotone, clinically sensible pattern whereby higher age pushes risk upward, while other variables exert smaller, sometimes bidirectional, effects. These are statements of model reliance rather than causal claims, but they furnish an auditable trail to accompany performance and calibration diagnostics.

Several aspects delimit current utility and suggest improvements. The precalibration reliability for logistic regression departs from the identity line at higher probabilities (Figure 4), illustrating why uncalibrated scores should not be operationalized. Postcalibration, random-forest probabilities remain compressed (Figure 6), which hampers nuanced prioritization even when discrimination is acceptable. Net-benefit magnitudes are small (Figure 3).

With tuned baselines, strict leakage control, and explicit calibration and decision analysis, a compact calibrated logistic model already delivers stable, threshold-aware utility on MIMIC-IV. The modest absolute gains and residual calibration limitations set the agenda for external and temporal validation, longitudinal feature enrichment, and stronger calibrated baselines (e.g., gradient-boosted trees), rather than undermining the value of a deployable, auditable reference.

4.1. Interpreting Net Benefit in Practice

Net benefit $\mathrm{NB}\left(t\right)$, plotted against the decision threshold t, quantifies benefit-adjusted true positives per patient. A value $\mathrm{NB}\left(t\right)=0.012$ means that, at threshold t, the model yields the equivalent of $1.2$ additional true positives per 100 patients, after accounting for the relative harm of false positives implied by t. The threshold encodes the risk–benefit trade-off via

$${\displaystyle \frac{\mathrm{harm}\mathrm{of}\mathrm{FP}}{\mathrm{benefit}\mathrm{of}\mathrm{TP}}}={\displaystyle \frac{t}{1-t}},$$

so, for example, $t=0.20$ corresponds to judging a false alarm as one-quarter as harmful as a missed case. Two operational translations are useful: “benefit-adjusted TPs per 100” equals $100\times \mathrm{NB}\left(t\right)$; and the net reduction in unnecessary interventions per 100 patients versus a treat-all policy is

$${\mathrm{NRUI}}_{100}\left(t\right)=100\times {\displaystyle \frac{{\mathrm{NB}}_{\mathrm{model}}\left(t\right)-{\mathrm{NB}}_{\mathrm{treat}-\mathrm{all}}\left(t\right)}{t/(1-t)}}.$$

In Table 1, these quantities illustrate how calibrated models can deliver positive benefit at thresholds where a treat-all approach is counterproductive.

Table 1. Threshold-wise net benefit (test set) for calibrated models. Treat-none is zero at all thresholds.

Threshold t	Net Benefit (LR_cal)	Net Benefit (RF_cal)	Treat-All	Treat-None
0.05	0.1903	0.1903	0.1444	0.0000
0.10	0.1903	0.1903	0.0968	0.0000
0.20	0.1903	0.1903	−0.0161	0.0000
0.30	0.1903	0.1903	−0.1612	0.0000

4.2. Implementation and Generalization in Clinical Settings

The workflow is designed to translate into a clinical decision support (CDS) service with minimal changes. On the data side, the same temporal guards and patient-level partitioning used in the study become pre-ingestion rules enforced in the hospital’s data platform: only events with timestamps at or before the current admission time are admitted to the feature builder, and the feature map is frozen and versioned to ensure parity with the validated model. Integration with the electronic health record can be achieved via standard interfaces (e.g., FHIR resources for demographics, encounters, observations, medications) and a lightweight inference endpoint that accepts a patient identifier and returns a calibrated probability along with the model version and a timestamp. Because downstream decisions consume probabilities, the calibrator selected on internal validation is exported and attached to the model so that the deployment returns calibrated risks rather than raw scores.

Generalization to new clinical settings follows a site-adaptation protocol that preserves the leakage-safe design while accommodating local prevalence and coding differences. The receiving site first reproduces the cohort and feature map from the site’s own tables and code systems, runs the trained model in “shadow mode” for a short prospective window, and evaluates discrimination, reliability, and net benefit against local labels without exposing outputs to clinicians. If reliability drifts, a lightweight recalibration step is performed on a site-specific validation split, leaving the core model weights unchanged; this preserves interpretability while aligning probabilities to local outcome frequencies. Thresholds are then selected using local decision-curve analysis to reflect resource constraints and clinical policy, and only after this prospective silent period and documentation of performance are alerts turned on.

Operationally, the service is instrumented for continuous monitoring of data drift, calibration drift, and outcome prevalence. The same reliability diagrams, Brier score, ROC–PR overlays, and decision curves used in the study are computed on a rolling window and logged alongside model and calibrator version identifiers. Any material change triggers a scheduled recalibration or a formal review. Governance artifacts—data cards for the cohort and feature space, model cards that report discrimination, calibration, decision-curve ranges, and limitations, and an audit log of versions and thresholds—are kept under change control to support clinical risk management. Clinician-facing displays present the calibrated probability with a short, plain-language explanation, a small set of global reliance cues (e.g., the top permutation-importance features), and links to guidance on follow-up actions defined by the local service line.

To contextualize our discrimination results on MIMIC-IV, we contrast them with recent EHR-based diabetes-risk studies. On our held-out test set, calibrated logistic regression achieved $\mathrm{AUROC}\approx 0.705$ and $\mathrm{AUPRC}\approx 0.308$ at a prevalence of $\pi \approx 0.19$. In a primary-care EHR cohort (age $>50$ years) using 53 routinely collected variables, Stiglic et al. reported incident T2D prediction with $\mathrm{AUROC}\sim 0.70–0.75$ (external validations varying by site and covariate set) [7]. Early EHR prediction efforts for gestational diabetes (a distinct endpoint but informative for EHR screening difficulty) also commonly fall in the $\mathrm{AUROC}\sim 0.70–0.80$ band [8]. While AUPRC is not directly comparable across studies due to varying prevalence [9], our $\mathrm{AUPRC}\approx 0.31$ at $\pi \approx 0.19$ is consistent with reports in adult EHR settings where prevalence typically ranges $10–25\%$. Taken together, these references situate our calibrated baselines within the expected range for compact, admission–time feature sets, and reinforce the need to couple discrimination with calibration and decision-analytic evaluation [9,10].

5. Related Work

Manco et al. [11] introduce HEALER, a zone-based healthcare data-lake architecture (Transient Landing, Raw, Process, Refined, Consumption) implemented with Apache NiFi for ingestion and HDFS for storage, plus Python 3.13.5 components for waveform processing and NLP. A proof-of-concept on MIMIC-III demonstrates increasing ingestion throughput with larger batch sizes, identifies the processing layer as the bottleneck, and argues for governance/security layers and streaming ingestion as future work—highlighting the operational, pipeline-centric nature rather than modeling advances.

Prior studies often frame comorbidity prediction through multitask learning (MTL), where shared representations are learned across related targets. Benchmark work on clinical time series demonstrated consistent, albeit modest, gains for MTL over single-task models under standardised tasks and data from a single health system [12]. While such results motivate cross-task sharing, they less frequently emphasize probability calibration, leakage control at the patient level, and threshold-dependent decision utility—factors that are central when predictions are consumed as probabilities in clinical workflows.

A complementary line of research focuses on deployment-oriented, tightly scoped pipelines that favor transparent models, explicit leakage guards, and task-appropriate evaluation. For example, data-driven diagnostic support for pediatric appendicitis demonstrates how careful variable curation, interpretable modeling, and clinically salient metrics can reduce misdiagnoses and unnecessary procedures in a real decision context [13]. In parallel, systems-engineering studies of healthcare analytics characterize big-data workflows and performance using multiformalism to guide capacity planning and reliable operation of end-to-end pipelines [14]. These perspectives underscore that success in practice hinges not only on raw discrimination but also on stability, calibration, and operational fit.

Beyond modeling architectures, several strands explicitly address reporting quality, calibration, decision-analytic evaluation, and design considerations for thresholded use. Collins et al. [15] outline TRIPOD-AI and PROBAST-AI, emphasizing transparent reporting and risk-of-bias assessment tailored to AI-based prediction models; this foregrounds calibration, data leakage prevention, and intended use. Van Calster et al. [16] argue that miscalibration is the “Achilles’ heel” of predictive analytics, recommending routine assessment and correction, and cautioning against relying on discrimination alone. Vickers and Elkin [3] formalize decision curve analysis (DCA), which compares net benefit across thresholds against treat-all and treat-none policies, aligning model evaluation with clinical trade-offs rather than single operating points. Whittle et al. [17] extend sample-size calculations for threshold-based evaluation, showing how planned operating thresholds and prevalence shape the precision of performance estimates and decision metrics.

Riley et al. [9] discuss a tutorial that bridges the gap between conventional model performance reporting and actual clinical usefulness. Specifically, it stresses that discrimination alone is insufficient and places calibration, decision thresholds, and net benefit at the center of evaluation. The paper provides practical guidance on reading reliability plots, decomposing performance, and linking metrics to decisions, which directly motivates our use of calibrated probabilities, reliability diagrams, and decision-curve analysis in Section 3.

Vickers et al. [10] formalize net benefit and its inference, showing how to quantify clinical utility with confidence intervals and hypothesis tests. They clarify the interpretation of thresholds as harm–benefit trade-offs and how to compare models against treat-all/none strategies. This supports our threshold-wise reporting, the interpretation of net benefit magnitudes, and the recommendation to select operating points via decision curves rather than a single fixed cutoff.

Moreover, Corbin  et al. [18] present a pragmatic framework for taking ML models from development to production in healthcare, covering integration, versioning, monitoring, and drift management. The emphasis on transparent artifacts, shadow deployment, calibration checks, and governance aligns with our deployment-oriented discussion: exporting the chosen calibrator with the model, monitoring calibration and prevalence over time, and performing lightweight site-specific recalibration prior to activation.

As per Table 2, this paper positions itself at that deployment-oriented intersection. We construct a leakage-safe, patient-level pipeline on MIMIC-IV hospital data; we train two single-task baselines (logistic regression and random forest) for incident diabetes at 365 days; we calibrate probabilities post hoc (Platt and isotonic) and evaluate with ROC/PR curves, reliability diagrams, and decision-curve analysis; and we summarize model reliance with permutation importance and SHAP for the calibrated baselines. In doing so, the work shows that, under strict temporal and subject-level guards with calibration and decision-analytic reporting, a compact calibrated baseline can offer competitive performance and clearer operational traceability on a widely used critical-care dataset. Relative to prior literature, our contribution is methodological and reporting-centric: we assemble an end-to-end, reproducible workflow that surfaces discrimination, calibration, decision utility, and model reliance in a single dossier suitable for clinical review, aligning explicitly with contemporary guidance on reporting, calibration, and threshold-based evaluation [3,15,16,17].

Table 2. Positioning relative to representative strands in the literature.

Scope	Typical Emphasis	Alignment to This Work
Healthcare data lake architecture (HEALER, DataPlat’23) [11]	Zone-based data lake for healthcare; NiFi–HDFS ingestion/storage; batch vs. streaming; waveform/NLP processing; performance characterization of ingestion/processing/access	Our work operates downstream of data-platform concerns; we assume a curated hospital module (MIMIC-IV) and focus on leakage-safe modeling, calibration, DCA, and explainability.
MTL benchmarks on single-system data [12]	Shared representation learning; multi-outcome prediction; modest average gains; limited calibration/decision reporting	We do not introduce new MTL; instead, we provide calibrated single-task baselines with decision-analytic evaluation on MIMIC-IV.
Deployment-oriented, task-specific clinical ML [13]	Leakage control, interpretable models, clinically meaningful metrics, and outcomes	We adopt the same philosophy: patient-level splits, calibration, ROC/PR plus decision curves, and concise explanations to support review.
Systems/operations for healthcare analytics [14]	Workflow characterization, performance/throughput analysis, reliability considerations for pipelines	Our modular scripts and artifact checks speak to these concerns, prioritizing reproducibility and operational readiness.
Reporting and risk-of-bias guidance (TRIPOD-AI, PROBAST-AI) [15]	Transparent reporting, bias assessment, intended-use clarity for AI prediction models	We align by documenting leakage guards, calibration choice, thresholds, and decision utility in a reproducible bundle.
Calibration as a first-class metric [16]	Calibration assessment/correction; limits of discrimination-only summaries	Reliability diagrams, Brier scores, and post hoc Platt/isotonic selection operationalize these recommendations.
Decision-analytic evaluation (DCA) with inference [10]	Net benefit across thresholds; treat-all/none comparators; confidence intervals and hypothesis testing for DCA	We report DCA curves and interpret net benefit magnitudes; our threshold selection guidance is consistent with these recommendations.
Threshold-based sample-size planning [17]	Precision of thresholded performance estimates as a function of prevalence and N	We discuss threshold-dependent precision and encourage prospective planning for operating-point evaluation.
From performance to clinical usefulness [9]	Bridging discrimination to decision impact; emphasis on calibration, thresholds, and net benefit	Our calibrated probabilities, reliability diagrams, and decision-curve analysis directly follow these best practices.
Deployment framework for real-world ML in healthcare [18]	EHR integration, versioning, monitoring, drift management; shadow deployment and governance	Our deployment notes (calibrator export, shadow runs, monitoring calibration/prevalence) mirror this operational blueprint.
This work	Leakage-safe, patient-level pipeline; post hoc calibration (Platt/isotonic); ROC/PR plus DCA; reliability diagrams; SHAP and permutation-importance summaries; compact, reproducible implementation.	Single-center retrospective data; moderate absolute gains and narrow net-benefit margins; no external or groupwise validation yet; admission-time features only.

It should be noticed that, prior MTL studies underscore the potential of shared representation learning across related clinical tasks but often underreport calibration and threshold-dependent clinical utility. Our findings complement that literature by demonstrating that a calibrated, transparent baseline—with explicit decision-analytic reporting—can provide a strong, readily deployable reference. By tying the pipeline to emerging reporting guidance and decision-focused evaluation, we offer a pragmatic template for external validation and, where justified by incremental utility, the integration of more expressive models.

6. Conclusions and Future Work

This study delivers a leakage-safe, calibration-first pipeline for incident diabetes prediction on MIMIC-IV, with strict temporal and patient-level guards, model-specific preprocessing, tuned baselines, and post hoc calibration.

For RQ1 (which calibrated baseline discriminates best), the tuned and calibrated logistic regression consistently exceeded the tuned and calibrated random forest on the held-out test set. Calibrated ROC and PR overlays indicate higher AUROC/AUPRC for logistic regression across operating regions (Figure 1 and Figure 2), suggesting that in this admission–time, tabular setting, a compact linear model is a strong and parsimonious choice relative to a deeper ensemble.

For RQ2 (does calibration improve probability quality and translate to decision utility), reliability diagrams show that post hoc calibration materially improves probability accuracy. The logistic model aligns more closely with the identity line from validation (uncalibrated) to test (calibrated) (Figure 4 and Figure 5); the calibrated random forest improves as well, albeit with compressed probability ranges (Figure 6). Decision-curve analysis shows modest but non-negative net benefit across clinically plausible thresholds, with calibrated logistic regression dominating random forest over most of the range. These findings support calibrated, threshold-aware deployment when probabilities are consumed for triage.

For RQ3 (which predictors most influence risk and are they clinically coherent), explanations are concordant with established risk profiles. SHAP summaries for logistic regression and permutation importance for both models highlight age as the dominant contributor, with additional reliance on index length of stay, sex, and ethnicity indicators (Figure 7, Figure 8, Figure 9 and Figure 10). These are associative statements of model reliance rather than causal effects, but they provide transparent context to accompany discrimination, calibration, and decision utility.

The contribution is implementation and reproducibility rather than methodological innovation: we consolidate best practices—leakage control, calibration, decision curves, and transparent reliance summaries—into a single, documented workflow that returns calibrated probabilities ready for thresholded use in clinical decision support. All artifacts necessary for clinical review and governance (metrics, reliability, decision curves, model/explainer versions, and data/model cards) are surfaced to enable auditing, monitoring, and safe integration.

Limitations include single-center retrospective data, no external or temporal validation in this release, absence of groupwise fairness analyses, and limited longitudinal features. Future work will (i) perform multi-site external and temporal validation with lightweight recalibration; (ii) report groupwise discrimination, calibration, and net benefit; (iii) enrich features with longitudinal trajectories while preserving leakage guards; (iv) use benchmark calibrated, still interpretable reference learners (e.g., gradient-boosted trees) under the same selection and calibration protocol. These steps aim to strengthen transportability and clinical usefulness while retaining the transparency required for adoption.