Explainable Machine Learning-Based Risk Prediction Model for In-Hospital Mortality after Continuous Renal Replacement Therapy Initiation
Abstract
1. Introduction
2. Materials and Methods
2.1. Study Participants
2.2. Outcomes
2.3. Feature Engineering
2.4. Statistical Analysis and ML Algorithm
3. Results
3.1. Study Population Characteristics
3.2. Model Prediction of In-Hospital Death after CRRT Initiation
3.3. Model Explanations
4. Discussion
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
Appendix A
Appendix B
Appendix C
Appendix D
References
- Hoste, E.A.; Bagshaw, S.M.; Bellomo, R.; Cely, C.M.; Colman, R.; Cruz, D.N.; Edipidis, K.; Forni, L.G.; Gomersall, C.D.; Govil, D.; et al. Epidemiology of acute kidney injury in critically ill patients: The multinational AKI-EPI study. Intensive Care Med. 2015, 41, 1411–1423. [Google Scholar] [CrossRef] [PubMed]
- Hoste, E.A.J.; Kellum, J.A.; Selby, N.M.; Zarbock, A.; Palevsky, P.M.; Bagshaw, S.M.; Goldstein, S.L.; Cerdá, J.; Chawla, L.S. Global epidemiology and outcomes of acute kidney injury. Nat. Rev. Nephrol. 2018, 14, 607–625. [Google Scholar] [CrossRef] [PubMed]
- Griffin, B.R.; Liu, K.D.; Teixeira, J.P. Critical Care Nephrology: Core Curriculum 2020. Am. J. Kidney Dis. 2020, 75, 435–452. [Google Scholar] [CrossRef] [PubMed]
- Macedo, E.; Mehta, R.L. Continuous Dialysis Therapies: Core Curriculum 2016. Am. J. Kidney Dis. 2016, 68, 645–657. [Google Scholar] [CrossRef]
- Karkar, A.; Ronco, C. Prescription of CRRT: A pathway to optimize therapy. Ann. Intensive Care 2020, 10, 32. [Google Scholar] [CrossRef]
- Kee, Y.K.; Kim, D.; Kim, S.J.; Kang, D.H.; Choi, K.B.; Oh, H.J.; Ryu, D.R. Factors Associated with Early Mortality in Critically Ill Patients Following the Initiation of Continuous Renal Replacement Therapy. J. Clin. Med. 2018, 7, 334. [Google Scholar] [CrossRef]
- Kang, M.W.; Kim, J.; Kim, D.K.; Oh, K.H.; Joo, K.W.; Kim, Y.S.; Han, S.S. Machine learning algorithm to predict mortality in patients undergoing continuous renal replacement therapy. Crit. Care 2020, 24, 42. [Google Scholar] [CrossRef]
- Palevsky, P.M.; Zhang, J.H.; O’Connor, T.Z.; Chertow, G.M.; Crowley, S.T.; Choudhury, D.; Finkel, K.; Kellum, J.A.; Paganini, E.; Schein, R.M.; et al. Intensity of renal support in critically ill patients with acute kidney injury. N. Engl. J. Med. 2008, 359, 7–20. [Google Scholar] [CrossRef]
- De Corte, W.; Dhondt, A.; Vanholder, R.; De Waele, J.; Decruyenaere, J.; Sergoyne, V.; Vanhalst, J.; Claus, S.; Hoste, E.A. Long-term outcome in ICU patients with acute kidney injury treated with renal replacement therapy: A prospective cohort study. Crit. Care 2016, 20, 256. [Google Scholar] [CrossRef]
- Singbartl, K.; Kellum, J.A. AKI in the ICU: Definition, epidemiology, risk stratification, and outcomes. Kidney Int. 2012, 81, 819–825. [Google Scholar] [CrossRef]
- da Hora Passos, R.; Ramos, J.G.; Mendonça, E.J.; Miranda, E.A.; Dutra, F.R.; Coelho, M.F.; Pedroza, A.C.; Correia, L.C.; Batista, P.B.; Macedo, E.; et al. A clinical score to predict mortality in septic acute kidney injury patients requiring continuous renal replacement therapy: The HELENICC score. BMC Anesthesiol. 2017, 17, 21. [Google Scholar] [CrossRef] [PubMed]
- Kim, Y.; Park, N.; Kim, J.; Kim, D.K.; Chin, H.J.; Na, K.Y.; Joo, K.W.; Kim, Y.S.; Kim, S.; Han, S.S. Development of a new mortality scoring system for acute kidney injury with continuous renal replacement therapy. Nephrology 2019, 24, 1233–1240. [Google Scholar] [CrossRef] [PubMed]
- Pattharanitima, P.; Vaid, A.; Jaladanki, S.K.; Paranjpe, I.; O’Hagan, R.; Chauhan, K.; Van Vleck, T.T.; Duffy, A.; Chaudhary, K.; Glicksberg, B.S.; et al. Comparison of Approaches for Prediction of Renal Replacement Therapy-Free Survival in Patients with Acute Kidney Injury. Blood Purif. 2021, 50, 621–627. [Google Scholar] [CrossRef] [PubMed]
- Kundu, S. AI in medicine must be explainable. Nat. Med. 2021, 27, 1328. [Google Scholar] [CrossRef]
- Jiang, Z.; Bo, L.; Xu, Z.; Song, Y.; Wang, J.; Wen, P.; Wan, X.; Yang, T.; Deng, X.; Bian, J. An explainable machine learning algorithm for risk factor analysis of in-hospital mortality in sepsis survivors with ICU readmission. Comput. Methods Programs Biomed. 2021, 204, 106040. [Google Scholar] [CrossRef]
- Ogundimu, E.O.; Altman, D.G.; Collins, G.S. Adequate sample size for developing prediction models is not simply related to events per variable. J. Clin. Epidemiol. 2016, 76, 175–182. [Google Scholar] [CrossRef]
- Chen, T.; Guestrin, C. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, 13–17 August 2016; pp. 785–794. [Google Scholar]
- Lee, H.J.; Son, Y.J. Factors Associated with In-Hospital Mortality after Continuous Renal Replacement Therapy for Critically Ill Patients: A Systematic Review and Meta-Analysis. Int. J. Environ. Res. Public Health 2020, 17, 8781. [Google Scholar] [CrossRef]
- Knaus, W.A.; Draper, E.A.; Wagner, D.P.; Zimmerman, J.E. APACHE II: A severity of disease classification system. Crit. Care Med. 1985, 13, 818–829. [Google Scholar] [CrossRef]
- Vincent, J.L.; Moreno, R.; Takala, J.; Willatts, S.; De Mendonça, A.; Bruining, H.; Reinhart, C.K.; Suter, P.M.; Thijs, L.G. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. On behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine. Intensive Care Med. 1996, 22, 707–710. [Google Scholar] [CrossRef]
- Yen, C.L.; Fan, P.C.; Kuo, G.; Lee, C.C.; Chen, J.J.; Lee, T.H.; Tu, Y.R.; Hsu, H.H.; Tian, Y.C.; Chang, C.H. Prognostic Performance of Existing Scoring Systems among Critically Ill Patients Requiring Continuous Renal Replacement Therapy: An Observational Study. J. Clin. Med. 2021, 10, 4592. [Google Scholar] [CrossRef]
- Kang, M.W.; Kim, S.; Kim, Y.C.; Kim, D.K.; Oh, K.H.; Joo, K.W.; Kim, Y.S.; Han, S.S. Machine learning model to predict hypotension after starting continuous renal replacement therapy. Sci. Rep. 2021, 11, 17169. [Google Scholar] [CrossRef] [PubMed]
- Pattharanitima, P.; Thongprayoon, C.; Kaewput, W.; Qureshi, F.; Qureshi, F.; Petnak, T.; Srivali, N.; Gembillo, G.; O’Corragain, O.A.; Chesdachai, S.; et al. Machine Learning Prediction Models for Mortality in Intensive Care Unit Patients with Lactic Acidosis. J. Clin. Med. 2021, 10, 5021. [Google Scholar] [CrossRef] [PubMed]
- Kim, J.H.; Kwon, Y.S.; Baek, M.S. Machine Learning Models to Predict 30-Day Mortality in Mechanically Ventilated Patients. J. Clin. Med. 2021, 10, 2172. [Google Scholar] [CrossRef] [PubMed]
- Ponce, D.; de Andrade, L.G.M.; Claure-Del Granado, R.; Ferreiro-Fuentes, A.; Lombardi, R. Development of a prediction score for in-hospital mortality in COVID-19 patients with acute kidney injury: A machine learning approach. Sci. Rep. 2021, 11, 24439. [Google Scholar] [CrossRef]
- Lundberg, S.M.; Lee, S.-I. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017; pp. 4768–4777. [Google Scholar]
- Kor, C.T.; Li, Y.R.; Lin, P.R.; Lin, S.H.; Wang, B.Y.; Lin, C.H. Explainable Machine Learning Model for Predicting First-Time Acute Exacerbation in Patients with Chronic Obstructive Pulmonary Disease. J. Pers. Med. 2022, 12, 228. [Google Scholar] [CrossRef]
- Oh, T.R.; Song, S.H.; Choi, H.S.; Suh, S.H.; Kim, C.S.; Jung, J.Y.; Choi, K.H.; Oh, K.H.; Ma, S.K.; Bae, E.H.; et al. Predictive Model for High Coronary Artery Calcium Score in Young Patients with Non-Dialysis Chronic Kidney Disease. J. Pers. Med. 2021, 11, 1372. [Google Scholar] [CrossRef]
- Lu, C.; Song, J.; Li, H.; Yu, W.; Hao, Y.; Xu, K.; Xu, P. Predicting Venous Thrombosis in Osteoarthritis Using a Machine Learning Algorithm: A Population-Based Cohort Study. J. Pers. Med. 2022, 12, 114. [Google Scholar] [CrossRef]
- Viton, F.; Elbattah, M.; Guérin, J.L.; Dequen, G. Heatmaps for Visual Explainability of CNN-Based Predictions for Multivariate Time Series with Application to Healthcare. In Proceedings of the 2020 IEEE International Conference on Healthcare Informatics (ICHI), Oldenburg, Germany, 30 November–3 December 2020; pp. 1–8. [Google Scholar]
- Stenwig, E.; Salvi, G.; Rossi, P.S.; Skjærvold, N.K. Comparative analysis of explainable machine learning prediction models for hospital mortality. BMC Med. Res. Methodol. 2022, 22, 53. [Google Scholar] [CrossRef]
- Mao, I.-C.; Lin, P.-R.; Wu, S.-H.; Hsu, H.-H.; Hung, P.-S.; Kor, C.-T. First 24-Hour Potassium Concentration and Variability and Association with Mortality in Patients Requiring Continuous Renal Replacement Therapy in Intensive Care Units: A Hospital-Based Retrospective Cohort Study. J. Clin. Med. 2022, 11, 3383. [Google Scholar] [CrossRef]
- Sheng, S.; Zhang, Y.H.; Ma, H.K.; Huang, Y. Albumin levels predict mortality in sepsis patients with acute kidney injury undergoing continuous renal replacement therapy: A secondary analysis based on a retrospective cohort study. BMC Nephrol. 2022, 23, 52. [Google Scholar] [CrossRef]
- Schork, A.; Moll, K.; Haap, M.; Riessen, R.; Wagner, R. Course of lactate, pH and base excess for prediction of mortality in medical intensive care patients. PLoS ONE 2021, 16, e0261564. [Google Scholar] [CrossRef] [PubMed]
- Hu, C.; Hu, B.; Wang, J.; Peng, Z.; Kashani, K.B.; Li, J. Diabetes mellitus is associated with lower mortality in intensive care unit: A multicenter historical cohort study. Res. Sq. 2020. [Google Scholar] [CrossRef]
- Rajawat, M.S.; Rathore, S.; Choudhary, M. pH disorders and mortality in surgical intensive care unit patients. Int. Surg. J. 2016, 3, 905–907. [Google Scholar] [CrossRef][Green Version]
Features | No in-Hospital Death | In-Hospital Death | p-Value | Train Dataset | Test Dataset | p-Value |
---|---|---|---|---|---|---|
Sample size | 908 | 2024 | 2345 | 587 | ||
Characteristics | ||||||
Age, yr | 66 (55–76) | 72 (60–81) | <0.001 | 70 (59–80) | 70 (57–81) | 0.768 |
BMI | 25.6 (22.5–29) | 24 (21.1–27.3) | <0.001 | 24.5 (21.4–27.9) | 24.7 (21.5–28.1) | 0.303 |
MDC | 5 (4–8) | 5 (4–7) | 0.979 | 5 (4–7) | 5 (4–8) | 0.198 |
APACHE II at admission | 26 (20–32) | 30 (24–37) | <0.001 | 29 (22–35) | 29 (22–35) | 0.571 |
Timing of initiated CRRT | ||||||
Early-strategy group § | 518 (57%) | 962 (47.5%) | <0.001 | 1200 (51.2%) | 280 (47.7%) | 0.132 |
Delayed Strategy | 390 (43%) | 1062 (52.5%) | 1145 (48.8%) | 307 (52.3%) | ||
Vital Sign at CRRT—no. (%) | ||||||
Systolic BP (mmHg) | 115.7 (105.4–130.3) | 108.2 (98.2–119.7) | <0.001 | 110.6 (100.3–122.8) | 109.6 (100.8–123) | 0.738 |
Diastolic BP (mmHg) | 60.5 (53.5–70) | 57.7 (50.3–65.5) | <0.001 | 58.6 (51.4–66.8) | 58.1 (50.7–66.4) | 0.480 |
Respiratory rate (/min) | 18.8 (16.3–21.4) | 20 (17–23.3) | <0.001 | 19.7 (16.7–22.7) | 19.7 (16.8–22.8) | 0.884 |
SPO2 | 97.8 (96–99.1) | 96.6 (93.5–98.6) | <0.001 | 97.1 (94.4–98.8) | 96.9 (94.4–98.7) | 0.302 |
Fluid balance before CRRT—ml/24 hr | 1663 (500–2992) | 2221 (1000–3750) | <0.001 | 2000 (817–3550) | 2000 (835–3429) | 0.942 |
Coexisting conditions—no. (%) | ||||||
Diabetes mellitus | 368 (40.5%) | 642 (31.7%) | <0.001 | 812 (34.6%) | 198 (33.7%) | 0.683 |
Multiple organ support before CRRT—no. (%) | ||||||
Invasive mechanical ventilation | 719 (79.2%) | 1832 (90.5%) | <0.001 | 2043 (87.1%) | 508 (86.5%) | 0.709 |
Vasopressors support with norepinephrine or epinephrine | 632 (69.6%) | 1792 (88.5%) | <0.001 | 1931 (82.3%) | 493 (84%) | 0.348 |
Vasopressin | 178 (19.6%) | 843 (41.7%) | <0.001 | 801 (34.2%) | 220 (37.5%) | 0.131 |
Medication use before CRRT—no. (%) | ||||||
Corticosteroids | 429 (47.2%) | 1247 (61.6%) | <0.001 | 1347 (57.4%) | 329 (56%) | 0.542 |
Parenteral Nutrition | 678 (74.7%) | 1769 (87.4%) | <0.001 | 1959 (83.5%) | 488 (83.1%) | 0.813 |
Antibiotics | 828 (91.2%) | 1948 (96.2%) | <0.001 | 2219 (94.6%) | 557 (94.9%) | 0.800 |
Furosemide | 438 (48.2%) | 1104 (54.5%) | 0.002 | 1218 (51.9%) | 324 (55.2%) | 0.158 |
Laboratory data before CRRT | ||||||
Serum creatinine (mg/dL) | 2.6 (1.4–5.2) | 2 (1.2–3.6) | <0.001 | 2.1 (1.3–4) | 2.2 (1.3–4.2) | 0.640 |
Serum potassium (mmol/L) | 3.9 (3.4–4.4) | 4 (3.4–4.8) | <0.001 | 3.9 (3.4–4.6) | 4 (3.4–4.7) | 0.699 |
Serum albumin (g/dL) | 2.6 (2.1–3.1) | 2.2 (1.7–2.7) | <0.001 | 2.3 (1.8–2.8) | 2.3 (1.8–2.8) | 0.448 |
Lactate, mmol/L | 2.9 (1.4–6.6) | 4.8 (2.2–10) | <0.001 | 4.1 (1.9–8.9) | 4 (1.9–9.5) | 0.906 |
Platelet count | 126 (75–199) | 96 (53–165) | <0.001 | 106 (59–175) | 105 (56–177) | 0.930 |
pH | 7.4 (7.3–7.4) | 7.3 (7.2–7.4) | <0.001 | 7.3 (7.2–7.4) | 7.3 (7.2–7.4) | 0.931 |
Serum sodium (mmol/L) | 138 (134–141) | 139 (134–144) | <0.001 | 138 (134–143) | 138 (134–143) | 0.269 |
RDW | 15.4 (14.3–17) | 16.1 (14.8–18.3) | <0.001 | 15.8 (14.6–17.8) | 16.1 (14.6–18.6) | 0.039 |
Mg | 2 (1.8–2.3) | 2.1 (1.8–2.4) | <0.001 | 2.1 (1.8–2.4) | 2.1 (1.8–2.5) | 0.090 |
INR | 1.2 (1.1–1.4) | 1.3 (1.1–1.7) | <0.001 | 1.3 (1.1–1.6) | 1.3 (1.1–1.7) | 0.785 |
APTT | 36.5 (30.5–54.1) | 41 (32.8–69.6) | <0.001 | 39.2 (31.8–61.5) | 40.8 (32.4–69.7) | 0.027 |
O2 Saturation | 98.8 (96.6–99.8) | 98 (95.1–99.5) | <0.001 | 98.3 (95.6–99.7) | 98.3 (95.7–99.6) | 0.955 |
Outcome | ||||||
In-hospital mortality | 0 (0%) | 2024 (100%) | -- | 1625 (69.0%) | 399 (67.0%) | 0.535 |
28 days mortality | 2 (0.2%) | 1733 (85.6%) | <0.001 | 1385 (59.1%) | 350 (59.6%) | 0.804 |
90 days mortality | 20 (2.2%) | 1984 (98%) | <0.001 | 1607 (68.5%) | 397 (67.6%) | 0.676 |
Model | AUC | Threshold | Sensitivity | Specificity | PPV | NPV | F1 Score | Accuracy |
---|---|---|---|---|---|---|---|---|
Support Vector Machine (SVM) with radial kernel | 0.7500 | 0.6749 | 62.66% | 78.72% | 86.21% | 49.83% | 72.57% | 67.80% |
Support Vector Machine (SVM) with polynomial kernel | 0.7836 | 0.6588 | 67.67% | 77.13% | 86.26% | 52.92% | 75.84% | 73.59% |
Support Vector Machine (SVM) with Sigmoid kernel | 0.7563 | 0.6897 | 71.43% | 75.53% | 86.10% | 55.47% | 78.08% | 72.74% |
Random Forest (RF) | 0.8161 | 0.6587 | 74.69% | 75.00% | 86.38% | 58.26% | 80.11% | 74.79% |
Extreme Gradient Boosting (XGBoost) | 0.8064 | 0.7234 | 73.43% | 80.32% | 88.79% | 58.75% | 80.38% | 75.64% |
Gradient boosted machines (GBMs) | 0.8227 | 0.7216 | 74.19% | 78.72% | 88.10% | 58.96% | 80.55% | 75.64% |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Hung, P.-S.; Lin, P.-R.; Hsu, H.-H.; Huang, Y.-C.; Wu, S.-H.; Kor, C.-T. Explainable Machine Learning-Based Risk Prediction Model for In-Hospital Mortality after Continuous Renal Replacement Therapy Initiation. Diagnostics 2022, 12, 1496. https://doi.org/10.3390/diagnostics12061496
Hung P-S, Lin P-R, Hsu H-H, Huang Y-C, Wu S-H, Kor C-T. Explainable Machine Learning-Based Risk Prediction Model for In-Hospital Mortality after Continuous Renal Replacement Therapy Initiation. Diagnostics. 2022; 12(6):1496. https://doi.org/10.3390/diagnostics12061496
Chicago/Turabian StyleHung, Pei-Shan, Pei-Ru Lin, Hsin-Hui Hsu, Yi-Chen Huang, Shin-Hwar Wu, and Chew-Teng Kor. 2022. "Explainable Machine Learning-Based Risk Prediction Model for In-Hospital Mortality after Continuous Renal Replacement Therapy Initiation" Diagnostics 12, no. 6: 1496. https://doi.org/10.3390/diagnostics12061496
APA StyleHung, P.-S., Lin, P.-R., Hsu, H.-H., Huang, Y.-C., Wu, S.-H., & Kor, C.-T. (2022). Explainable Machine Learning-Based Risk Prediction Model for In-Hospital Mortality after Continuous Renal Replacement Therapy Initiation. Diagnostics, 12(6), 1496. https://doi.org/10.3390/diagnostics12061496