Explainable Machine Learning-Based Risk Prediction Model for In-Hospital Mortality after Continuous Renal Replacement Therapy Initiation
Abstract
:1. Introduction
2. Materials and Methods
2.1. Study Participants
2.2. Outcomes
2.3. Feature Engineering
2.4. Statistical Analysis and ML Algorithm
3. Results
3.1. Study Population Characteristics
3.2. Model Prediction of In-Hospital Death after CRRT Initiation
3.3. Model Explanations
4. Discussion
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
Appendix A
Appendix B
Appendix C
Appendix D
References
- Hoste, E.A.; Bagshaw, S.M.; Bellomo, R.; Cely, C.M.; Colman, R.; Cruz, D.N.; Edipidis, K.; Forni, L.G.; Gomersall, C.D.; Govil, D.; et al. Epidemiology of acute kidney injury in critically ill patients: The multinational AKI-EPI study. Intensive Care Med. 2015, 41, 1411–1423. [Google Scholar] [CrossRef] [PubMed]
- Hoste, E.A.J.; Kellum, J.A.; Selby, N.M.; Zarbock, A.; Palevsky, P.M.; Bagshaw, S.M.; Goldstein, S.L.; Cerdá, J.; Chawla, L.S. Global epidemiology and outcomes of acute kidney injury. Nat. Rev. Nephrol. 2018, 14, 607–625. [Google Scholar] [CrossRef] [PubMed]
- Griffin, B.R.; Liu, K.D.; Teixeira, J.P. Critical Care Nephrology: Core Curriculum 2020. Am. J. Kidney Dis. 2020, 75, 435–452. [Google Scholar] [CrossRef] [PubMed]
- Macedo, E.; Mehta, R.L. Continuous Dialysis Therapies: Core Curriculum 2016. Am. J. Kidney Dis. 2016, 68, 645–657. [Google Scholar] [CrossRef] [Green Version]
- Karkar, A.; Ronco, C. Prescription of CRRT: A pathway to optimize therapy. Ann. Intensive Care 2020, 10, 32. [Google Scholar] [CrossRef]
- Kee, Y.K.; Kim, D.; Kim, S.J.; Kang, D.H.; Choi, K.B.; Oh, H.J.; Ryu, D.R. Factors Associated with Early Mortality in Critically Ill Patients Following the Initiation of Continuous Renal Replacement Therapy. J. Clin. Med. 2018, 7, 334. [Google Scholar] [CrossRef] [Green Version]
- Kang, M.W.; Kim, J.; Kim, D.K.; Oh, K.H.; Joo, K.W.; Kim, Y.S.; Han, S.S. Machine learning algorithm to predict mortality in patients undergoing continuous renal replacement therapy. Crit. Care 2020, 24, 42. [Google Scholar] [CrossRef] [Green Version]
- Palevsky, P.M.; Zhang, J.H.; O’Connor, T.Z.; Chertow, G.M.; Crowley, S.T.; Choudhury, D.; Finkel, K.; Kellum, J.A.; Paganini, E.; Schein, R.M.; et al. Intensity of renal support in critically ill patients with acute kidney injury. N. Engl. J. Med. 2008, 359, 7–20. [Google Scholar] [CrossRef]
- De Corte, W.; Dhondt, A.; Vanholder, R.; De Waele, J.; Decruyenaere, J.; Sergoyne, V.; Vanhalst, J.; Claus, S.; Hoste, E.A. Long-term outcome in ICU patients with acute kidney injury treated with renal replacement therapy: A prospective cohort study. Crit. Care 2016, 20, 256. [Google Scholar] [CrossRef] [Green Version]
- Singbartl, K.; Kellum, J.A. AKI in the ICU: Definition, epidemiology, risk stratification, and outcomes. Kidney Int. 2012, 81, 819–825. [Google Scholar] [CrossRef] [Green Version]
- da Hora Passos, R.; Ramos, J.G.; Mendonça, E.J.; Miranda, E.A.; Dutra, F.R.; Coelho, M.F.; Pedroza, A.C.; Correia, L.C.; Batista, P.B.; Macedo, E.; et al. A clinical score to predict mortality in septic acute kidney injury patients requiring continuous renal replacement therapy: The HELENICC score. BMC Anesthesiol. 2017, 17, 21. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Kim, Y.; Park, N.; Kim, J.; Kim, D.K.; Chin, H.J.; Na, K.Y.; Joo, K.W.; Kim, Y.S.; Kim, S.; Han, S.S. Development of a new mortality scoring system for acute kidney injury with continuous renal replacement therapy. Nephrology 2019, 24, 1233–1240. [Google Scholar] [CrossRef] [PubMed]
- Pattharanitima, P.; Vaid, A.; Jaladanki, S.K.; Paranjpe, I.; O’Hagan, R.; Chauhan, K.; Van Vleck, T.T.; Duffy, A.; Chaudhary, K.; Glicksberg, B.S.; et al. Comparison of Approaches for Prediction of Renal Replacement Therapy-Free Survival in Patients with Acute Kidney Injury. Blood Purif. 2021, 50, 621–627. [Google Scholar] [CrossRef] [PubMed]
- Kundu, S. AI in medicine must be explainable. Nat. Med. 2021, 27, 1328. [Google Scholar] [CrossRef]
- Jiang, Z.; Bo, L.; Xu, Z.; Song, Y.; Wang, J.; Wen, P.; Wan, X.; Yang, T.; Deng, X.; Bian, J. An explainable machine learning algorithm for risk factor analysis of in-hospital mortality in sepsis survivors with ICU readmission. Comput. Methods Programs Biomed. 2021, 204, 106040. [Google Scholar] [CrossRef]
- Ogundimu, E.O.; Altman, D.G.; Collins, G.S. Adequate sample size for developing prediction models is not simply related to events per variable. J. Clin. Epidemiol. 2016, 76, 175–182. [Google Scholar] [CrossRef] [Green Version]
- Chen, T.; Guestrin, C. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, 13–17 August 2016; pp. 785–794. [Google Scholar]
- Lee, H.J.; Son, Y.J. Factors Associated with In-Hospital Mortality after Continuous Renal Replacement Therapy for Critically Ill Patients: A Systematic Review and Meta-Analysis. Int. J. Environ. Res. Public Health 2020, 17, 8781. [Google Scholar] [CrossRef]
- Knaus, W.A.; Draper, E.A.; Wagner, D.P.; Zimmerman, J.E. APACHE II: A severity of disease classification system. Crit. Care Med. 1985, 13, 818–829. [Google Scholar] [CrossRef]
- Vincent, J.L.; Moreno, R.; Takala, J.; Willatts, S.; De Mendonça, A.; Bruining, H.; Reinhart, C.K.; Suter, P.M.; Thijs, L.G. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. On behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine. Intensive Care Med. 1996, 22, 707–710. [Google Scholar] [CrossRef]
- Yen, C.L.; Fan, P.C.; Kuo, G.; Lee, C.C.; Chen, J.J.; Lee, T.H.; Tu, Y.R.; Hsu, H.H.; Tian, Y.C.; Chang, C.H. Prognostic Performance of Existing Scoring Systems among Critically Ill Patients Requiring Continuous Renal Replacement Therapy: An Observational Study. J. Clin. Med. 2021, 10, 4592. [Google Scholar] [CrossRef]
- Kang, M.W.; Kim, S.; Kim, Y.C.; Kim, D.K.; Oh, K.H.; Joo, K.W.; Kim, Y.S.; Han, S.S. Machine learning model to predict hypotension after starting continuous renal replacement therapy. Sci. Rep. 2021, 11, 17169. [Google Scholar] [CrossRef] [PubMed]
- Pattharanitima, P.; Thongprayoon, C.; Kaewput, W.; Qureshi, F.; Qureshi, F.; Petnak, T.; Srivali, N.; Gembillo, G.; O’Corragain, O.A.; Chesdachai, S.; et al. Machine Learning Prediction Models for Mortality in Intensive Care Unit Patients with Lactic Acidosis. J. Clin. Med. 2021, 10, 5021. [Google Scholar] [CrossRef] [PubMed]
- Kim, J.H.; Kwon, Y.S.; Baek, M.S. Machine Learning Models to Predict 30-Day Mortality in Mechanically Ventilated Patients. J. Clin. Med. 2021, 10, 2172. [Google Scholar] [CrossRef] [PubMed]
- Ponce, D.; de Andrade, L.G.M.; Claure-Del Granado, R.; Ferreiro-Fuentes, A.; Lombardi, R. Development of a prediction score for in-hospital mortality in COVID-19 patients with acute kidney injury: A machine learning approach. Sci. Rep. 2021, 11, 24439. [Google Scholar] [CrossRef]
- Lundberg, S.M.; Lee, S.-I. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017; pp. 4768–4777. [Google Scholar]
- Kor, C.T.; Li, Y.R.; Lin, P.R.; Lin, S.H.; Wang, B.Y.; Lin, C.H. Explainable Machine Learning Model for Predicting First-Time Acute Exacerbation in Patients with Chronic Obstructive Pulmonary Disease. J. Pers. Med. 2022, 12, 228. [Google Scholar] [CrossRef]
- Oh, T.R.; Song, S.H.; Choi, H.S.; Suh, S.H.; Kim, C.S.; Jung, J.Y.; Choi, K.H.; Oh, K.H.; Ma, S.K.; Bae, E.H.; et al. Predictive Model for High Coronary Artery Calcium Score in Young Patients with Non-Dialysis Chronic Kidney Disease. J. Pers. Med. 2021, 11, 1372. [Google Scholar] [CrossRef]
- Lu, C.; Song, J.; Li, H.; Yu, W.; Hao, Y.; Xu, K.; Xu, P. Predicting Venous Thrombosis in Osteoarthritis Using a Machine Learning Algorithm: A Population-Based Cohort Study. J. Pers. Med. 2022, 12, 114. [Google Scholar] [CrossRef]
- Viton, F.; Elbattah, M.; Guérin, J.L.; Dequen, G. Heatmaps for Visual Explainability of CNN-Based Predictions for Multivariate Time Series with Application to Healthcare. In Proceedings of the 2020 IEEE International Conference on Healthcare Informatics (ICHI), Oldenburg, Germany, 30 November–3 December 2020; pp. 1–8. [Google Scholar]
- Stenwig, E.; Salvi, G.; Rossi, P.S.; Skjærvold, N.K. Comparative analysis of explainable machine learning prediction models for hospital mortality. BMC Med. Res. Methodol. 2022, 22, 53. [Google Scholar] [CrossRef]
- Mao, I.-C.; Lin, P.-R.; Wu, S.-H.; Hsu, H.-H.; Hung, P.-S.; Kor, C.-T. First 24-Hour Potassium Concentration and Variability and Association with Mortality in Patients Requiring Continuous Renal Replacement Therapy in Intensive Care Units: A Hospital-Based Retrospective Cohort Study. J. Clin. Med. 2022, 11, 3383. [Google Scholar] [CrossRef]
- Sheng, S.; Zhang, Y.H.; Ma, H.K.; Huang, Y. Albumin levels predict mortality in sepsis patients with acute kidney injury undergoing continuous renal replacement therapy: A secondary analysis based on a retrospective cohort study. BMC Nephrol. 2022, 23, 52. [Google Scholar] [CrossRef]
- Schork, A.; Moll, K.; Haap, M.; Riessen, R.; Wagner, R. Course of lactate, pH and base excess for prediction of mortality in medical intensive care patients. PLoS ONE 2021, 16, e0261564. [Google Scholar] [CrossRef] [PubMed]
- Hu, C.; Hu, B.; Wang, J.; Peng, Z.; Kashani, K.B.; Li, J. Diabetes mellitus is associated with lower mortality in intensive care unit: A multicenter historical cohort study. Res. Sq. 2020. [Google Scholar] [CrossRef]
- Rajawat, M.S.; Rathore, S.; Choudhary, M. pH disorders and mortality in surgical intensive care unit patients. Int. Surg. J. 2016, 3, 905–907. [Google Scholar] [CrossRef] [Green Version]
Features | No in-Hospital Death | In-Hospital Death | p-Value | Train Dataset | Test Dataset | p-Value |
---|---|---|---|---|---|---|
Sample size | 908 | 2024 | 2345 | 587 | ||
Characteristics | ||||||
Age, yr | 66 (55–76) | 72 (60–81) | <0.001 | 70 (59–80) | 70 (57–81) | 0.768 |
BMI | 25.6 (22.5–29) | 24 (21.1–27.3) | <0.001 | 24.5 (21.4–27.9) | 24.7 (21.5–28.1) | 0.303 |
MDC | 5 (4–8) | 5 (4–7) | 0.979 | 5 (4–7) | 5 (4–8) | 0.198 |
APACHE II at admission | 26 (20–32) | 30 (24–37) | <0.001 | 29 (22–35) | 29 (22–35) | 0.571 |
Timing of initiated CRRT | ||||||
Early-strategy group § | 518 (57%) | 962 (47.5%) | <0.001 | 1200 (51.2%) | 280 (47.7%) | 0.132 |
Delayed Strategy | 390 (43%) | 1062 (52.5%) | 1145 (48.8%) | 307 (52.3%) | ||
Vital Sign at CRRT—no. (%) | ||||||
Systolic BP (mmHg) | 115.7 (105.4–130.3) | 108.2 (98.2–119.7) | <0.001 | 110.6 (100.3–122.8) | 109.6 (100.8–123) | 0.738 |
Diastolic BP (mmHg) | 60.5 (53.5–70) | 57.7 (50.3–65.5) | <0.001 | 58.6 (51.4–66.8) | 58.1 (50.7–66.4) | 0.480 |
Respiratory rate (/min) | 18.8 (16.3–21.4) | 20 (17–23.3) | <0.001 | 19.7 (16.7–22.7) | 19.7 (16.8–22.8) | 0.884 |
SPO2 | 97.8 (96–99.1) | 96.6 (93.5–98.6) | <0.001 | 97.1 (94.4–98.8) | 96.9 (94.4–98.7) | 0.302 |
Fluid balance before CRRT—ml/24 hr | 1663 (500–2992) | 2221 (1000–3750) | <0.001 | 2000 (817–3550) | 2000 (835–3429) | 0.942 |
Coexisting conditions—no. (%) | ||||||
Diabetes mellitus | 368 (40.5%) | 642 (31.7%) | <0.001 | 812 (34.6%) | 198 (33.7%) | 0.683 |
Multiple organ support before CRRT—no. (%) | ||||||
Invasive mechanical ventilation | 719 (79.2%) | 1832 (90.5%) | <0.001 | 2043 (87.1%) | 508 (86.5%) | 0.709 |
Vasopressors support with norepinephrine or epinephrine | 632 (69.6%) | 1792 (88.5%) | <0.001 | 1931 (82.3%) | 493 (84%) | 0.348 |
Vasopressin | 178 (19.6%) | 843 (41.7%) | <0.001 | 801 (34.2%) | 220 (37.5%) | 0.131 |
Medication use before CRRT—no. (%) | ||||||
Corticosteroids | 429 (47.2%) | 1247 (61.6%) | <0.001 | 1347 (57.4%) | 329 (56%) | 0.542 |
Parenteral Nutrition | 678 (74.7%) | 1769 (87.4%) | <0.001 | 1959 (83.5%) | 488 (83.1%) | 0.813 |
Antibiotics | 828 (91.2%) | 1948 (96.2%) | <0.001 | 2219 (94.6%) | 557 (94.9%) | 0.800 |
Furosemide | 438 (48.2%) | 1104 (54.5%) | 0.002 | 1218 (51.9%) | 324 (55.2%) | 0.158 |
Laboratory data before CRRT | ||||||
Serum creatinine (mg/dL) | 2.6 (1.4–5.2) | 2 (1.2–3.6) | <0.001 | 2.1 (1.3–4) | 2.2 (1.3–4.2) | 0.640 |
Serum potassium (mmol/L) | 3.9 (3.4–4.4) | 4 (3.4–4.8) | <0.001 | 3.9 (3.4–4.6) | 4 (3.4–4.7) | 0.699 |
Serum albumin (g/dL) | 2.6 (2.1–3.1) | 2.2 (1.7–2.7) | <0.001 | 2.3 (1.8–2.8) | 2.3 (1.8–2.8) | 0.448 |
Lactate, mmol/L | 2.9 (1.4–6.6) | 4.8 (2.2–10) | <0.001 | 4.1 (1.9–8.9) | 4 (1.9–9.5) | 0.906 |
Platelet count | 126 (75–199) | 96 (53–165) | <0.001 | 106 (59–175) | 105 (56–177) | 0.930 |
pH | 7.4 (7.3–7.4) | 7.3 (7.2–7.4) | <0.001 | 7.3 (7.2–7.4) | 7.3 (7.2–7.4) | 0.931 |
Serum sodium (mmol/L) | 138 (134–141) | 139 (134–144) | <0.001 | 138 (134–143) | 138 (134–143) | 0.269 |
RDW | 15.4 (14.3–17) | 16.1 (14.8–18.3) | <0.001 | 15.8 (14.6–17.8) | 16.1 (14.6–18.6) | 0.039 |
Mg | 2 (1.8–2.3) | 2.1 (1.8–2.4) | <0.001 | 2.1 (1.8–2.4) | 2.1 (1.8–2.5) | 0.090 |
INR | 1.2 (1.1–1.4) | 1.3 (1.1–1.7) | <0.001 | 1.3 (1.1–1.6) | 1.3 (1.1–1.7) | 0.785 |
APTT | 36.5 (30.5–54.1) | 41 (32.8–69.6) | <0.001 | 39.2 (31.8–61.5) | 40.8 (32.4–69.7) | 0.027 |
O2 Saturation | 98.8 (96.6–99.8) | 98 (95.1–99.5) | <0.001 | 98.3 (95.6–99.7) | 98.3 (95.7–99.6) | 0.955 |
Outcome | ||||||
In-hospital mortality | 0 (0%) | 2024 (100%) | -- | 1625 (69.0%) | 399 (67.0%) | 0.535 |
28 days mortality | 2 (0.2%) | 1733 (85.6%) | <0.001 | 1385 (59.1%) | 350 (59.6%) | 0.804 |
90 days mortality | 20 (2.2%) | 1984 (98%) | <0.001 | 1607 (68.5%) | 397 (67.6%) | 0.676 |
Model | AUC | Threshold | Sensitivity | Specificity | PPV | NPV | F1 Score | Accuracy |
---|---|---|---|---|---|---|---|---|
Support Vector Machine (SVM) with radial kernel | 0.7500 | 0.6749 | 62.66% | 78.72% | 86.21% | 49.83% | 72.57% | 67.80% |
Support Vector Machine (SVM) with polynomial kernel | 0.7836 | 0.6588 | 67.67% | 77.13% | 86.26% | 52.92% | 75.84% | 73.59% |
Support Vector Machine (SVM) with Sigmoid kernel | 0.7563 | 0.6897 | 71.43% | 75.53% | 86.10% | 55.47% | 78.08% | 72.74% |
Random Forest (RF) | 0.8161 | 0.6587 | 74.69% | 75.00% | 86.38% | 58.26% | 80.11% | 74.79% |
Extreme Gradient Boosting (XGBoost) | 0.8064 | 0.7234 | 73.43% | 80.32% | 88.79% | 58.75% | 80.38% | 75.64% |
Gradient boosted machines (GBMs) | 0.8227 | 0.7216 | 74.19% | 78.72% | 88.10% | 58.96% | 80.55% | 75.64% |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Hung, P.-S.; Lin, P.-R.; Hsu, H.-H.; Huang, Y.-C.; Wu, S.-H.; Kor, C.-T. Explainable Machine Learning-Based Risk Prediction Model for In-Hospital Mortality after Continuous Renal Replacement Therapy Initiation. Diagnostics 2022, 12, 1496. https://doi.org/10.3390/diagnostics12061496
Hung P-S, Lin P-R, Hsu H-H, Huang Y-C, Wu S-H, Kor C-T. Explainable Machine Learning-Based Risk Prediction Model for In-Hospital Mortality after Continuous Renal Replacement Therapy Initiation. Diagnostics. 2022; 12(6):1496. https://doi.org/10.3390/diagnostics12061496
Chicago/Turabian StyleHung, Pei-Shan, Pei-Ru Lin, Hsin-Hui Hsu, Yi-Chen Huang, Shin-Hwar Wu, and Chew-Teng Kor. 2022. "Explainable Machine Learning-Based Risk Prediction Model for In-Hospital Mortality after Continuous Renal Replacement Therapy Initiation" Diagnostics 12, no. 6: 1496. https://doi.org/10.3390/diagnostics12061496