Prognostic Performance of Existing Scoring Systems among Critically Ill Patients Requiring Continuous Renal Replacement Therapy: An Observational Study

Background: Among critical patients, few studies have evaluated the discrimination of current illness scoring systems in predicting outcomes after continuous renal replacement therapy (CRRT) initiation. Methods: Patients receiving CRRT in the ICU between 2005 and 2018 from the Chang Gung Research Database were extracted. All the components of the Acute Physiology Assessment and Chronic Health Evaluation (APACHE) III, Sequential Organ Failure Assessment (SOFA), qSOFA, and MOSAIC scoring systems on days 1, 3, and 7 of CRRT were recorded. Patients older than 80 years were identified and analyzed separately. Results: We identified 3370 adult patients for analysis. The discrimination ability of the scoring systems was acceptable at day 7 after CRRT initiation, including SOFA (area under the receiver operating characteristic curve, 74.1% (95% confidence interval, 71.7–76.5%)), APACHEIII (74.7% (72.3–77.1%)), and MOSAIC (71.3% (68.8%–73.9%)). These systems were not ideal on days 1 and 3, and that of qSOFA was poor at any time point. The discrimination performance was slightly better among patients ≥80 years. Conclusions: APACHE III, MOSAIC, and SOFA can be intensivists and families’ reference to make their decision of withdrawing or withholding CRRT after a short period of treatment, especially in adults ≥80 years old.


Introduction
Acute kidney injury (AKI) is a common but serious complication in critically ill patients requiring treatment in intensive care units (ICUs) [1,2]. AKI is a part of multiple organ failure syndrome and results in several challenging conditions, such as hyperkalemia, fluid accumulation, and severe acidosis; accordingly, the prognosis of patients with AKI has remained poor, despite advances in ICU care [3,4]. Among these AKI populations, administration of continuous renal replacement therapy (CRRT), because of its gentle but continuous fluid clearance and better hemodynamic tolerance, caused renal function to be preserved, particularly in those with unstable hemodynamic status and poor fluid control [5,6]. Along with the increasing age and severity of ICU populations, the incidence of AKI requiring CRRT has been increasing considerably [7,8]. As CRRT-associated resources are not unlimited, the incremental need of CRRT imposes a burden on the ICU system [9]. Moreover, even with appropriate CRRT, the mortality rate of these patients remains very high, ranging between 40% and 80% [10,11]. Thus, for better allocation of CRRT-associated resources and modification of treatment strategies, a reliable model to assess conditions and predict outcomes of severely ill patients with CRRT requiring AKI is urgently required, 2 of 12 especially in older people (such as those older than 80 years). Considering the poor prognosis and shorter life expectancy, conservative treatment with less invasive procedures is always a more favorable option for this population [12]. A precise assessment model may therefore help patients and families in decision making regarding choosing conservative treatment or switching to it after CRRT initiation.
Several popular illness-scoring systems exist for identifying high-risk ICU patients independent of their underlying disease processes, including the Acute Physiology Assessment and Chronic Health Evaluation system (APACHE), the Sequential Organ Failure Assessment (SOFA), and quick SOFA (qSOFA) systems. The APACHE system is a comprehensive assessment of different physiologic indexes, but its complex algorithms limit its application in daily practice [13,14]. The SOFA score system was initially introduced to assess the severity of organ failure among patients with sepsis and was then expanded to other critical diseases [15][16][17]. The qSOFA system was introduced to be an easy predictive tool for patients with suspected infection [18]. In addition, Kim et al., in their study involving an 828-patient cohort, developed a new system, the Mortality Scoring system for AKI with CRRT (MOSAIC), to predict mortality in patients undergoing CRRT [19]. Although these predictive systems have been validated in several ICU populations, they have not been effectively tested in people with AKI requiring CRRT or have only been tested in small cohorts.
Therefore, in this study, we used a large comprehensive medical database, the Chang Gung Research Dataset (CGRD), to assess the performance of several scores on predicting mortality in patients with AKI requiring CRRT and in patients older than 80 years with AKI requiring CRRT.

Data Source
The CGRD is based on medical information from Taiwan's largest hospital network, Chang Gung Memorial Hospital system. It includes the detailed laboratory and physical information of more than 5000 ICU patients receiving CRRT between 2005 and 2018. The Chang Gung Memorial Hospital system comprises four tertiary medical centers and three teaching hospitals and covers approximately 10% of the annual medical services of Taiwan, including approximately 20,000 ICU patients per year [20,21]. The CGRD is a deidentified dataset based on the electronic medical records of the Chang Gung Memorial Hospital system. It contains comprehensive medical records, including ICU admission, inpatient orders, medication prescriptions, procedure interventions, laboratory data, and examination reports, thus making the database suitable for assessing the application of illness predictive systems. Moreover, any information that can identify a specific person in the CGRD is scrambled before releasing the data for research purposes, and de-identified consistent data encryption is used to link medical information. Thus, this study was approved with a waiver of the need for consent from the Institutional Review Board of Chang Gung Medical Foundation (approval number: 201900835B0).

Study Population
All ICU admissions with CRRT between 2005 and 2018 in CGRD were identified. The first episode of admission was considered an index admission if a patient experienced two or more ICU admissions during the study period. Patients who received RRT or dialysis prior to the index admission were excluded. Patients who were younger than 20 years old or had missing demographic data (age and sex) were also excluded. In addition, a subgroup of patients older than 80 years was identified to better understand the application of the illness predictive systems in this age group ( Figure 1). or had missing demographic data (age and sex) were also excluded. In addition, a subgroup of patients older than 80 years was identified to better understand the application of the illness predictive systems in this age group (Figure 1).

Clinical Parameters and Outcomes
All the components of APACHE III, SOFA, qSOFA, and MOSAIC systems on days 1, 3, and 7 of CRRT initiation were retrospectively collected from the CGRD. While these scores were generated according to the collected information, the missing variables at day 1 of CRRT were imputed as normal as the default [22,23]. Consequently, the missing variables at days 3 and 7 of CRRT were imputed as the last available data or as normal if no data were available. The baseline comorbidities were detected based on two or more outpatient diagnoses prior to the index admission. Demographic data, laboratory data, and use of mechanical ventilators or inotropic agents were recorded according to medical records on the day of CRRT initiation.
The outcomes of interest were 3-day and 7-day mortality after CRRT initiation, ICU mortality, and in-hospital mortality. The length of CRRT, ICU stay, and hospitalization were also measured from the day of CRRT initiation until the end of CRRT, transfer, discharge, or mortality.

Statistical Analysis
The characteristics of the patients who died and who survived during the index hospitalization were compared using a chi-square test for categorical variables or an independent-samples t test for continuous variables. The ability of the existing systems to predict mortality was assessed using the area under the receiver operating characteristic curve (AUC), with standard error calculated using DeLong's nonparametric method. Performance was considered acceptable when the AUC was ≥70% [24]. The optimal cutoffs were further determined using the Youden index when AUC was >70%. A two-sided p value of <0.05 was considered statistically significant. Statistical analyses were performed using MedCalc 19 (MedCalc Software, Ostend, Belgium).

Clinical Parameters and Outcomes
All the components of APACHE III, SOFA, qSOFA, and MOSAIC systems on days 1, 3, and 7 of CRRT initiation were retrospectively collected from the CGRD. While these scores were generated according to the collected information, the missing variables at day 1 of CRRT were imputed as normal as the default [22,23]. Consequently, the missing variables at days 3 and 7 of CRRT were imputed as the last available data or as normal if no data were available. The baseline comorbidities were detected based on two or more outpatient diagnoses prior to the index admission. Demographic data, laboratory data, and use of mechanical ventilators or inotropic agents were recorded according to medical records on the day of CRRT initiation.
The outcomes of interest were 3-day and 7-day mortality after CRRT initiation, ICU mortality, and in-hospital mortality. The length of CRRT, ICU stay, and hospitalization were also measured from the day of CRRT initiation until the end of CRRT, transfer, discharge, or mortality.

Statistical Analysis
The characteristics of the patients who died and who survived during the index hospitalization were compared using a chi-square test for categorical variables or an independent-samples t test for continuous variables. The ability of the existing systems to predict mortality was assessed using the area under the receiver operating characteristic curve (AUC), with standard error calculated using DeLong's nonparametric method. Performance was considered acceptable when the AUC was ≥70% [24]. The optimal cutoffs were further determined using the Youden index when AUC was >70%. A twosided p value of <0.05 was considered statistically significant. Statistical analyses were performed using MedCalc 19 (MedCalc Software, Ostend, Belgium).

Prediction of 3-Day and 7-Day Mortality after CRRT
All four systems exhibited poor performance in predicting 3-day mortality on the first day of CRRT initiation, with AUCs near or below 60% (Supplemental Table S1). The results were similar when the cohort was restricted to patients older than 80 years. Table 2 presents the results of predicting 7-day mortality after CRRT by the scoring systems at days 1 and 3 of CRRT initiation. The systems exhibited poor performance in predicting 3-day mortality on day 1, but the performance improved on day 3, especially for APACHE III in both the whole cohort (AUC 76.1%, 95% confidence interval (CI), 73.6-78.5%) and in patients older than 80 years (AUC 78.9%; 95% CI, 73.1-84.8%). The optimal cutoffs and the corresponding sensitivity and specificity are provided in Supplemental Table S2. CRRT, continuous renal replacement therapy; AUC, area under the curve; CI, confidence interval; SOFA, Sequential Organ Failure Assessment; qSOFA, quick Sequential Organ Failure Assessment; APACHE, Acute Physiology and Chronic Health Evaluation; MOSAIC, Mortality Scoring system for AKI with CRRT. a : the missing components of the score system were imputed by a score of 0; b : the missing components of the score system were imputed using the score at day 1 at CRRT; otherwise, they were imputed by a score of 0. Table 3 displays the results of predicting ICU mortality on days 1, 3, and 7 of CRRT initiation. No scoring systems had satisfactory performance on days 1 and 3 after CRRT initiation, with all the AUCs being less than 70%. However, the performance of predicting ICU mortality improved on day 7 for SOFA (AUC, 74.1%, 95% CI, 71.7-76.5%), APACHE III (AUC, 74.7%, 95% CI, 72.3-77.1%), and MOSAIC (AUC, 71.3%, 95% CI, 68.8-73.9%), whereas that of qSOFA remained unsatisfactory, with AUCs <70% in the whole population or in patients older than 80 years (Table 3). CRRT, continuous renal replacement therapy; AUC, area under the curve; CI, confidence interval; SOFA, Sequential Organ Failure Assessment; qSOFA, quick Sequential Organ Failure Assessment; APACHE, Acute Physiology and Chronic Health Evaluation; MOSAIC, Mortality Scoring system for AKI with CRRT. a : the missing components of the score system were imputed by a score of 0; b : the missing components of the score system were imputed using the score at day 1 at CRRT; otherwise, they were imputed by a score of 0; c : the missing components of the score system were imputed using the score at day 3 at CRRT; otherwise, they were imputed by a score of 0. Table 4 presents the results of predicting in-hospital mortality on days 1, 3, and 7 of CRRT initiation. Similar to the results of ICU mortality, the performance was acceptable on day 7 for SOFA (AUC, 71.8%, 95% CI, 69.3-74.3%) and APACHE III (AUC, 73.5%, 95% CI, 71.0-76.0%). For MOSAIC, the performance on day 7 was poor for the whole cohort but was satisfactory when only patients older than 80 years were considered (AUC, 74.7%, 95% CI, 67.9-81.4%) ( Table 4). CRRT, continuous renal replacement therapy; AUC, area under the curve; CI, confidence interval; SOFA, Sequential Organ Failure Assessment; qSOFA, quick Sequential Organ Failure Assessment; APACHE, Acute Physiology and Chronic Health Evaluation; MOSAIC, Mortality Scoring system for AKI with CRRT. a : the missing components of the score system were imputed by a score of 0; b : the missing components of the score system were imputed using the score at day 1 at CRRT; otherwise, they were imputed by a score of 0; c : the missing components of the score system were imputed using the score at day 3 at CRRT; otherwise, they were imputed by a score of 0.

Prediction of ICU Mortality and In-Hospital Mortality
The optimal cutoffs and the corresponding sensitivity and specificity are provided in Supplemental Table S2. The distributions of SOFA, MOSAIC, and APACHE III scores on day 7 after CRRT initiation and their relationship with ICU mortality and in-hospital mortality in the whole cohort (

Discussion
CRRT has become a crucial treatment in ICU. In this study, around 2.1% of to patients had received CRRT before, which was similar to previous research stu France (4.3%) [25] and Korea (3.3%) [8]. Although illness scoring systems have bee for assessing severity and predicting mortality among ICU populations [16,23], few ies have evaluated their use in allocating limited CRRT resources and helping f opting for or switching to conservative treatment before or during CRRT if poor pro is predicted, which is especially crucial in older people. The current large observ

Discussion
CRRT has become a crucial treatment in ICU. In this study, around 2.1% of total ICU patients had received CRRT before, which was similar to previous research studies in France (4.3%) [25] and Korea (3.3%) [8]. Although illness scoring systems have been used for assessing severity and predicting mortality among ICU populations [16,23], few studies have evaluated their use in allocating limited CRRT resources and helping families opting for or switching to conservative treatment before or during CRRT if poor prognosis is predicted, which is especially crucial in older people. The current large observational study evaluated the prognostic performance of four current scoring systems in predicting outcomes after CRRT initiation and demonstrated that SOFA, APACHE III, and MOSAIC can satisfactorily predict ICU or in-hospital mortality only on day 7, with AUCs for ICU mortality being 74.1, 74.1, and 71.3, respectively, and optimal cutoffs being 14, 82, and 10, respectively. For short-term outcomes (7-day mortality), only APACHE III on day 3 was found to have a good discrimination (AUC = 76.1), with the optimal cutoff being 107. Unlike previous studies assessing the predictive accuracy of scoring systems on the first day of ICU admission [26] or diagnosis of sepsis [27], our data indicated that the use of these systems on the first day of CRRT initiation was unsatisfactory for predicting short-term or long-term outcomes. Thus, we speculate that the key factor influencing the survival of these critically ill patients with AKI depends on whether their physiological function improves after CRRT initiation, such as hyperkalemia-induced arrhythmia and fluid overload-induced respiratory failure, or whether renal failure is a part of progressive multiple organ dysfunction. It is challenging to determine these two situations by using only scoring systems. In other words, SOFA, APACHE III, and MOSAIC are more suitable for assessing outcomes after a few days of CRRT initiation rather than for deciding whether to initiate CRRT. Among the three models, APACHE III was noted to be the best at predicting short-term; however, its complex algorithms and non-superior prediction of ICU mortality or in-hospital mortality make it less attractive in daily ICU practice. MOSAIC, which is a combination of part of SOFA and APACHE II [19], was proven to have a noninferior prognostic performance compared with APACHE III or SOFA. However, the lack of other proven utilities limits its use in critical care. Simplicity and ease of use are the most crucial advantages of the SOFA score [28,29], and its discrimination is equivalent to other complex models. In addition to the CRRT population, SOFA's good prognostic performance has been proven in several scenarios of ICU care, such as sepsis [27], or cardiac intensive care units [23], which makes it more attractive for use in daily care. The prognostic performance of qSOFA in predicting mortality amount CRRT patients, however, was found to be unsatisfactory in all analyses in the present study, similar to previous research [16]; this indicates that qSOFA might not be appropriate for predicting outcomes in critically ill patients.
In developed countries, patients older than 80 years may be the fastest expanding subgroup in ICU populations [30]. However, these patients have many comorbidities, exhibit a decline in organ function, and are frail, leading to extremely high in-hospital mortality and after-discharge mortality among these populations compared with their younger counterparts [31,32]. Although old age is an independent risk factor for AKI development among the ICU population, no large-scale study has evaluated outcome prediction in patients older than 80 years requiring CRRT. In the present study, we found that SOFA, APACHE III, and MOSAIC had better prediction performance in patients older than 80 years than in the whole cohort, with AUCs for ICU mortality being 75.4, 72.6, and 74.2, respectively, and optimal cutoffs being 13, 85, and 13, respectively. The day 3 APACHE III and SOFA models had acceptable prediction performance for ICU mortality and in-hospital mortality. Notably, in patients older than 80 years, the day 7 SOFA and MOSAIC had better prediction performance of ICU mortality and in-hospital mortality than APACHE III. This might be attributed to the lower rates of residual vital organ function in this patient subgroup and the lower probability of organ function recovery after injury. Thus, among patients older than 80 years, the index of organ failure, such as SOFA and MOSAIC, may surpass the physical index APACHE III in predicting ICU or in-hospital mortality. Further research is warranted to validate these findings. Thus, these scoring systems may help patients older than 80 years, their families, and intensivists make decisions to withdraw CRRT if the prognosis is determined to be poor, which may save medical costs, and notably, reduce suffering.
This study had the strength of a large sample size from multiple centers. Moreover, this was the first study to evaluate the prediction of outcomes among critically ill patients older than 80 years requiring CRRT. Nevertheless, several limitations of this study should be acknowledged. First, detailed information on CRRT prescriptions, such as dialysate dose, blood flow rate, and anticoagulation, was not available in the CGRD. Second, the observational design precluded the determination of causal relationships or analysis of long-term outcomes after discharge. Third, the missing variables were imputed as normal as the default or imputed as the last available data since CRRT initiation in this study, which may lead to an underestimation of mortality risk. In addition, imputing missing variables as the last available data may lead to discrepancies in the time the data were obtained, which may further reduce the accuracy of illness score systems. Fourth, in this study, real serum creatinine was used to calculate illness scores instead of setting these renal components of scoring systems to maximum after initiation of CRRT, which could simplify the calculations for intensivists but would inevitably underestimate the entire score values.

Conclusions
We comprehensively reviewed the application of four popular illness scoring systems in predicting outcomes of ICU patients with AKI requiring CRRT by using the largest cohort thus far. We demonstrated that APACHE III, MOSAIC, and SOFA scores 7 days after CRRT initiation favorably discriminated ICU mortality and in-hospital mortality. In patients older than 80 years, the prognostic performances of these scoring systems were even better. Although these scoring systems cannot be used to decide whether to initiate CRRT or not, they can be provided for intensivists and families' reference to make their decisions on withdrawing or withholding the CRRT after a short period of treatment, which could help shorten the suffering of patients with poor prognosis and allocate more resources to patients to whom the CRRT may be more beneficial.
Supplementary Materials: The following are available online at https://www.mdpi.com/article/10 .3390/jcm10194592/s1, Table S1: Performance of the scoring systems on day 1 of CRRT initiation in predicting 3-day mortality in the entire cohort and in patients older than 80 years, Table S2: The optimal cutoffs and the corresponding sensitivity and specificity in the analysis with an AUC ≥70%.  Informed Consent Statement: Patient consent was waived due to any information that can identify a specific person in the CGRD being scrambled before releasing data for research purposes, and de-identified consistent data encryption being used to link medical information.
Data Availability Statement: Not applicable.