Validation of a Novel Predictive Algorithm for Kidney Failure in Patients Suffering from Chronic Kidney Disease: The Prognostic Reasoning System for Chronic Kidney Disease (PROGRES-CKD)

Current equation-based risk stratification algorithms for kidney failure (KF) may have limited applicability in real world settings, where missing information may impede their computation for a large share of patients, hampering one from taking full advantage of the wealth of information collected in electronic health records. To overcome such limitations, we trained and validated the Prognostic Reasoning System for Chronic Kidney Disease (PROGRES-CKD), a novel algorithm predicting end-stage kidney disease (ESKD). PROGRES-CKD is a naïve Bayes classifier predicting ESKD onset within 6 and 24 months in adult, stage 3-to-5 CKD patients. PROGRES-CKD trained on 17,775 CKD patients treated in the Fresenius Medical Care (FMC) NephroCare network. The algorithm was validated in a second independent FMC cohort (n = 6760) and in the German Chronic Kidney Disease (GCKD) study cohort (n = 4058). We contrasted PROGRES-CKD accuracy against the performance of the Kidney Failure Risk Equation (KFRE). Discrimination accuracy in the validation cohorts was excellent for both short-term (stage 4–5 CKD, FMC: AUC = 0.90, 95%CI 0.88–0.91; GCKD: AUC = 0.91, 95% CI 0.86–0.97) and long-term (stage 3–5 CKD, FMC: AUC = 0.85, 95%CI 0.83–0.88; GCKD: AUC = 0.85, 95%CI 0.83–0.88) forecasting horizons. The performance of PROGRES-CKD was non-inferior to KFRE for the 24-month horizon and proved more accurate for the 6-month horizon forecast in both validation cohorts. In the real world setting captured in the FMC validation cohort, PROGRES-CKD was computable for all patients, whereas KFRE could be computed for complete cases only (i.e., 30% and 16% of the cohort in 6- and 24-month horizons). PROGRES-CKD accurately predicts KF onset among CKD patients. Contrary to equation-based scores, PROGRES-CKD extends to patients with incomplete data and allows explicit assessment of prediction robustness in case of missing values. PROGRES-CKD may efficiently assist physicians’ prognostic reasoning in real-life applications.


Introduction
Multiple behavioral and pharmacological interventions have proven effective in reducing the burden of risk factors for chronic kidney disease (CKD) progression [1][2][3][4]. Furthermore, timely transition management (i.e., vascular access creation and training) for patients needing Kidney Replacement Therapy (KRT) is associated with prolonged survival and reduced complication rates once on dialysis, while delayed referrals are associated with increased morbidity, mortality, and healthcare costs [5], as well as worse patient quality of life [6]. Therefore, early identification of high risk patients is an essential prerequisite of personalized clinical decision making [7][8][9].
Several prediction models were developed to assist physicians in forecasting CKD progression [10]. However, most of them have not been consistently implemented in clinical practice [9,11,12]. Indeed, the majority of published risk scores lack external validation [11,13,14], leading to suboptimal discrimination in external populations [12] and limited generalizability to clinical settings [11]. One prominent exception is represented by the Kidney Failure Risk Equations (KFREs) developed by Tangri and colleagues [15], which showed stable discrimination in different validation studies [16][17][18]. However, KFREs do not provide short-term forecasts, are not calculable for patients with incomplete data, and need re-calibration when applied to CKD populations with risk factor distributions departing from those of the original derivation dataset.
To overcome such limitations, we developed the Prognostic Reasoning System for Chronic Kidney Disease (PROGRES-CKD), a risk score application for adult patients suffering from CKD stages 3-5. PROGRES-CKD is based on a naïve Bayes Classifier (NBC) algorithm and it was trained on a large-multinational clinical dataset, reflecting real-world clinical practice. The application includes PROGRES-CKD-6 for 6-month forecasting and PROGRES-CKD-24 for 24-month forecasting.
In the present study, we reported the training and validation of both PROGRES-CKD-6 and PROGRES-CKD-24 in two independent samples of CKD patients: the FMC Nephro-Care cohort (European Clinical Database, EuCliD ® , [19,20]) and the German Chronic Kidney Disease (GCKD) study cohort [21]. Moreover, we compared the PROGRES-CKD discrimination accuracy and suitability for clinical practice against the KFREs equations.

Materials and Methods
In reporting PROGRES-CKD training and validation studies we adhered to the Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) statement [22] and to the Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research [23].

Description of Naïve Bayes Classifiers
All PROGRES-CKD models are NBCs. NBCs are probabilistic models based on application of the Bayes' theorem. The basic assumption of NBCs is conditional independence of predictors given the outcome. NBCs are represented through directed acyclic graphs ( Figure 1). NBCs have been previously used in medical applications for diagnostic and prognostic reasoning in several therapeutic areas [24,25]. In fact, once derived and validated, NBCs generate metrics informing medical prognostic reasoning. First, they generate a risk score representing the expected incidence of a disease/event given a vector of known patient characteristics. Furthermore, NBCs can be used to generate value of information (VOI) statistics and impact metrics. VOI statistics represent the reduction in uncertainty (i.e., entropy) in the outcome variable that would be obtained had the value of missing variables been observed instead [26]. Therefore, it can be used to prioritize additional diagnostic testing or biomarker assays for patients with incomplete medical records. Third, NBCs can provide impact metrics (i.e., Normalized Likelihood (NL) [27]) for each observed variable. Impact metrics can be interpreted as the magnitude of association of different subsets of evidence on the outcome variable. derived and validated, NBCs generate metrics informing medical prognostic reaso First, they generate a risk score representing the expected incidence of a disease/e given a vector of known patient characteristics. Furthermore, NBCs can be use generate value of information (VOI) statistics and impact metrics. VOI statistics repr the reduction in uncertainty (i.e., entropy) in the outcome variable that would be obta had the value of missing variables been observed instead [26]. Therefore, it can be us prioritize additional diagnostic testing or biomarker assays for patients with incom medical records. Third, NBCs can provide impact metrics (i.e., Normalized Likeli (NL) [27]) for each observed variable. Impact metrics can be interpreted as the magn of association of different subsets of evidence on the outcome variable.

PROGRES-CKD Training
In this application of NBCs, we aimed at developing a model to predict the risk of KRT initiation within 6 and 24 months. The risk score is anchored at 0.00 = no risk at all to 1.00 = certainty of failure within the prediction horizons.
We derived model weights for the PROGRES-CKD by a data-driven algorithm, exploiting the wealth of information collected in the European Clinical Database (EuCliD ® , Fresenius Medical Care Deutschland GmbH, Bad Homburg, Germany), a large, multinational, database of CKD patients. All nephrology clinics belonging to the Fresenius Medical Care (FMC) NephroCare network confer data collected for healthcare practice into this cen-tralized data-repository. EuCliD ® is a fully codified database recording clinical, laboratory, socio-demographic, treatment and prescription data for each medical encounter [19,20]. Information is collected by healthcare professionals either manually or by means of interfaces to existing local data managing systems.
All non-dialysis dependent, stage 3-5 CKD patients receiving care in outpatient renal clinics belonging to the NephroCare network from 2017 to 2018 were screened for eligibility. We enrolled only patients who received at least one outpatient visit and one serum creatinine (s-cr) assessment. The endpoints of interest were KRT initiation within 6 and 24 months. We excluded patients dying before reaching the endpoint or before the end-of-follow-up (i.e., 6 or 24 months, depending on endpoint of interest). Overall, 22,535 subjects met the inclusion criteria. This initial dataset was randomly partitioned into 2 analytical samples: development (70%, n = 17,775), and validation (30%, n = 6760). The derivation of NBC weights was obtained with Hugin 8.5.

Endpoint Definition
The primary endpoint was KRT initiation within 6 and 24 months. Outcome definition does not include episodes of dialysis treatment for acute and transient kidney derangement.
We defined patients as "lost to follow" when no additional s-cr assessments after end of follow-up date and no dialysis-dependence onset notes were present in the clinical records.

Input Variables
A list of all the variables included in the final model is provided in Table 1. The final model for the 6-month forecast incorporates 28 independent variables, while the model for the 24-month forecast includes 34 variables. We assessed demographic, anthropometric, and lifestyle variables at index visit; blood biomarkers were collected and averaged over 12 months before index date (i.e., during the ascertainment period); their slope (i.e., change rate) was likewise calculated. Lifetime occurrence of comorbidities was evaluated by abstracting ICD10 codes [28] from outpatient medical records (Supplementary Material). Finally, etiologies of kidney disease were also noted.

Design and Setting of PROGRES-CKD Validation Studies
For the validation study we randomly selected one visit from patients' histories (index date) before occurrence of study endpoint. All information collected before the index data was used as an input variable for the model. Patients dying before reaching the endpoint or before the end-of-follow-up (i.e., 6 or 24 months, depending on endpoint of interest) were excluded.
Based on this general design setting, we validated PROGRES-CKD models in two independent cohorts.

Study A
The first validation study was performed in the testing cohort derived from 30% partitioning of the clinical data abstracted from the FMC NephroCare cohort.

Study B
A second analysis evaluated PROGRES-CKD performance using data from the German CKD study [21]. Briefly, the GCKD study is an ongoing prospective observational national study that recruited 5217 patients with CKD of various etiologies. The enrolment period started in July 2011 and ended in 2012. Patient recruitment and follow-up is orga-nized through a network of academic nephrology centers collaborating with practicing nephrologists throughout Germany. The main study endpoints were mortality, decline in kidney function, and cardiovascular events. At the time of recruitment, patients were under nephrological care and showed either eGFR of 30-60 mL/min/1.73 m 2 or overt urin protein in the presence of an eGFR > 60 mL/min/1.73 m 2 . In our validation analysis, only patients subjected to serum creatinine evaluation at baseline and followed for at least 2 years were considered.

Study C
We conducted an impact study assessing concordance of nephrologists' and PROGRES-CKD-24 ratings of risk. Four experts were asked to forecast KRT initiation risk for 78 CKD patients based on their demographic, anthropometric, and clinical data. These patients were randomly selected from the FMC NephroCare cohort and had complete clinical history up to 24 months after the index date. Information related to all input variables used by the model were extracted from existing clinical records. Information extracts for each patient were collected in real-world clinical practice by physicians during outpatient visits. Doctors were asked to rate KRT risk on a 10-point rating scale anchored at 1 (risk is negligible, almost no patient with these characteristics would require RRT within 2 years), 5 (about 50% of patients with these characteristics would require RRT within 2 years) and 10 (almost 100% patients with these characteristics would require RRT within 2 years). Risk ratings provided by the physicians were then compared to scores obtained from PROGRES-CKD-24 for the same patients. Comparative analysis included accuracy, sensitivity, and specificity based on score cut-off that maximized Youden's Index. Thereafter, we investigated the potential impact of using risk scores provided by either experts or PROGRES-CKD-24 in referring patterns to intensified healthcare prevention programs aimed at delaying CKD progression. We simulated the use of risk estimates on a large, hypothetical CKD population of stage 3-5 CKD patients (n = 10,000), assuming an ESRD incidence within 24 months of 4.6% (i.e., n = 460 expected ESKD cases) and an intervention effect size of 1.5 (i.e., patients in the standard of care arm would face 50% higher risk of ESKD compared to those allocated in the intensified healthcare program). The intervention effect size was estimated based on expert opinion and several intensified intervention programs reported in diabetic and non-diabetic CKD [30][31][32].

Statistical Analysis
We computed the cumulative incidence and the incidence density of KRT initiation events in the study population and their 95% confidence intervals based on the Poisson distribution.
Since PROGRES-CKD models are NBCs, no data manipulation was required to explicitly handle missing variables.
Model performance was evaluated by concordance statistic and calibration charts in the FMC NephroCare and the GCKD cohorts. Discrimination was quantified by calculating the area under the receiver operating characteristic curve (ROC AUC) [33]. An AUC >0.70 was considered acceptable. Calibration was visually inspected by plotting observed outcome incidence by quintiles of the risk score [34].
A further analysis investigated non-inferiority (defined as ∆AUC < 0.05) of both PROGRES-CKD-6 and PROGRES-CKD-24 relative to the KFREs [15] calibrated for the European population [16]. Briefly, Tangri's models were developed using Cox proportional hazards regression methods in stage 3-5 CKD patients. In the present study, the following Tangri's equations were used: (1) 4 Variables (4VAR), includes Age, Gender, eGFR, and Albumin-Creatinine Ratio (ACR); (2) 6 Variables (6VAR), includes Age, Gender, eGFR, ACR, Diabetes, and Hypertension. We could not apply the 8 Variables (8VAR) equation given the lack of serum bicarbonate assessments in both study cohorts. Non-inferiority was assessed by checking whether a one-sided confidence interval of the AUC remained entirely above the non-inferiority threshold (0.05). In case non-inferiority was achieved, we evaluated superiority of PROGRES-CKD compared to benchmark models; superiority was set at ∆AUC ≥ 0.05. Given the sequential nature of testing in a fixed order method approach, type I error is not inflated by multiple testing. Superiority was tested with the DeLong non-parametric approach [35]. Statistical significance was claimed at α < 0.05.
For study C, the following accuracy parameters were considered: Sensitivity, Specificity, Positive Predictive Value (PPV), and False Omission Rate (FOR). We also calculated the number needed to treat (NNT) in order to avoid 1 KRT event as the reciprocal of the absolute risk difference between the hypothetical prevention program and standard of care for all patients: Model training was performed using Hugin Explorer. All analyses for the validation study were performed with SAS 9.4 ® . Table 2 reports baseline demographic and clinical data of the whole FMC NephroCare cohort. Among 22,535 non-dialysis-dependent stage 3-5 CKD patients, 18,504 and 9407 patients had 6 and 24 months of follow-up, respectively. KRT events were 801 within 6 months (8.66 events/100 person-year) and 1817 within 24 months (9.66 events/100 person-year). On the other hand, KRT events in the validation sample (derived from 30% partitioning of the whole FMC cohort) were 248 (2.24 events/100 person-year) and 537 (9.36 events/ 100 person-year) within 6 and 24 months, respectively. A second validation study was performed using data from the GCKD study. As shown in Table 2, a total of 4058 stage 3-5 CKD patients were included, of whom 3888 and 3687 subjects had 6 and 24 months of follow-up, respectively. RRT events were 11 within 6 months (0.5 events/100 person-year) and 80 (1.1 events/100 person-year) within 24 months.

Cohort Characteristics
Early CKD stages were predominantly represented in the GCKD study, whereas patients in stage 5 CKD were mostly enrolled in the FMC NephroCare cohort. Loss to follow-up within 6 months was 4031 (17.9%) and 170 (4.2%) participants, while loss to follow-up in 24 months was 13,128 (58.3%) and 371 (9.1%) participants in the FMC NephroCare and GCKD cohorts, respectively.
Calibration of predicted versus observed risk is represented in Figure 2.
Evaluation of ratios of observed risk across quintiles of predicted risk indicated that the model best discriminated low and high-risk patients compared to those classified in the central quintile or risk score distribution (Figure 3).  Evaluation of ratios of observed risk across quintiles of predicted risk indicated that the model best discriminated low and high-risk patients compared to those classified in the central quintile or risk score distribution (Figure 3).  Table 3 shows the comparison in discrimination accuracy between PROGRES-CKD and KFREs equations. Since KFREs equations are computable only for complete information cases, patients with missing data were listwise deleted from this analysis. Given the large amount of missing information for ACR, we converted timed proteinuria assays (proteinuria g/24 h) into ACR when available. The conversion was based on a published correspondence table (Supplementary Material).  Table 3 shows the comparison in discrimination accuracy between PROGRES-CKD and KFREs equations. Since KFREs equations are computable only for complete information cases, patients with missing data were listwise deleted from this analysis. Given the large amount of missing information for ACR, we converted timed proteinuria assays (proteinuria g/24 h) into ACR when available. The conversion was based on a published correspondence table (Supplementary Material). Table 3. Comparison between discrimination ability of (A) PROGRES-CKD-6 and (B) PROGRES-CKD-24 and that of Tangri's Kidney Failure Risk Equations (KFREs) in the FMC and the GCKD cohort. The two scores were computed considering only complete cases (column "Effective sample size"), while patients with missing data were not included in the analysis. Endpoint horizons: 6 months for PROGRES-CKD-6; 24 months for PROGRES-CKD-24. Imputation method: Listwise. Non-inferiority was defined as AUC < 0.05, while superiority was set at ∆AUC ≥ 0.05. * Delta AUC: AUC of Tangri's KFRE-AUC of PROGRES-CKD model. Based on the superiority test criteria, the discrimination accuracy of PROGRES-CKD-6 was greater than KFRE equations for short term RRT risk among stage 4-5 CKD patients (Table 3). PROGRES-CKD-24 discrimination was not inferior to that of the gold standard algorithms (Table 3).

Potential Impact Simulation
A potential impact study compared the risk of KRT estimated by nephrologists with those calculated by PROGRES-CKD-24 and investigated the potential incremental efficiency of using PROGRES-CKD compared to physicians' assessments to inform referral to an intensified multidisciplinary prevention program to delay progression to ESKD. Table 4 reports ratings of CKD progression risks provided by either physicians or the prediction model. In the evaluation sample, 25 patients required KRT within 2 years, while 53 patients did not reach the study endpoint. PROGRES-CKD-24 had excellent discrimination within this dataset (AUC = 0.96), while experts' ratings demonstrated good discrimination (average AUC = 0.79), with average sensitivity = 0.64 and average specificity = 0.85 at the optimal cut-off point (score > 6). Therefore, experts were less discriminative of endpoint occurrence compared to PROGRES-CKD-24 (∆M-E = 0.17, p = 0.005). The correlation of physicians' ratings with PROGRES-CKD-24 ratings was moderate (r = 0.50, p < 0.01); furthermore, experts showed different abilities to discriminate patients' risk. (Table 4).  Figure 4 shows the results of our impact simulation. Based on the experts' ratings (PPV = 17%; FOR = 2%), n = 1725 (17.3%) patients would be assigned to the high-risk category, while n = 8275 (82.8%) would be recommended to the standard care program (Figure 4, panel A). Based on the assumptions set for the simulation exercise (i.e., ESKD overall incidence without intervention: 2.3 events/100 patient-years; ESKD risk is reduced by 50% in the intensified intervention group) there would be 362 ESKD events overall. Therefore, in this scenario, physicians' referral to the intensified program would delay 98 ESKD cases (i.e., an Overall Program Effect Size of 1.27). The number of patients needed to treat would be NNT = 18 (Figure 4, panel D). Conversely, risk stratification by PROGRES-CKD-24 (PPV = 48%; FOR = 1.2%) leads to referral of n = 732 (0.73%) patients to intensified intervention (Figure 4, panel B). In this case, 117 ESRD events would be prevented, i.e., an Overall Program Effect Size of 1.36. The number needed to treat would be NNT = 6 ( Figure 4, panel D). Finally, under a hypothetical risk averse policy that would refer all stage 3 CKD patients to the intensified program, 153 ESRD events would be prevented with NNT = 65 (Figure 4, panel C). Figure 4 shows the results of our impact simulation. Based on the experts' ratings (PPV = 17%; FOR = 2%), n = 1725 (17.3%) patients would be assigned to the high-risk category, while n = 8275 (82.8%) would be recommended to the standard care program (Figure 4, panel A). Based on the assumptions set for the simulation exercise (i.e., ESKD overall incidence without intervention: 2.3 events/100 patient-years; ESKD risk is reduced by 50% in the intensified intervention group) there would be 362 ESKD events overall. Therefore, in this scenario, physicians' referral to the intensified program would delay 98 ESKD cases (i.e., an Overall Program Effect Size of 1.27). The number of patients needed to treat would be NNT = 18 (Figure 4, panel D). Conversely, risk stratification by PROGRES-CKD-24 (PPV = 48%; FOR = 1.2%) leads to referral of n = 732 (0.73%) patients to intensified intervention (Figure 4, panel B). In this case, 117 ESRD events would be prevented, i.e., an Overall Program Effect Size of 1.36. The number needed to treat would be NNT = 6 ( Figure 4, panel D). Finally, under a hypothetical risk averse policy that would refer all stage 3 CKD patients to the intensified program, 153 ESRD events would be prevented with NNT = 65 (Figure 4, panel C).

Discussion
The present study reports the derivation and validation of the PROGRES-CKD algorithm in two independent cohorts of non-dialysis dependent CKD patients. Discrimination accuracy of PROGRES-CKD was excellent for both the short-term prediction horizon (6 months) and the long-term prediction horizon (24 months).
Of note is the fact that PROGRES-CKD-6 and PROGRES-CKD-24 had reproducible discrimination accuracy in both validation studies. The FMC NephroCare cohort included real-world clinical data of stage 3-5 CKD patients from 15 countries (Europe, South-America, Africa), while the GCKD study is a prospective CKD cohort study recruiting a wider range of NDD-CKD patients with moderate GFR impairment in Germany [21]. Given the substantial differences between the two cohorts in geographical area of recruitment (international vs. national), inclusion/exclusion criteria, and data collection strategies (real-world vs. pre-specified protocol), the observed consistency in discrimination and calibration corroborates the generalizability of PROGRES-CKD across different CKD subpopulations and clinical settings.
To further characterize PROGRES-CKD accuracy, we compared its discrimination performance against KFREs which were extensively validated in different CKD patient populations [11,17,18] and are routinely used in clinical practice. PROGRES-CKD was as accurate as KFREs for 24-month prediction in both validation cohorts and more accurate for 6-month forecasting in the GCKD study. Even though the two algorithms showed comparable performance in long-term prediction, the KFRE risk score could not be computed in a vast share of patients of the FMC NephroCare cohort because of missing information of key input variables ( Figure 5). Conversely, PROGRES-CKD was available for all patients due to accurate handling of missing variables inherent to naïve Bayes classifiers ( Figure 5) [36]. In fact, PROGRES-CKD potentially incorporates input from as many as 32 clinical parameters, yet its prediction can be computed with any subset of information. Therefore, PROGRES-CKD performance remained stable even for patients with many missing parameters representative of a real-world clinical practice setting. Furthermore, by assessment of VOI metrics, PROGRES-CKD allows the graphical representation of the uncertainty around prediction due to missing data. Given that VOI metrics are calculated for each missing clinical parameter within the patient's health records, they can be used to rank the potential prognostic benefit of additional diagnostic testing or biomarker assays for patients with incomplete medical data. These peculiar features of PROGRES-CKD significantly increase its clinical usability in that they enable to address the problem of missing predictors in real-world data [17] by exploiting the full wealth of information collected in routine clinical practice. One additional advantage of NBCs such as PROGRES-CKD over traditional equation-based prediction tools rest in their ability to generate personalized, patient- One additional advantage of NBCs such as PROGRES-CKD over traditional equationbased prediction tools rest in their ability to generate personalized, patient-specific impact metrics representing the relative contribution of each predictor to a patient's risk. Impact metrics can be used to estimate the potential impact of interventions addressing modifiable risk factors. This has important implications for patient care, since there can be considerable heterogeneity in underlying diseases, demographics, co-morbidities, and risk for progression among CKD patients and, consequently, optimal intervention strategies might deviate between patients with the same overall risk estimate depending on their individual high impact risk parameters. Therefore, both VOI and impact metrics could help physicians within their decision-making processes in tailoring interventions according to each individual patient's needs and characteristics [37]. Adoption of a more personalized clinical approach would lead not only to improved CKD clinical management (targeted diagnostic and treatment investigations with minimum adverse events and maximum efficacy, and consequently increased adherence to treatment), but it could also contribute towards optimizing the utilization of healthcare resources. In fact, ranking clinical parameters by their impact on risk score computation helps physicians' reasoning on priority and enables strategic and rational formulation of therapeutic plans considering both patient/disease-related factors and resource availability.
One specification of PROGRES-CKD allows the identification of patients whose kidney function is more likely to deteriorate within 6 months, a feature enabling timely referral to vascular access creation services and transition management [38,39]. The potential advantages of accurate short-term progression are two-fold. Patients starting on chronic dialysis with an arteriovenous fistula (AVF) rather than catheter have improved clinical outcomes in terms of survival, hospitalization, and complications [40]. On the other hand, inappropriate AVF creation in stage 4 and 5 patients who do not rapidly progress to KF is associated with complications and premature loss of patency [38].
Accurate risk prediction is a challenging task for physicians in real-world clinical practice, due to a number of disease, clinician, and organization related factors, including: inherent heterogeneity and variability in CKD progression rates [41,42], incomplete information, unrecognized case ambiguity, overconfidence leading to reduced analytical scrutiny, wrong perception of average population risk, over-generalization, fatigue, working overload, aging, altered affect impairing executive memory, switch of analytic scrutiny, and inexperience [43][44][45][46][47][48]. Therefore, readily available risk scores which prove to be accurate, generalizable to a wide array of CKD subpopulations and settings, and robust to missing data patterns observed in real-life applications may considerably assist clinical decision making, particularly when providing the opportunity to simulate the impact of interventions to individual patient cases.
In order to estimate the potential impact of improved prognostication around CKD progression on process outcomes, clinical outcomes, and costs [38,49], we conducted a simplified simulation using PROGRES-CKD as a patient stratification system for referral to intensified prevention programs for non-dialysis dependent (NDD)-CKD patients. In our simulation, risk estimates provided by either PROGRES-CKD or nephrology experts were used to stratify CKD patients. Subjects assigned to the "high-risk" category are referred to an intensified healthcare program aimed at reducing the risk of CKD progression. Our analysis suggested that PROGRES-CKD-driven referral to the intensified program would be more effective and largely more efficient than referral patterns determined by both healthcare expert risk assessment and an "all-in strategy" (i.e., all patients are referred to the intensified healthcare program when they reach stage 3 CKD). Therefore, personalized, risk-based referral may improve the efficiency of healthcare systems by enhancing the appropriateness of resource allocation in terms of direct expenditures and staff utilization. Personalized referral, however, is not just a matter of mere efficiency. In fact, inappropriate referral to the intensified intervention would involve unnecessary medicalization with greater risks of adverse events, impoverishment of quality of life even in people with a very low risk of progression, increased rate of therapeutic fatigue, and reduced adherence.
Conversely, accurate and reliable patient stratification helps physicians and healthcare providers balance individual patient needs with overall resource utilization, ultimately leading to more effective care for both the individual patient and the population [50].

Limitations
Validation of risk score should be considered a continuous process of generalization tests rather than a single experiment. While the performance of PROGRES-CKD was stable in both well-conducted longitudinal cohort studies (i.e., GCKD) and historical cohorts of real-life practice (i.e., FMC NephroCare), evidence concerning PROGRES-CKD robustness with real-world-representing clinical practices outside FMC NephroCare is still missing. For this reason, PROGRES-CKD undergoes a periodical process of performance monitoring while external cohorts for validation exercises are actively sought for.

Conclusions
The Prognostic Reasoning System for CKD patients (PROGRES-CKD) demonstrated excellent discrimination accuracy in two independent cohorts of NDD-CKD patients. The underlying models provide accurate prediction for both 24 and 6 months KRT risk. Contrary to traditional equation-based algorithms which cannot be applied to a large proportion of patients with incomplete data, PROGRES-CKD extends to all patients and allows explicit assessment of prediction robustness in case of missing values for key risk factors. Furthermore, PROGRES-CKD enhances prognostic reasoning by providing patientspecific impact metrics representing the relative contribution of each predictor to a patient's risk and can be used to estimate the potential impact of tailored interventions in addressing individual and modifiable risk factors. While PROGRES-CKD-24 may contribute to efficient and effective referral to intensified prevention programs for NDD-CKD patients, prediction of short-term outcomes (PROGRES-CKD-6) can be a key enabler of timely AVF creation and transition management. Given these results, both PROGRES-CKD algorithms reported here have the potential to advance current standards in routine CKD risk estimation, patient stratification, and individualizing interventions.  Institutional Review Board Statement: The study was approved by the ethics committees of all participating institutions and registered in the national registry for clinical studies (DRKS 00003971).

Informed Consent Statement:
No patients' personal information has been used for the present study since all input data for modeling were aggregated statistics.

Data Availability Statement:
We are unable to share the raw clinical data of the FMC NephroCare because data sharing would violate the terms and conditions under which Fresenius Medical Care acquired the data. Data from the GCKD study are not publicly available. External collaborators with a specific research proposal can access deidentified participant data only after review and approval of their proposal by the steering committee.