Comparative Performance of Four Established Neonatal Disease Scoring Systems in Predicting In-Hospital Mortality and the Potential Role of Thromboelastometry

Background: To compare the prognostic accuracy of the most commonly used indexes of mortality over time and evaluate the potential of adding thromboelastometry (ROTEM) results to these well-established clinical scores. Methods: The study population consisted of 473 consecutive term and preterm critically-ill neonates. On the first day of critical illness, modified Neonatal Multiple Organ Dysfunction (NEOMOD) scoring system, Score for Neonatal Acute Physiology (SNAP II), Perinatal extension of SNAP (SNAPPE), and SNAPPE II, were calculated and ROTEM standard extrinsically activated (EXTEM) assay was performed simultaneously. Time-to-event methodology for competing-risks was used to assess the performance of the aforementioned indexes in predicting in-hospital mortality over time. Time-dependent receiver operator characteristics curves for censored observation were compared across indexes. The addition of EXTEM parameters to each index was tested in terms of discrimination capacity. Results: The modified NEOMOD score performed similarly to SNAPPE. Both scores performed significantly better than SNAP II and SNAPPE II. Amplitude recorded at 10 min (A10) was the EXTEM parameter most strongly associated with mortality (A10 < 37 mm vs. ≥37 mm; sHR = 5.52; p < 0.001). Adding A10 to each index apparently increased the prognostic accuracy in the case of SNAP II and SNAPPE II. However, these increases did not reach statistical significance. Conclusion: Although the four existing indexes considered showed good to excellent prognostic capacity, modified NEOMOD and SNAPPE scores performed significantly better. Though larger studies are needed, adding A10 to well-established neonatal severity scores not including biomarkers of coagulopathy might improve their prediction of in-hospital mortality.

. Flow-chart of study population.
The inclusion and exclusion criteria have been previously described [9]. Data on demographics, maternal and pregnancy history, maternal medications during pregnancy, neonatal physiological parameters, and clinical findings were recorded. On the first day of sepsis, suspected sepsis, perinatal hypoxia, and/or respiratory distress syndrome (RDS), arterial blood anticoagulated with 0.109 mol/L trisodium citrate (9:1, v/v blood anticoagulant) was analyzed on the ROTEM analyzer (Tem Innovations GmbH, Munich, Germany) using the extrinsically activated (EXTEM) thromboelastometry assay, as previously described [10,11]. The following EXTEM parameters were measured: Clotting Time (CT, seconds), defined as the time from test start until a clot firmness amplitude of 2 mm is reached; Clot Formation Time (CFT, seconds), indicating the time from the end of the CT until clot firmness amplitude of 20 mm is achieved; Alpha-angle (α°), the angle between the baseline and a tangent to the clotting curve through the 2 mm point; Clot firmness amplitude recorded at 10, 20, and 30 min (A10, A20, and A30); Maximum Clot Firmness (MCF, mm), representing the final strength of the clot; Lysis Index at 60 min (LI 60%), defined as the percentage of remaining clot in relation to the MCF over the 60-min observation period after CT; and Maximum Lysis (ML), which was observed during the run time, described as the percentage of MCF.
On the same day of ROTEM testing and prior to initiating antibiotic therapy, blood specimens for culture, routine biochemical tests, complete blood count, and C-reactive protein (CRP) were obtained. Chest radiograph, cerebrospinal fluid culture, and urine culture were performed whenever clinically indicated, as per our NICU protocol. Modified NEOMOD scoring system [12], SNAP II, SNAPPE II, and SNAPPE [13,14] were calculated at the same time with the EXTEM analysis. Data collection took place within the first 24 h after the disease onset. Clinical status was evaluated daily until discharge or death [15,16]. Each neonate included in the study was evaluated only once.

Statistical Analysis
We present descriptive statistics of the baseline characteristics and ROTEM parameters as means ± standard deviations (SD), medians and interquartile ranges (IQR), or percentages when appropriate. The inclusion and exclusion criteria have been previously described [9]. Data on demographics, maternal and pregnancy history, maternal medications during pregnancy, neonatal physiological parameters, and clinical findings were recorded. On the first day of sepsis, suspected sepsis, perinatal hypoxia, and/or respiratory distress syndrome (RDS), arterial blood anticoagulated with 0.109 mol/L trisodium citrate (9:1, v/v blood anticoagulant) was analyzed on the ROTEM analyzer (Tem Innovations GmbH, Munich, Germany) using the extrinsically activated (EXTEM) thromboelastometry assay, as previously described [10,11]. The following EXTEM parameters were measured: Clotting Time (CT, seconds), defined as the time from test start until a clot firmness amplitude of 2 mm is reached; Clot Formation Time (CFT, seconds), indicating the time from the end of the CT until clot firmness amplitude of 20 mm is achieved; Alpha-angle (α • ), the angle between the baseline and a tangent to the clotting curve through the 2 mm point; Clot firmness amplitude recorded at 10, 20, and 30 min (A10, A20, and A30); Maximum Clot Firmness (MCF, mm), representing the final strength of the clot; Lysis Index at 60 min (LI 60%), defined as the percentage of remaining clot in relation to the MCF over the 60-min observation period after CT; and Maximum Lysis (ML), which was observed during the run time, described as the percentage of MCF.
On the same day of ROTEM testing and prior to initiating antibiotic therapy, blood specimens for culture, routine biochemical tests, complete blood count, and C-reactive protein (CRP) were obtained. Chest radiograph, cerebrospinal fluid culture, and urine culture were performed whenever clinically indicated, as per our NICU protocol. Modified NEOMOD scoring system [12], SNAP II, SNAPPE II, and SNAPPE [13,14] were calculated at the same time with the EXTEM analysis. Data collection took place within the first 24 h after the disease onset. Clinical status was evaluated daily until discharge or death [15,16]. Each neonate included in the study was evaluated only once.

Statistical Analysis
We present descriptive statistics of the baseline characteristics and ROTEM parameters as means ± standard deviations (SD), medians and interquartile ranges (IQR), or percentages when appropriate.
In agreement with previous studies, in-hospital mortality was selected as the study outcome [13,[17][18][19][20]. Time-to-event (survival) methodology for competing risks was used to assess how the four most frequently used indexes for mortality perform over time in predicting in-hospital death. Time to event was defined as the time from ROTEM analysis to the date of death or censoring (i.e., we censored data for patients still hospitalized in the NICU at the time of data extraction). Patient discharge was considered as a competing event for death [21], as it prevents the observation of the event of interest and is associated with it (i.e., the probability of dying at a certain time point is drastically lower in the group of patients discharged as compared to the group of patients still being followed in the NICU).
The cumulative incidence functions were estimated and plotted for the competing events of death and discharge. The association between the study outcome and the four prognostic indexes (SNAP II, SNAPPE II, SNAPPE, and modified NEOMOD scores, respectively) was quantified by means of Fine and Gray competing risk regression with robust estimators of the standard error and considering the sub-hazard function of the event of primary interest (death) [22]. The sub-hazard ratios (sHR) were presented with 95% confidence intervals (CI) and the respective p-values.
Time-dependent receiver operating characteristic (ROC) curve analysis for censored observation, by means of inverse probability of censoring weighting, was used to evaluate the discrimination power of the four indexes in predicting patients who will die [23]. In time-dependent ROC-curve analysis, the status of an individual is observed and updated at each time point taking into account censored observations and competing events. Thus, the ROC curve can be constructed at several time points, and the discrimination power curve (i.e., the area under curve, AUC) can be compared during time, allowing a dynamic estimate of the performance of each individual index [23]. As a sensitivity analysis, we also calculated and plotted the prognostic performances of the four indexes in preterm (<37 weeks) and term (≥37 weeks) neonates.
A secondary aim of the study was to test the hypothesis that ROTEM variables could improve the prognostic power of indexes of mortality frequently used in clinical practice. In a very similar cohort of neonates, A10, CT, and MCF EXTEM ROTEM variables had been shown as significantly associated with mortality [9]. In the current study, A10 was the variable most strongly associated with mortality. We plotted the cumulative incidence curves for A10 dichotomized at the value of 37 mm, as previously reported [9], and compared them with the Gray's test [24]. Then, we applied multivariable Fine and Gray regression, and calculated the sHR for death in models including SNAP II, SNAPPE II, SNAPPE, and modified NEOMOD, respectively, in addition to A10. We calculated and plotted the AUC over time for each one of these four multivariable models, and compared them with that of the corresponding univariable models, including the mortality indexes alone. The analyses were based on non-missing data (i.e., missing data were not imputed); less than 1% of observations were missing. For statistical analysis, we used Stata 15 (Stata Corp., College Station, TX, USA), and the R software. For all the tests, a two-tailed p-value < 0.05 was considered statistically significant.
Out of the 473 neonates, 42 (8.9%) died, 426 were discharged, and 6 were still in the NICU at data extractions (censored). The cumulative incidence curve showed an overall cumulative probability of death of 0.0871 (95% CI, 0.0831-0.0910) over an overall time at risk of 38.5 person-years ( Figure 2).

Performance over Time of SNAP II, SNAPPE II, SNAPPE and Modified NEOMOD Score
The four ROC curves over time displaying the prognostic performance of SNAP II, SNAPPE II, SNAPPE, and the modified NEOMOD score in predicting death, are presented in Figure 3. The modified NEOMOD score performed similarly to SNAPPE with no statistically significant difference at any time point (Table 2). The modified NEOMOD score performed similarly to SNAPPE with no statistically significant difference at any time point (Table 2).  Both the modified NEOMOD and the SNAPPE scores were significantly better than SNAP II and SNAPPE II at any time point besides at two weeks from study enrolment, when the difference did not reach significance ( Table 2). The modified NEOMOD score was significantly better than SNAP II at any time point. As a sensitivity analysis, we assessed the performance of the indexes in preterm and term neonates ( Figure 4).  In preterm neonates, SNAPPE was apparently the best performing index; however, the differences with the modified NEOMOD score and SNAPPE II did not reach significance. The AUC over time of SNAPPE was significantly better than that of SNAP II at any time point besides the first week (Table 3).  In preterm neonates, SNAPPE was apparently the best performing index; however, the differences with the modified NEOMOD score and SNAPPE II did not reach significance. The AUC over time of SNAPPE was significantly better than that of SNAP II at any time point besides the first week (Table 3).
In term neonates, the modified NEOMOD score was apparently the best performing index; however, it was significantly better than SNAP II and SNAPPE II in predicting exclusively short-term mortality (Figure 4 and Table 3). In fact, the difference in terms of AUC between the modified NEOMOD score and SNAPPE II was significant at days 1-13, while there was a statistically significant difference with SNAP II at days 1-10. We found no statistically significant difference regarding the prognostic power of the modified NEOMOD score and SNAPPE at any time point.

Multivariable Models including the A10 EXTEM Parameter
A10 was the ROTEM parameter most strongly associated with mortality in Fine and Gray univariable regression (A10 < 37 mm vs. ≥37 mm; sHR = 5.52; 95% CI: 2.98-10.2; p < 0.001). The plotted cumulative incidence curves displayed an about five-fold higher risk of dying in the group of neonates showing values of A10 < 37 mm (Gray's test p < 0.001; Figure 5).
We plotted the AUC over time of the four univariable models including the established indexes of mortality, and the four multivariable models after the addition of the continuous A10 parameter (Figure 6). the ROTEM variable were independent, statistically significant predictors of mortality. This was not the case considering the model including the modified NEOMOD score together with A10 (sHR(NEOMOD) = 1.46, p < 0.001; sHR(A10) = 0.990, p = 0.49).
We plotted the AUC over time of the four univariable models including the established indexes of mortality, and the four multivariable models after the addition of the continuous A10 parameter (Figure 6). Apparently, the addition of A10 to the established index of mortality increased the prognostic power in the case of SNAP II and SNAPPE II. However, we observed no statistically significant difference between the AUC of any univariable model and the respective multivariable model including A10.

Discussion
To the best of our knowledge, this is the first study simultaneously assessing the time-dependent performance of four different prognostic indexes (SNAP II, SNAPPE II, SNAPPE, modified NEOMODs) in predicting mortality in critically-ill neonates over the entire hospital stay. In NICU-treated neonates, despite the changes in illness severity over time and several medical interventions, the risk of death still remains high during their hospitalization. Thus, the quantification of the time-dependent performance of these indexes might be of significant clinical value. The modified NEOMOD and the SNAPPE scores were the ones that performed best at almost any time point. Although SNAP II and SNAPPE II performed worse, especially in preterm infants, their use is very common since these indexes are much easier to calculate than SNAPPE. The addition of EXTEM A10 increased the prognostic power in the case of SNAP II and SNAPPE II, but without reaching statistical significance.
The use of prognostic indexes in NICUs has been established many years ago and they have been evaluated over time as useful tools to predict outcome, to quantify the initial risk of mortality and morbidity in critically-ill neonates along with stratifying them according to the risk level, to guide the optimal interventions, and to assess their effectiveness. These indexes have also served to compare lifetime outcomes in these vulnerable populations across hospitals or NICUs [2]. The SNAP, SNAPPE, and their next generation variants SNAP II and SNAPPE II are the most commonly used admission scores for the prediction of mortality risk in ill neonates. The original SNAP developed by Richardson et al. scores the worst physiologic derangements in each organ system in the first 24 h and involves 28 physiological parameters [25], whereas SNAPPE, the perinatal extension of SNAP by adding BW, small for gestational age (SGA), and five minute Apgar score, quantifies both physiological instability parameters and perinatal risk factors in one score [14]. However, these tools are labor-intensive and time-consuming, because of the large number of included components, while requiring up to 15 min for evaluation. Thus, Richardson et al. simplified them and created the SNAP II and SNAPPE II scores [13]. While the primary use of all these scores was to assess illness severity in the first 12-24 h of life, limited studies used them at later time points and for sequential measurements [26,27]. Griffin et al. found higher SNAP scores for up to 24 h before the clinical suspicion of sepsis [28]. Sundaram et al. concluded that neonates with severe septicemia are at a significantly higher risk of dying if they have high SNAP II score within the first 12 h from the onset of severe sepsis [29].
Taking into account the changes of clinical status of ill neonates over time and the effectiveness of clinical interventions, it seems rather precarious to evaluate illness severity based only on data collected during the first day of life. In line with this concept, in our study, the discriminative ability of SNAPPE, SNAP II, and SNAPPE II scores was assessed within the first 24 h of abrupt clinical deterioration of our subjects, with the SNAPPE showing significantly better prognostic performance than the other two scores (AUC point estimates: from 0.828 at 5 days to 0.862 at 120 days) at any time point, except on the 15th day of study enrolment when the difference did not reach significance. This finding was quite expected, as the SNAPPE score consists of an extensive list of objective laboratory variables in association with perinatal risk factors, such as five-minute Apgar score, BW, and SGA, and therefore more accurately describes illness severity. Additionally, SNAPPE II and SNAPPE showed better performance in preterm neonates when compared to SNAP II, probably because BW (included in SNAPPE and SNAPPE II) is the physiologic parameter with the major contribution to mortality rate in preterms [30,31], while all three scores had a similar performance in term neonates.
MODS is the most common cause of death for neonates admitted to NICUs. Janota et al. developed the NEOMOD score, which characterizes the severity of dysfunction in seven organ systems in very low birth weight (VLBW) neonates, and concluded that this score could evaluate the severity of MODS and predict mortality with accuracy [3,15]. As microvascular system derangement might be the earliest sign of MODS in neonates, Çetinkaya et al. developed the modified NEOMOD score, in which the microvascular system was added as the eighth system resulting in an extension of the current criteria of the NEOMOD scoring system [12].
In our study, the modified NEOMOD score showed better performance in predicting mortality overtime (AUC point estimates: from 0.858 at 5 days to 0.860 at 120 days) compared to SNAP-II and SNAPPE II scores. This finding is probably attributable to the fact that the modified NEOMOD score incorporates more clinical and laboratory parameters than the other two scores, possibly better reflecting the critically-ill neonate's clinical status. In preterm neonates, the modified NEOMOD score showed performance similar to SNAP II, SNAPPE II, and SNAPPE scores at any time point. Additionally, the modified NEOMOD score could accurately predict mortality over time in term neonates (AUC point estimates: from 0.928 at 5 days to 0.881 at 120 days), with significantly better performance than SNAP II and SNAPPE II during the first five days after study enrollment. The prognostic power of the modified NEOMOD and SNAPPE scores was similar at any time point irrespective of gestational age.
To our knowledge, this is the first study simultaneously assessing the discrimination ability of SNAP II, SNAPPE II, SNAPPE, and modified NEOMOD scores in predicting long-term in-hospital mortality in critically-ill neonates at disease onset. Lee et al. tested the time-dependent performance of the CRIB II in predicting mortality among VLBW neonates and reported that certain CRIB II cutoffs were significantly associated with time-dependent mortality, particularly within the first 90 days after birth [32]. In our study, CRIB II was not calculated as it is designed for use within the first hour of NICU admission [33].
Coagulation derangement often complicates the clinical course of critically-ill neonates and has the potential to be a life-threatening situation and increase mortality [34]. Thus, early and proper identification of hemostatic disorders is of great importance for the management of these patients. Among the laboratory tests used for diagnosing these disorders, global coagulation tests such as PT and APTT cannot provide a complete insight on patient's hemostatic status. Contrarily, as VMs can detect and quantify dynamic changes in the hemostatic properties of a blood sample during the clot formation process, they could provide more specific information regarding the coagulation profile.
In two recent studies conducted by our research group using ROTEM, a hypocoagulable profile was detected in hypoxic and septic neonates, expressed with prolonged CT, CFT, and reduced A10 and MCF as compared to healthy neonates [10,11]. Recently, a hypocoagulable profile and impaired fibrinolysis on disease onset were identified as independent risk factors for in-hospital mortality in critically-ill neonates [9]. In the current study, A10 was the EXTEM parameter most strongly associated with mortality, as neonates showing values of A10 < 37 mm presented an about five-fold higher risk of in-hospital mortality. Adamzik et al. [35] and Ostrowski et al. [36] noted that a hypocoagulable profile on admission could predict mortality in adults with sepsis. The plausible mechanism underlying the association of coagulopathy with clinical outcome is probably the devastating host immune response leading to enhanced fibrin formation and its microvascular deposition, which results in subsequent organs dysfunction [37].
Neonatal coagulopathy and especially disseminated intravascular coagulation (DIC) are associated with worse prognosis in critically-ill neonates [34]. Although the hemostatic system has been increasingly recognized as a key player in the development and course of critical illness, except from platelets, no other biomarkers of coagulopathy are currently incorporated into the neonatal prognostic scores. Taking into account all the above information, we further examined the potential of increasing the prognostic power of each of the aforementioned scores (SNAP II, SNAPPE II, SNAPPE, modified NEOMOD) when adding EXTEM A10. The addition of A10 to SNAP II and SNAPPE II indexes apparently increased their discrimination capacity in predicting mortality (e.g., at 120 days; SNAP II, AUC point estimate: 0.768 vs. SNAP II plus A10, AUC point estimate: 0.801), though the increase was not statistically significant. Notably, the addition of A10 did not improve the performance of either SNAPPE or modified NEOMOD. These suggestive findings could be attributed to the fact that hemostatic balance is partially reflected in SNAPPE and modified NEOMOD by platelet count. Although no statistically significant difference was observed between the AUC of any univariable model and the respective multivariable model including A10, probably due to the small number of non-survivors in our study, our findings may suggest that EXTEM parameters might improve the performance of neonatal severity scores not including biomarkers of coagulopathy.
Several limitations have to be acknowledged. As disease severity scores were calculated only on the first day of disease onset along with EXTEM measurements, the dynamic changes in clinical status of critically-ill neonates over time and the potential effectiveness of clinical interventions were not evaluated and this might have influenced the prognostic power of these scores. However, this reflects the intended use of these scores, and it is noteworthy that the predictive performance of the neonatal severity scores remained stable over time. Furthermore, this is a single center study resulting in a relatively limited sample size. The number of deceased neonates is also rather small, probably inflating our chance of type-two error. On the other hand, data derived from a single center minimizes the effects of different practices on clinical approach and patient evaluation, and indirectly, on the prediction of in-hospital mortality.
In conclusion, the incorporation of thromboelastometry variables into the established neonatal severity score systems not including biomarkers of coagulopathy might contribute to the improvement of their prediction performance. The inclusion of conventional or viscoelastic hemostatic variables representing the extent of the coagulopathy seems essential for optimizing the prognostic performance of neonatal disease scoring systems. Further studies with a larger sample size are needed to confirm these findings. T. and P.K. were involved in analysis and data interpretation. A.E.T., R.S., A.K., S.B., D.P. and M.T. wrote the manuscript. All the co-authors critically revised and finally approved the manuscript. All authors agree to be held accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. All authors have read and agreed to the published version of the manuscript.

Funding:
The research received no external funding.

Institutional Review Board Statement:
The study protocol was designed and conducted in compliance with the Declaration of Helsinki, and was approved by the Institutional Review Board of Nikaia General Hospital (approval code 32/3, approval date 15 July 2014 ).

Informed Consent Statement:
Written informed consent has been obtained by parents or guardians of enrolled neonates.

Data Availability Statement:
The data presented in this study are available on request from the corresponding author.

Conflicts of Interest:
The authors declare no conflict of interest.