Construction and Validation of Mortality Risk Nomograph Model for Severe/Critical Patients with COVID-19

Abstract Objective: A nomograph model of mortality risk for patients with coronavirus disease 2019 (COVID-19) was established and validated. Methods: We collected the clinical medical records of patients with severe/critical COVID-19 admitted to the eastern campus of Renmin Hospital of Wuhan University from January 2020 to May 2020 and to the north campus of Shanghai Ninth People’s Hospital, Shanghai JiaoTong University School of Medicine, from April 2022 to June 2022. We assigned 254 patients to the former group, which served as the training set, and 113 patients were assigned to the latter group, which served as the validation set. The least absolute shrinkage and selection operator (LASSO) and multivariable logistic regression were used to select the variables and build the mortality risk prediction model. Results: The nomogram model was constructed with four risk factors for patient mortality following severe/critical COVID-19 (≥3 basic diseases, APACHE II score, urea nitrogen (Urea), and lactic acid (Lac)) and two protective factors (percentage of lymphocyte (L%) and neutrophil-to-platelets ratio (NPR)). The area under the curve (AUC) of the training set was 0.880 (95% confidence interval (95%CI), 0.837~0.923) and the AUC of the validation set was 0.814 (95%CI, 0.705~0.923). The decision curve analysis (DCA) showed that the nomogram model had high clinical value. Conclusion: The nomogram model for predicting the death risk of patients with severe/critical COVID-19 showed good prediction performance, and may be helpful in making appropriate clinical decisions for high-risk patients.


Introduction
COVID-19, caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), has been defined by the World Health Organization (WHO) as a "public health emergency of international concern", seriously threatening the safety of human life [1]. Studies have shown that the 28 d mortality rate of patients with severe COVID-19 has been as high as 61.5% [2]. Therefore, early identification of critical patients and early accurate treatment can effectively reduce COVID-19-related mortality rates [3]. In recent years, many risk models for predicting the death of patients with COVID-19 have been published internationally [4][5][6][7][8], but most of them had only internal validation, without external validation across regions. In March 2022, an epidemic of the Omicron mutant broke out in Shanghai. On 17 April 2022, the intensive-care team for Shanghai aid from Remmin Hospital of Wuhan University entered the ICU ward of the north campus of Shanghai Ninth People's Hospital, Shanghai JiaoTong University School of Medicine. In the course of treatment, we learned from the valuable experience of anti-epidemic approaches taken in Wuhan and summarized the characteristics of the anti-epidemic approach taken in Shanghai. Based on the clinical analysis of common test results and the universality of detection in patients, we constructed a nomogram model of mortality risk using the data of severe/critical COVID-19 patients in Wuhan, and externally verified the data of severe/critical COVID-19 patients in Shanghai.

General Data
This study was a double-center, retrospective, and observational study. It was approved by the ethics committee of Renmin Hospital of Wuhan University and Shanghai Ninth People's Hospital, affiliated with Shanghai JiaoTong University School of Medicine. The study obtained ethics exemption. We collected the clinical medical records of patients with severe/critical COVID-19 admitted to the eastern campus of Renmin Hospital of Wuhan University from January 2020 to May 2020 and those admitted to the north campus of Shanghai Ninth People's Hospital, Shanghai JiaoTong University School of Medicine from April 2022 to June 2022. The data of the patients admitted to the former hospital were used as the training set (n = 254), and data of the patients in the latter were used as the validation set (n = 113). The patients with severe/critical COVID-19 were taken as the starting point and the patients who died in hospital or were followed-up for 28 days were taken as the end point. The follow-up date was 26 June 2022.

Inclusion Criteria
The inclusion criteria for this study were as follows: (1) age ≥ 18 years old; (2) patients with complete case data; (3) positive for the novel coronavirus nucleic acid detected by fluorescence quantitative PCR.

Enrollment Criteria for Patients in Shanghai
According to recommendations in the ninth edition of the COVID-19 diagnosis and treatment guidelines [10], those who met one of the following criteria were diagnosed as severe cases: (1) shortness of breath-RR ≥ 30 bpm; (2) SpO2 ≤ 93% on room air at rest state; (3) PaO2/FiO2 ≤ 300 mmHg; (4) progressive aggravation of clinical symptoms, and with > 50% lesions progression within 24~48 h in pulmonary imaging. Those who met one of the following criteria were diagnosed as critically ill cases: (1) respiratory failure occurred and mechanical ventilation was required; (2) shock occurred; (3) complications with other organ failure that require monitoring and treatment.

Exclusion Criteria
The exclusion criteria for this study were as follows: (1) age < 18 years old; (2) pregnant or lactating; (3) treatment period less than 3 days; (4) patients and their families who requested cessation of active treatment and could not receive routine comprehensive treatment; (5) patients who lost contact during follow-up.

Data Collection and Grouping
The general information, physical examination, and auxiliary examination data for the patients were collected, and the classification was determined according to the recommendations of the guidelines [9,10]. The worst blood samples of patients with severe/critical COVID-19 within 24 h after admission were collected, including blood routine, blood biochemistry, coagulation function, blood gas analysis, and other related indexes, and the scores of acute physiology and chronic health evaluation II (APACHE) were calculated.

Statistical methods
SPSS 26.0 and R 4.1.3 were used for statistical analysis and mapping. The measurement data of this study did not obey the normal distribution, and are represented by the median (quartile) (M (Q L , Q U )). Comparison between groups is expressed by Mann-Whitney U test. The count data are expressed as [n (%)], and the Chi-square test, the Chi-square test of continuous correction, or Fisher's precision probability test were used for the comparison between groups. Whether or not death occurred within 28 days after the diagnosis of severe/critical COVID-19 (secondary outcome index) was taken as the dependent variable. Lasso regression was used to screen the independent variables, and multivariate logistic regression analysis was used to screen the risk factors. The nomogram prediction model was constructed by using R software rms package, and the ability of the model to the predict a prognosis of death was evaluated by ROC. The bootstrap method was used to repeatedly sample 1000 times for internal validation, and the difference of C-index was compared. Furthermore, a calibration curve and DCA were used to evaluate the prediction model. The difference was statistically significant (p < 0.05).

Analysis of Baseline Data of Patients
Of the 254 patients in the training set, 74 (29.13%) eventually died; among these patients, 57 (22.44%) were still COVID-19-positive at the time of death, but 17 (6.69%) patients were COVID-19-negative according to two consecutive coronavirus nucleic acid tests before death (sampling time at least 24 h apart). Of the 113 patients in the validation set, 21 (18.58%) eventually died; among these patients, 5 (4.42%) were still COVID-19-positive according to the novel coronavirus nucleic acid test at the time of death, and 16 (14.16%) patients were COVID-19-negative according to two consecutive novel coronavirus nucleic acid tests before death (at least 24 h apart at the time of sampling). The causes of death were other irreversible serious complications or basic diseases. The baseline data are shown in Table 1. In the training set and the validation set, the patients in the death group were older, more critically ill, had more than three basic diseases, and had higher APACHE II scores (p < 0.05). As shown in Table 2, leukocyte count, the neutrophil-to-lymphocyte ratio (NLR), NPR, procalcitonin, direct bilirubin, Lac, B-type natriuretic peptide, troponin, and D-dimer were higher in the group of deceased patients in the training set; however, hemoglobin and albumin were lower in the validation set (p < 0.05).    Note: ECMO-extracorporeal membrane oxygenation; APACHE II score-acute physiology and chronic health evaluation II score; a -the chi-square value of continuous correction; b -Fisher's test. Note: WBC-white blood cell; N%-percentage of neutrophils; N-neutrophils; L-lymphocytes; L%-percentage of lymphocytes; NLR-neutrophil-to-lymphocyte ratio; Hb-hemoglobin; PLT-platelet; NPR-neutrophil-toplatelets ratio; CRP-C-reactive protein; SAA-serum amyloid A; PCT-procalcitonin; ALT-alanine aminotransferase; AST-aspartate aminotransferase; TBIL-total bilirubin; DBIL-direct bilirubin; ALB-albumin; Urea-urea nitrogen; Cr-creatinine; eGFR-estimated glomerular filtration rate; Lac-lactic acid; BNP-B-type natriuretic peptide; cTnI-troponin; PT-prothrombin time; APTT-activated partial thrombin time.

Lasso and Logistic Regression Analysis
A total of 51 potential death-related risk factors were included in the study. The variables of the training set were dimensionally reduced by lasso regression, and the most representative characteristic variables were selected. When selecting the optimal lambda parameters, 5-fold cross-validation was used, and the lambda value with the smallest cross-validation error was taken as the optimal value of the model (Figures 1 and 2), and the number of variables at this time was counted. The results of Lasso regression analysis showed that 11 independent variables, including age ≥65 years, ≥3 basic diseases, APACHE II score, L%, Urea, D-dimer, Lac, NLR, NPR, leukocyte count, and troponin, were the characteristic variables affecting the death of patients with COVID−19. Taking the occurrence of death as the dependent variable and the 11 characteristic variables screened by lasso regression as the independent variables, the results of multivariate logistic regression analysis (Table 3) show that ≥3 basic diseases, APACHE II score, Urea, and Lac were the independent risk factors for death, while L% and NPR were protective factors against death.

Lasso and Logistic Regression Analysis
A total of 51 potential death-related risk factors were included in the study. The variables of the training set were dimensionally reduced by lasso regression, and the most representative characteristic variables were selected. When selecting the optimal lambda parameters, 5-fold cross-validation was used, and the lambda value with the smallest cross-validation error was taken as the optimal value of the model (Figures 1 and 2), and the number of variables at this time was counted. The results of Lasso regression analysis showed that 11 independent variables, including age ≥65 years, ≥3 basic diseases, APACHE II score, L%, Urea, D-dimer, Lac, NLR, NPR, leukocyte count, and troponin, were the characteristic variables affecting the death of patients with COVID−19. Taking the occurrence of death as the dependent variable and the 11 characteristic variables screened by lasso regression as the independent variables, the results of multivariate logistic regression analysis (Table 3) show that ≥3 basic diseases, APACHE II score, Urea, and Lac were the independent risk factors for death, while L% and NPR were protective factors against death.

Establishment of Nomogram Model
Multivariate logistic regression analysis finally selected 6 independent variables, including ≥3 basic diseases, APACHE II score, Urea, Lac, L%, and NPR to construct the nomogram model, which could be obtained by visual analysis of R language ( Figure 3). Critical care physicians could assess the death risk of severe/critical COVID-19 patients in a visual and individualized quantitative way according to these six easily available indicators. The C-index of the training set was 0.880 (95%CI, 0.837~0.923); The larger the C-index was, the better the discrimination of the model was, suggesting the nomogram had better prediction accuracy.

Internal and External Validation of the Model
The AUC of the training set was 0.880 (95% CI, 0.837~0.923), with a sensitivity of 87.2% and specificity of 71.6%; the AUC of the validation set was 0.817 (95% CI, 0.712~0.923), with a sensitivity of 78.5% and a specificity of 81.0%, indicating good prediction performance, as shown in Figure 4. The training set is internally validated by bootstrap to have the same C-index, that is, the predicted results of the model are consistent with the actual results. The calibration curve was drawn to evaluate the judgment ability of the nomogram model, and Hosmer-Lemeshow goodness-of-fit test was performed. The calibration curve of the training set and the validation set showed that the nomogram and the reference line had goodness of fit, with p values of 0.800 and 0.533, respectively (both >0.05), as shown in Figure 5. The Brier scores in the training set and the validation set were 0.049 and 0.086, respectively, which were both close to 0, indicating that the nomogram model predicted a good consistency between the probability of death among se-

Internal and External Validation of the Model
The AUC of the training set was 0.880 (95% CI, 0.837~0.923), with a sensitivity of 87.2% and specificity of 71.6%; the AUC of the validation set was 0.817 (95% CI, 0.712~0.923), with a sensitivity of 78.5% and a specificity of 81.0%, indicating good prediction performance, as shown in Figure 4. The training set is internally validated by bootstrap to have the same C-index, that is, the predicted results of the model are consistent with the actual results. The calibration curve was drawn to evaluate the judgment ability of the nomogram model, and Hosmer-Lemeshow goodness-of-fit test was performed. The calibration curve of the training set and the validation set showed that the nomogram and the reference line had goodness of fit, with p values of 0.800 and 0.533, respectively (both >0.05), as shown in Figure 5. The Brier scores in the training set and the validation set were 0.049 and 0.086, respectively, which were both close to 0, indicating that the nomogram model predicted a good consistency between the probability of death among severe/critical COVID-19 patients and the actual percentage of death in the observed population.
87.2% and specificity of 71.6%; the AUC of the validation set was 0.817 (95% CI, 0.712~0.923), with a sensitivity of 78.5% and a specificity of 81.0%, indicating good prediction performance, as shown in Figure 4. The training set is internally validated by bootstrap to have the same C-index, that is, the predicted results of the model are consistent with the actual results. The calibration curve was drawn to evaluate the judgment ability of the nomogram model, and Hosmer-Lemeshow goodness-of-fit test was performed. The calibration curve of the training set and the validation set showed that the nomogram and the reference line had goodness of fit, with p values of 0.800 and 0.533, respectively (both >0.05), as shown in Figure 5. The Brier scores in the training set and the validation set were 0.049 and 0.086, respectively, which were both close to 0, indicating that the nomogram model predicted a good consistency between the probability of death among severe/critical COVID-19 patients and the actual percentage of death in the observed population.

Clinical Decision Curve Analysis (Figure 6)
DCA determines the clinical application value of nomogram model by calculating the net benefit under the probability of each death risk threshold. The abscissa of DCA is the high-risk threshold probability, and the ordinate is the net benefit (NB). When the model reaches a certain value, the probability of patient death is recorded as Pi, and it is defined as positive when Pi reaches a certain threshold (recorded as Pt). The high-risk threshold is set as (0, 1), and the net benefit rate and effective prediction probability range is the high-risk threshold probability, and the ordinate is the net benefit (NB). When the model reaches a certain value, the probability of patient death is recorded as Pi, and it is defined as positive when Pi reaches a certain threshold (recorded as Pt). The high-risk threshold is set as (0, 1), and the net benefit rate and effective prediction probability range are evaluated by deducting the false-positive population misjudged by the model. When all patients survive or die, the nomogram model has no clinical application value. The threshold probabilities of the training set and the validation set are all between 0.01~0.94, and the net benefit rate is >0, which has clinical practical value, suggesting that the model has good clinical application value in predicting the death of severe/critical COVID-19 patients.

Discussion
COVID-19 is a viral respiratory infectious disease, which has been a global pandemic in recent years and has had serious impacts on the economy and public health [11,12]. Studies have shown that patients with mild and common COVID-19 have a good prognosis, while the mortality of severe patients is 1.4~4.5% [13], and the mortality of critical patients is as high as 38.73% [14]. The prediction model is not only suitable for the allocation of limited medical resources, but also can help reduce mortality rates [15]. Compared with previously published death prediction models [4,5,7], our study mainly focused on patients with severe/critical COVID-19; we developed and externally validated a new practical model to identify patients at higher risk of death. The development and validation of the model should follow the requirements and recommendations in the Tripod statement [16]. Because the total sample size included in this study was small, with many independent variables and collinearity between different variables, Lasso regression analysis was used to screen potential independent variables. Compared with the currently commonly used stepwise regression method and optimal subset method, Lasso regression can reduce the dimension of multiple parameters and provide a solution to the collinearity problem of multiple parameters, avoiding overfitting and ensuring global optimization [17]. The model was composed of six variables, including having under three basic diseases, APACHE II score, Urea, Lac, L%, and NPR, which were obtained at admission and were all obtained from our data. The model was externally verified in the validation set, showing high discrimination and good prediction performance. Future studies may refine predictors by selecting other characteristic variables, such as CT scans, lymphocyte subtypes, organ injury markers, and cytokines.
All the predictive indicators are routine clinical testing items with fast detection speeds and low prices. Studies have found that advanced age, high APACHE II score,

Discussion
COVID-19 is a viral respiratory infectious disease, which has been a global pandemic in recent years and has had serious impacts on the economy and public health [11,12]. Studies have shown that patients with mild and common COVID-19 have a good prognosis, while the mortality of severe patients is 1.4~4.5% [13], and the mortality of critical patients is as high as 38.73% [14]. The prediction model is not only suitable for the allocation of limited medical resources, but also can help reduce mortality rates [15]. Compared with previously published death prediction models [4,5,7], our study mainly focused on patients with severe/critical COVID-19; we developed and externally validated a new practical model to identify patients at higher risk of death. The development and validation of the model should follow the requirements and recommendations in the Tripod statement [16]. Because the total sample size included in this study was small, with many independent variables and collinearity between different variables, Lasso regression analysis was used to screen potential independent variables. Compared with the currently commonly used stepwise regression method and optimal subset method, Lasso regression can reduce the dimension of multiple parameters and provide a solution to the collinearity problem of multiple parameters, avoiding overfitting and ensuring global optimization [17]. The model was composed of six variables, including having under three basic diseases, APACHE II score, Urea, Lac, L%, and NPR, which were obtained at admission and were all obtained from our data. The model was externally verified in the validation set, showing high discrimination and good prediction performance. Future studies may refine predictors by selecting other characteristic variables, such as CT scans, lymphocyte subtypes, organ injury markers, and cytokines.
All the predictive indicators are routine clinical testing items with fast detection speeds and low prices. Studies have found that advanced age, high APACHE II score, lymphopenia, high lactate dehydrogenase, and C-reactive protein are associated with poor prognosis [18,19]. A meta-analysis study found that the past medical histories of patients with COVID-19, including hypertension, chronic lung disease, and cardiovascular disease, may be risk factors for severe infection [20]. Our study found that certain factors, including patients with less than three previous diseases, APACHE II score, Urea, Lac, L%, and NPR, were closely related to the probability of COVID-19-related death. Different from previous studies [21], this study did not find serum amyloid A, lactate dehydrogenase, and C-reactive protein to be risk factors for death in patients with severe/critical COVID-19.
Our study found that the L% of patients who died of COVID-19 decreased significantly, indicating that the immune response might be involved in the progress of the disease. More and more evidence showed that cytokine storms play an important role in the process of disease progression to severe COVID-19 infection, and the cause of death might be related to excessive virus-induced inflammation [22]. However, we did not analyze cytokines in this study due to the lack of data. It is known that the presence of more than three basic diseases is a risk factor for COVID-19-related death, which may be related to the aggravation of inflammatory response, acute stress, and hypoxemia caused by immune dysfunction. However, the selection of immunosuppressants, such as glucocorticoids, may effectively prevent the excessive inflammation caused by viruses and selectively reduce the mortality of severe patients.
The data used in this study were those of severe patients, so this model is suitable for severely affected adult patients. Patients with low and medium death risk are suitable for the general ward, while patients with high risk of death may need to be monitored and treated in ICUs. Our model is helpful for identifying patients with high risk of death, and early intervention and appropriate treatment decisions after early identification may improve the prognosis of such patients.
However, our study still has the following limitations: First, this was a cross-regional, double-center, retrospective, observational cohort study based in China, which may not represent COVID-19 patients in other countries due to racial/ethnic differences. Therefore, international data sets and other multicenter, prospective, large-sample studies are required for external verification. Second, this study was a retrospective study, and the data obtained from the electronic medical record system were not complete; therefore, we attempted to overcome the issues raised by the missing data using the random forest method. Third, we did not collect the follow-up drug treatment of patients, and the effects of different treatment options were not taken into account in the nomogram. Fourth, some indicators reported in the literature, such as CT scan, lymphocyte subtype, organ damage markers, and cytokines, were not evaluated in our nomogram due to the lack of data. Fifth, because none of the training set patients in this study were vaccinated against COVID-19, it was not possible to determine whether vaccination affected the mortality outcome. Finally, the laboratory data may change with the progress of the disease, and it was impossible to include the dynamic changes of various indicators in the model for analysis due to the nature of retrospective studies.

Conclusions
The nomogram model presented in this study was shown to have good predictive ability and discrimination and can provide a reference for early screening of severe COVID-19 patients with high risk of death. This study carried out both internal validation and external validation in combination with the epidemic situation in Shanghai. The model has high stability, reliability, and repeatability, and is worthy of clinical promotion and application.