Available Bleeding Scoring Systems Poorly Predict Major Bleeding in the Acute Phase of Pulmonary Embolism

We aimed to compare six available bleeding scores, in a real-life cohort, for prediction of major bleeding in the early phase of pulmonary embolism (PE). We recorded in-hospital characteristics of 2754 PE patients in a prospective observational multicenter cohort contributing 18,028 person-days follow-up. The VTE-BLEED (Venous Thrombo-Embolism Bleed), RIETE (Registro informatizado de la enfermedad tromboembólica en España; Computerized Registry of Patients with Venous Thromboembolism), ORBIT (Outcomes Registry for Better Informed Treatment), HEMORR2HAGES (Hepatic or Renal Disease, Ethanol Abuse, Malignancy, Older Age, Reduced Platelet Count or Function, Re-Bleeding, Hypertension, Anemia, Genetic Factors, Excessive Fall Risk and Stroke), ATRIA (Anticoagulation and Risk Factors in Atrial Fibrillation), and HAS-BLED (Hypertension, Abnormal Renal/Liver Function, Stroke, Bleeding History or Predisposition, Labile International Normalized Ratio, Elderly, Drugs/Alcohol) scores were assessed at baseline. International Society on Thrombosis and Haemostasis (ISTH)-defined bleeding events were independently adjudicated. Accuracy of the overall original 3-level and newly defined optimal 2-level outcome of the scores were evaluated and compared. We observed 82 first early major bleedings (3.0% (95% CI, 2.4–3.7)). The predictive power of bleeding scores was poor (Harrel’s C-index from 0.57 to 0.69). The RIETE score had numerically higher model fit and discrimination capacity but without reaching statistical significance versus the ORBIT, HEMORR2HAGES, and ATRIA scores. The VTE-BLEED and HAS-BLED scores had significantly lower C-index, integrated discrimination improvement, and net reclassification improvement compared to the others. The rate of observed early major bleeding in score-defined low-risk patients was high, between 15% and 34%. Current available scoring systems have insufficient accuracy to predict early major bleeding in patients with acute PE. The development of acute-PE-specific risk scores is needed to optimally target bleeding prevention strategies.


Introduction
Anticoagulation is the cornerstone of the treatment of pulmonary embolism (PE) and should be initiated promptly when PE is diagnosed or a high clinical suspicion exists.Anticoagulant therapy aims to reduce mortality, morbidity of thrombus extension, and recurrence [1].Moreover, 3% of patients present with a high-risk PE, and around 5% of intermediate-risk PE patients who develop secondary hemodynamic collapse require emergent advanced therapies, mostly by infusing systemic thrombolysis [2][3][4].
Bleeding events are the main drawback of antithrombotic therapies.Cohort studies report that the mortality linked to major bleeding events is up to 20%, i.e., twice as high as the rate of death from recurrent PE [5].Major bleeding was identified as a predictor of short and 1-year mortality [5][6][7] and occurred more frequently within the first 7 days [5,8].
A cohort study evaluating the impact of long-term dose adjustment of direct oral anticoagulants (DOACs) previously reported that physicians in charge were intuitively aware of patients' bleeding risk in the acute phase of PE [9].However, more standardized and reproducible approaches using bleeding scoring systems have been developed, which may help to define the optimal antithrombotic management.Two bleeding risk-prediction scores for patients with venous thromboembolism (VTE), RIETE (Registro informatizado de la enfermedad tromboembólica en España; Computerized Registry of Patients with Venous Thromboembolism) and VTE-BLEED (Venous Thrombo-Embolism Bleed) scores, have been proposed and externally validated [10,11].In addition, several bleeding scores for patients with atrial fibrillation (AF) are available (e.g., ORBIT (Outcomes Registry for Better Informed Treatment), HEMORR 2 HAGES (Hepatic or Renal Disease, Ethanol Abuse, Malignancy, Older Age, Reduced Platelet Count or Function, Re-Bleeding, Hypertension, Anemia, Genetic Factors, Excessive Fall Risk and Stroke), ATRIA (Anticoagulation and Risk Factors in Atrial Fibrillation), and HAS-BLED (Hypertension, Abnormal Renal/Liver Function, Stroke, Bleeding History or Predisposition, Labile International Normalized Ratio, Elderly, Drugs/Alcohol) scores [12][13][14][15].All these scores were built to assess bleeding risk in stable patients receiving long-term anticoagulation.Currently, few data are available regarding their ability to predict in-hospital major bleeding in the context of acute PE [6,16].
Therefore, we aimed to externally validate and compare the predictive value of the RIETE, VTE-BLEED, ORBIT, HAEMORR 2 HAGES, ATRIA, and HAS-BLED bleeding scores for the occurrence of major bleeding during the hospital stay of acute PE patients.

Materials and Methods
This cohort study is a non-interventional retrospective post hoc analysis based on prospectively collected data from five French centers (two tertiary care facilities and three general hospitals) between January 2011 and September 2019 and recorded in the BFC-FRANCE registry [17].This registry received approval from the national commission for data privacy and protection.This study was conducted in accordance with the amended Declaration of Helsinki.All patients provided written informed consent, and our institutional review board approved the study.We report the study methods and results in accordance with the STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) guidelines [18].

Patients and Setting
We prospectively recorded all consecutive patients ≥18 years with a confirmed diagnosis of PE by computed tomography pulmonary angiography (CTPA) or ventilationperfusion (V-Q) scan.For confirmation of the diagnosis of PE, we required an intraluminal filling defect on CTPA [19], or a high probability V-Q scan according to the prospective investigation of pulmonary embolism diagnosis (PIOPED) criteria [20].There were no exclusion criteria.Management was at the discretion of the physician in charge and was in accordance with current guidelines [3,21,22].Anticoagulant therapy included parenteral anticoagulant (i.e., unfractionated heparin, low molecular weight heparin, and fondaparinux) and oral anticoagulant (i.e., vitamin K antagonist (VKA) and direct oral anticoagulant (DOAC)).Reperfusion therapy included systemic thrombolysis and surgical embolectomy.Pulmonary embolism was risk stratified according to the European Society of Cardiology (ESC) guidelines [3].

Bleeding Definition
Early bleeding was defined as a bleeding event that occurred during the hospital stay (i.e., between PE diagnosis and hospital discharge).Major bleeding was defined according to the definition proposed by the Control of Anticoagulation Subcommittee of the International Society on Thrombosis and Hemostasis (ISTH): (1) fatal bleeding, and/or (2) symptomatic bleeding in a critical area or organ, such as intracranial, intraspinal, intraocular, retroperitoneal, intra-articular, pericardial, or intramuscular with compartment syndrome, and/or (3) bleeding causing a drop of hemoglobin level of 20 g/L or more, or leading to transfusion of two or more units of red blood cells [23].All bleeding events were classified by a central adjudication committee (CM and RC).An independent data safety monitoring board periodically reviewed the outcome.Disagreement was resolved by a third author (NM).

Bleeding Predicting Scores
Based on a critical review of the literature, six different bleeding prediction scores were selected and calculated in all study patients at baseline: the RIETE score [11], the VTE-BLED score [10], the ORBIT score [15], the HAEMORR 2 HAGES score [14], the ATRIA score [12], and the HAS-BLEED score [13].All scores but one, were calculated prospectively (the VTE-BLED score was developed in 2016 and was calculated retrospectively between 2011 and 2016).Since CYP 2C9 single-nucleotide polymorphisms were not assessed as part of this study, all patients were scored 0 points for this item in the HEMORR 2 HAGES [14] score.For calculation of the HAS-BLED score, all patients were scored with 0 points for "labile INR" since therapeutic anticoagulation with VKA was not initiated yet at baseline [13].OR-BIT, HEMORR 2 HAGES, ATRIA, HAS-BLED, VTE-BLEED, and RIETE scores and staging systems for risk of major bleeding complications are provided in Supplementary Table S1.

Statistical Analysis
Continuous variables are expressed as mean ± standard deviation.Categorical variables are expressed as number (percentage).Unadjusted differences between patients who experienced in-hospital major bleeding and those who did not were compared using the chi-square test for categorical variables and Student's t-test for continuous variables.The use of multiple imputation was not required as the rate of missing data was <1% for all covariates [24].The potential for covariate multiple collinearity was tested using the variance inflation factor (VIF) and condition number (CN), with VIF < 10 and CN < 30 as reference values [25].The cumulative rate of a first major bleeding event was illustrated using the Kaplan-Meier method.Independent predictors of in-hospital major bleeding, inhospital mortality, and length of stay were determined by multivariable logistic regression, adjusted for baseline characteristics, and in-hospital therapies that yielded a p value < 0.10 by univariable analysis.Results are reported as odds ratio (OR) with 95% confidence interval (CI).The relationship between dichotomized bleeding risk scores and in-hospital major bleeding was also analyzed with logistic regressions.
The global model fit of the six bleeding risk scores was assessed by calculation of Nagelkerke's R 2 , the Bayes information criterion (BIC), and the Akaike information criterion (AIC).Discrimination of models was evaluated by Harrell's C-index [26].Receiver operating characteristics (ROC) curves illustrated discriminative capacities of each model.Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of each model were derived from the ROC curves.Model calibrations were assessed visually by plotting the mean of model-predicted in-hospital major bleeding in each decile of predicted in-hospital major bleeding against the observed in-hospital major bleeding estimated by the Kaplan-Meier method.The original RIETE, ORBIT, HAEMORR 2 HAGES, ATRIA, and HAS-BLEED scores used 3-level categories: low-, intermediate-and high-risk.The RIETE score did not classify any patient as low-risk since all patients received one point for PE diagnosis.The VTE-BLED score was not developed as a 3-level category.To enable a more practical 2-level interpretation, i.e., low-and high-risk categories, an optimal threshold was determined by ROC curve analyses.
To determine the most accurate bleeding risk model for the prediction of in-hospital major bleeding in acute PE, we compared Harrell's C-indices using the approach proposed by Kang et al. as well as the net reclassification improvement (NRI) and the integrated discrimination improvement (IDI) between models [27,28].To assess the robustness of the findings, we performed sensitivity analyses by estimating the c-indices of the bleeding scores in the following subgroups: (1) non-high risk PE patients, and (2) patients who did not receive reperfusion therapy (i.e., systemic thrombolysis or surgical embolectomy).
A p value < 0.05 was considered significant.Analyses were performed using SAS 9.4 (SAS institute Inc., Cary, NC, USA).

Results
In total, 2757 patients were admitted to the participating centers during the study period with an objective diagnosis of acute PE.In-hospital data were not recorded for 3 patients (0.1%).The remaining 2754 patients comprised the study population.Mean age was 67.3 ± 17.4 years; 1414 (51.3%) were women.One hundred and thirty-three patients (4.8%) had high-risk PE, 584 patients (21.2%) intermediate-high risk PE, 1594 patients (57.9%) intermediate-low risk PE, and 443 patients (16.1%) low-risk PE.

Major Bleeding Events
During the in-hospital stay, 82 patients (3.0%; 95% CI, 2.4-3.7)had major bleeding with a median time to event of 2.8 days (Q1-Q3, 1.2-3.9,ranging from 1 to 21 days).Figure 1 illustrates the cumulative time from admission to major bleeding.Bleeding events were classified according to the ISTH definition as major because of the occurrence of at least one of the following criteria: bleeding-related death, 9 patients (10.9%), symptomatic bleeding in a critical area or organ, 28 patients (34.1%), bleeding requiring surgery, 13 patients (15.8%), bleeding causing a fall in hemoglobin level of 2.0 g/dL, 58 patients (70.7%), and bleeding leading to transfusion of two or more units of whole blood or red cells, 48 patients (58.5%).Bleeding in a critical area or organ was intracranial for 18 patients (0.6%), intraspinal for 1 patient (0.03%), intraocular for 1 patient (0.03%), retroperitoneal for 2 patients (0.06%), and intramuscular for 6 patients (0.21%).Overall, patients who suffered early bleeding were more frequently women, had a more severe hemodynamic profile, RV dysfunction, and positive troponin resulting in a higher sPESI (simplified Pulmonary Embolism Severity Index) and more severe ESCdefined risk stratification (Table 1).Among patients with a bleeding event, 1.2% were treated with DOAC, 19.5% with VKA, 68.3% with parenteral anticoagulant, and 11.0% with systemic thrombolysis.Concomitant medication usage predisposing to bleeding, syncope, heart rate > 80 b.p.m, renal dysfunction and anemia at admission were factors related to early major bleeding after multivariable adjustment (Table 2).The occurrence of early major bleeding was related to a longer adjusted length of stay (OR, 4.2; 95% CI, 2.9-5.8) and a higher rate of adjusted in-hospital mortality (OR, 8.4; 95% CI, 4.0-17.6)(Table 2).

Bleeding Scores and Prediction of Early Major Bleeding
Median (Q1-Q3) bleeding risk scores in the study population were as follows: RIETE, 3 (2-4); VTE-BLEED, 2.5 (1.5-3.5);ORBIT, 1 (0-2); HAEMORR 2 HAGES, 1 (0-1); ATRIA, 1 (0-3); and HAS-BLED, 1 (0-1) (Table 1).Using the original 3-level risk categories, the ORBIT, HAEMORR 2 HAGES, and ATRIA scores classified the majority of patients in the low-risk category (70.5-86.1%)whereas the RIETE and the HAS-BLED classified the majority of patients in the intermediate category (78.1% and 65.5%, respectively).All but one (i.e., the ORBIT score) were able to categorize patients with increasing rates of observed early major bleeding across the 3-level categories.The rate of observed major bleeding ranged between 1.8% and 2.4% in the low-risk categories, 2.3% and 7.7% in the intermediaterisk categories, and 5.3% and 7.0% in the high-risk categories.After dichotomization of the six bleeding scores into 2-level risk categories, all scores were able to distinguish between low and high risk of observed early major bleeding (Table 3).Overall, the RIETE score had the best global model fit with the lowest BIC and AIC and the highest Nagelkerke's R 2 .Harrell's c index ranged from 0.570 with the HAS-BLED to 0.692 with the RIETE score (Table 4).Figure 2 displays the ROC curves of the six early major bleeding scores as well as their corresponding sensitivity, specificity, PPV, and NPV based on the 2-level risk categories.The RIETE score had a numerically higher Harrell's c-index than the HAEMORR 2 HAGES, ORBIT, ATRIA, and VTE-BLEED bleeding scores.Reclassification parameters (i.e., IDI and NRI) were significantly higher with the RIETE, ORBIT, and ATRIA scores than the others.The HAS-BLED score had the lowest discriminatory and reclassification capacities (Table 5).All bleeding scores were not well calibrated with the predicted risks and their confidence intervals were not distributed around the observed early bleeding risks (Figure 3).ROC assessment for dichotomy isolation of the bleeding risk scores are displayed in Supplementary Figure S1.The 2-level RIETE, ORBIT, ATRIA, and HAEMORR 2 HAGES scores were independently associated with early major bleeding after multi-variable adjustment (Figure 4).

Sensitivity Analysis
In total, 2621 patients (95.2%) had non-high risk PE and 120 patients (4.3%) were treated with advanced therapy (3.8% with systemic thrombolysis, and 0.5% with sur-gical embolectomy).Discrimination performances of all bleeding scores were similar across patients with non-high risk PE and those who did not receive advanced therapy (Supplementary Figure S2).

Discussion
In our multicenter cohort analysis, the RIETE score had better global fit and higher discriminatory and reclassification capacities compared to the other available bleedingprediction scores for the assessment of early major bleeding risk after an acute PE.However, the accuracy of the RIETE score remains low with corresponding sensitivity and specificity of 65.8% and 66.4%, respectively, generating high rates of both false positives and false negatives in the bleeding risk appraisal.Since bleeding risk assessment is of importance in the acute phase of PE with a well-demonstrated close relationship between bleeding events and early mortality [5][6][7], the development of a dedicated early risk score is crucial.
The RIETE score was validated in 15,206 patients from the RIETE registry treated with three months of anticoagulation for the treatment of PE with a fair c-index of 0.719 (95% CI, 0.689-0.749)[29].To the best of our knowledge, the present cohort study is the largest evaluating performances of available bleeding risk scores for the prediction of in-hospital bleeding.Our results are consistent with and strengthen those reported by Klok et al., who showed low risk prediction accuracy of the Kuijer, the RIETE, the HAEMORR 2 HAGES, the HAS-BLED, and the ATRIA scores with c-indices ranging from 0.57 to 0.64 (c-index for the RIETE score, 0.60 (95 % CI, 0.47-0.72)) in 655 patients from the single-center PERGO registry [16].Our results regarding prediction performance of the VTE-BLEED score are the opposite of those recently reported in 655 patients.In this single center cohort, the authors observed a higher c-index (i.e., 0.69 (95% CI, 0.58-0.80))than ours for the discrimination capacity of this score to predict in-hospital major bleeding, as well as an independent relationship between VTE-BLEED score and in-hospital bleeding events after multivariable adjustment.Differences in study design (i.e., exclusion of patients receiving reperfusion therapy in the aforementioned study) and a nearly 4-fold lower sample size may explain these discrepancies [6].
We observed almost similar discrimination capacity of the ORBIT score as compared to the RIETE score in our analysis (Harrell's c-index, 0.681 vs 0.692), with a high sensitivity of the ORBIT score when collapsed into 2-level categories (84.1%).Nevertheless, the associated low specificity of 43.8% renders the ORBIT score useless in clinical practice with a related high rate of false positives and a corresponding low rate of true negatives for low-risk bleeding risk prediction.Nevertheless, bleeding risk scores in the acute phase of PE should probably not be derived from AF populations as patient characteristics differ widely from PE patient characteristics.For instance, mean age was 69.8 years in 17,162 AF patients versus 60.2 years in 11,842 VTE patients in large international registries [30,31].
The accurate identification of acute PE patients at high risk of bleeding with the use of bleeding prediction scores, together with individualized decision-making may prompt alternative therapeutic strategies.Low molecular weight heparin may be a preferred option rather than unfractionated heparin in high bleeding risk patients, to avoid supratherapeutic anticoagulation when advanced therapy is planned [32].Direct oral anticoagulants have been shown to be associated with a lower risk of bleeding than the standard heparin and vitamin K antagonist regimen [33].The bleeding risk of patients treated with systemic thrombolysis can potentially be overcome by the use of alternative reperfusion strategies, such as ultrasound-facilitated catheter-based therapy or surgical embolectomy [34][35][36][37].The 2019 ESC guidelines recommend inferior vena cava filter implantation for patients with an absolute contraindication to anticoagulant therapy, based on a lower risk of recurrent PE over the first month compared with patients not receiving this device [3,38].Finally, the identification of high bleeding risk patients should prompt providers to mitigate other modifiable risk factors such as concomitant anti-platelet therapy or hypertension [39].The recent development of a dedicated in-hospital risk score may help to fill this gap [40].
The strengths of the present study include the prospective patient recording in different centers, the high rate of consecutive inclusions (99.9%), the independent adjudication of clinical end-points, and the robustness of statistical approaches.In contrast, therapeutic decision making was left to the discretion of the treating physicians.Thus, the type of initial anticoagulation and measures for anticoagulation quality control were not standardized.
We similarly applied the ISTH criteria for the definition of major bleeding events and did not evaluate other bleeding definitions such as the GUSTO or CRUSADE definitions [41,42].Finally, although the rate of events (specifically major bleeding) in the overall population was low, which may be a limitation, it is nonetheless similar to rates reported in other publications [6,16].

Conclusions
Among six available bleeding risk scores, the RIETE score had the best performance profile.However, the accuracy of the RIETE score remains low, generating high rates of both false positives and false negatives in the bleeding risk appraisal.

Figure 1 .
Figure 1.Cumulative rate of a first major bleeding event.

Figure 2 .
Figure 2. Receiving operator curves analyses of the six-bleeding risk scores (A) and their corresponding sensitivity, specificity, positive predicting value, and negative predictive value based on the adjusted-threshold 2-level categories (B).PPV: positive predictive value; NPV: negative predictive value.VTE-BLEED score: Venous Thrombo-Embolism Bleed; RIETE: Registro informatizado de la enfermedad tromboembólica en España; Computerized Registry of Patients with Venous Thromboembolism; ORBIT: Outcomes Registry for Better Informed Treatment; HAEMORR 2 HAGES score: Hepatic or Renal Disease, Ethanol Abuse, Malignancy, Older Age, Reduced Platelet Count or Function, Re-Bleeding, Hypertension, Anemia, Genetic Factors, Excessive Fall Risk and Stroke; ATRIA score: Anticoagulation and Risk Factors in Atrial Fibrillation; HAS-BLED score: Hypertension, Abnormal Renal/Liver Function, Stroke, Bleeding History or Predisposition, Labile International Normalized Ratio, Elderly, Drugs/Alcohol.

Figure 3 .
Figure 3. Decile calibration plots of six in-hospital major bleeding prediction risk scores.All six scores were not well calibrated with the predicted risks and their confidence intervals were not distributed around the observed in-hospital bleeding risks.(A) VTE-BLEED score; (B) RIETE score; (C) ORBIT score; (D) HAEMORR 2 HAGES score; (E) ATRIA score; (F) HAS-BLED score.

Figure 4 .
Figure 4. Adjusted-threshold 2-level category major bleeding scores as predictors of in-hospital major bleeding after multivariable adjustment.OR: odds ratio; CI: confidence interval.

Table 1 .
Baseline characteristics and in-hospital management of 2754 study patients according to the occurrence or not of early major bleeding.

Table 1 .
Cont.Registro informatizado de la enfermedad tromboembólica en España; Computerized Registry of Patients with Venous Thromboembolism; ORBIT: Outcomes Registry for Better Informed Treatment; HAEMORR 2 HAGES score: Hepatic or Renal Disease, Ethanol Abuse, Malignancy, Older Age, Reduced Platelet Count or Function, Re-Bleeding, Hypertension, Anemia, Genetic Factors, Excessive Fall Risk and Stroke; ATRIA score: Anticoagulation and Risk Factors in Atrial Fibrillation; HAS-BLED score: Hypertension, Abnormal Renal/Liver Function, Stroke, Bleeding History or Predisposition, Labile International Normalized Ratio, Elderly, Drugs/Alcohol; ESC-defined risk PE category: pulmonary embolism risk category according to the guidelines of the European Society of Cardiology.
BMI: body mass index; VTE: venous thromboembolism; DVT: deep vein thrombosis; HR: heart rate; b.p.m: beat per minute; SBP: systolic blood pressure; Sa: oxygen saturation; eGRF CKD-EPI : estimated glomerular function by using the Chronic Kidney Disease Epidemiology Collaboration equation; RV: right ventricle; sPESI: simplified Pulmonary Embolism Severity Index; UFH: unfractionated heparin; LMWH: low molecular weight heparin; DOAC: direct oral anticoagulant; ECMO: extra-corporeal membrane oxygenation.a Active or anti-tumor therapy within the last 6 months, or metastatic state according to the 2019 European Society of Cardiology guidelines.b Within the past 4 weeks.c Antiplatelet therapy, non-steroidal anti-inflammatory drug.VTE-BLEED score: Venous Thrombo-Embolism Bleed; RIETE:

Table 2 .
Univariable and multivariate predictors of early major bleeding, length of stay and inhospital all-cause mortality.
CI: confidence interval; OR: odds ratio; BMI: body mass index; RV: right ventricle.a Within the past 4 weeks.b Antiplatelet therapy, non-steroidal anti-inflammatory drug.c Glomerular filtration rate calculated with the Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) formula.d Defined as a value > 99th percentile of healthy subjects with a coefficient of variation of 10%.e Defined by the presence of at least one of the following on echography: increased end-diastolic right ventricle/left ventricle diameter > 1.0 in the apical four-chamber view, flattened intraventricular septum, decrease tricuspid annular plane systolic excursion < 16 mm, or right heart thrombus detected in right heart cavities.

Table 3 .
Frequency of observed early major bleeding according to the risk categories of bleeding prediction scores.

Table 4 .
Global model fit and discrimination of bleeding scores for the prediction of early major bleeding.

Table 5 .
C-statistics, integrated discrimination improvement, and net reclassification improvement of bleeding scores for in-hospital major bleeding discrimination and reclassification.