Next Article in Journal
Global Expression Profiling Identifies a Novel Hyaluronan Synthases 2 Gene in the Pathogenesis of Lower Extremity Varicose Veins
Previous Article in Journal
Systematic Analysis of Transcriptomic Profile of Chondrocytes in Osteoarthritic Knee Using Next-Generation Sequencing and Bioinformatics
 
 
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Validation of the REGARDS Severe Sepsis Risk Score

1
Department of Emergency Medicine, The University of Texas Health Science Center at Houston, Houston, TX 77030, USA
2
Department of Learning Health Sciences, University of Michigan, Ann Arbor, MI 48109, USA
3
Department of Critical Care Medicine, University of Pittsburgh, Veterans Affairs Pittsburgh Healthcare System, Pittsburgh, PA 15213, USA
4
Department of Epidemiology, University of Alabama at Birmingham, Birmingham, AL 35294, USA
5
Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Boston, MA 02215, USA
6
Department of Epidemiology, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, USA
7
Department of Health Research and Policy, Stanford University, Stanford, CA 94305, USA
8
Department of Epidemiology, University of Pittsburgh, Pittsburgh, PA 15213, USA
9
Department of Medicine, Weill-Cornell Medical Center, New York, NY 10065, USA
*
Author to whom correspondence should be addressed.
J. Clin. Med. 2018, 7(12), 536; https://doi.org/10.3390/jcm7120536
Received: 5 November 2018 / Revised: 4 December 2018 / Accepted: 7 December 2018 / Published: 11 December 2018
(This article belongs to the Section Hematology)

Abstract

:
There are no validated systems for characterizing long-term risk of severe sepsis in community-dwelling adults. We tested the ability of the REasons for Geographic and Racial Differences in Stroke-Severe Sepsis Risk Score (REGARDS-SSRS) to predict 10-year severe sepsis risk in separate cohorts of community-dwelling adults. We internally tested the REGARDS-SSRS on the REGARDS-Medicare subcohort. We then externally validated the REGARDS-SSRS using (1) the Cardiovascular Health Study (CHS) and (2) the Atherosclerosis Risk in Communities (ARIC) cohorts. Participants included community-dwelling adults: REGARDS-Medicare, age ≥65 years, n = 9522; CHS, age ≥65 years, n = 5888; ARIC, age 45–64 years, n = 11,584. The primary exposure was 10-year severe sepsis risk, predicted by the REGARDS-SSRS from participant sociodemographics, health behaviors, chronic medical conditions and select biomarkers. The primary outcome was first severe sepsis hospitalizations, defined as the concurrent presence of ICD-9 discharge diagnoses for a serious infection and organ dysfunction. Median SSRS in the cohorts were: REGARDS-Medicare 11 points (IQR 7–16), CHS 10 (IQR 6–15), ARIC 7 (IQR 5–10). Severe sepsis incidence rates were: REGARDS-Medicare 30.7 per 1000 person-years (95% CI: 29.2–32.2); CHS 11.9 (10.9–12.9); ARIC 6.8 (6.3–7.3). SSRS discrimination for first severe sepsis events were: REGARDS-Medicare C-statistic 0.704 (95% CI: 0.691–0.718), CHS 0.696 (0.675–0.716), ARIC 0.697 (0.677–0.716). The REGARDS-SRSS may potentially play a role in identifying community-dwelling adults at high severe sepsis risk.

1. Introduction

The sepsis syndrome is a major public health problem responsible for over 850,000 emergency department visits, 200,000 deaths and $16.7 billion in medical expenditures in the United States (US) each year [1,2,3,4]. Current scientific and clinical initiatives focus almost entirely on the early recognition and treatment of acute sepsis [5,6]. However, the onset of acute disease is often influenced by underlying precursors. For example, acute myocardial infarction and stroke were once viewed as acute conditions triggered by random exposures or processes that could not be anticipated or prevented. The recognition of myocardial infarction and stroke as the end result of underlying chronic medical conditions led to new strategies of cardiovascular risk prediction, management and prevention, contributing to dramatic declines in heart disease and stroke mortality [7,8,9]. Like myocardial infarction and stroke in the past, relatively little attention has been paid to the factors that may heighten an individual’s risk of acute sepsis. An understanding of the factors that influence an individual’s baseline sepsis risk could provide new opportunities for mitigating the societal burden of sepsis [10].
Fundamental steps in disease prevention include identifying vulnerable individuals and quantifying their degree of risk. In prior studies, we used the population-based REasons for Geographic and Racial Differences in Stroke (REGARDS) cohort to show that sepsis hospitalizations are predictable and associated with sociodemographic factors, health behaviors, chronic diseases, and perturbations in the immune system [11]. We used these findings to derive a risk prediction rule—the REGARDS Severe Sepsis Risk Score (REGARDS-SSRS)—quantifying the 10-year risk of severe sepsis hospitalizations [12] (Table 1 and Appendix Table A1).
To validate their accuracy, robustness and generalizability, risk prediction rules must be tested in independent cohorts [13]. In this study we first tested the ability of the REGARDS-SSRS to identify severe sepsis events in a subcohort of REGARDS participants enrolled in Medicare. We then sought to externally validate the REGARDS-SSRS in two large population-based cohorts: The Cardiovascular Health Study (CHS) and the Atherosclerotic Risk in Communities (ARIC) Study.

2. Experimental Section

2.1. Study Design and Overview

We used a population-based cohort design. This study was approved by the Institutional Review Boards of the University of Alabama at Birmingham and the University of Texas Health Science Center at Houston.
While we originally derived the REGARDS-SSRS using severe sepsis events identified by adjudicated review of medical records, there were no other existing cohorts with similarly identified severe sepsis events available for validation of the prediction rule. However, many population-based cohorts have discharge diagnoses available for all hospitalizations, providing an alternate strategy for identifying severe sepsis events. We therefore developed strategies to validate the REGARDS-SSRS in discharge-diagnosis based data sets.
As an important initial step, we first internally applied the SSRS to the REGARDS-Medicare cohort; this subcohort linked Medicare-enrolled REGARDS participants with their respective Medicare claims. This initial analysis allowed us to evaluate the ability of the SRSS to predict sepsis cases in the REGARDS cohort using hospital discharge diagnoses. We then sought to externally validate the SSRS using the CHS and ARIC cohorts, two of the most widely recognized longitudinal population-based cohorts in the US.

2.2. Study Setting and Participants—The REGARDS-Medicare, CHS and ARIC Cohorts

REGARDS is one of the nation’s largest ongoing population-based cohorts [14]. Consisting of 30,239 community-dwelling adults aged ≥45 years from across the United States, the REGARDS cohort was designed to evaluate the predictors of racial and geographic differences in stroke mortality. Recruitment of REGARDS participants occurred between January 2003 and October 2007 with ascertainment of baseline participant information including medical history, functional status, health behaviors, physical characteristics, physiologic measures, current medications, diet, family history of diseases, psychosocial factors and prior residences. The study also obtained comprehensive biologic specimens at the time of enrollment. Among REGARDS participants, 42% are African American, 45% are male, and 69% are over 60 years old.
The REGARDS-Medicare subcohort consists of 9,522 REGARDS participants who were enrolled in Medicare during 2003–2012 [15,16,17,18]. For this study, we limited the analysis to REGARDS participants age ≥65 years enrolled in Medicare Parts A (inpatient services) and B (outpatient services) throughout the entire REGARDS follow-up period 2003–2012. We excluded those with Medicare Part C coverage, who transitioned into or out of Medicare coverage during the study period. The parent REGARDS study identified hospitalizations through adjudicated chart review of medical records for each participant-reported health event. In contrast, the REGARDS-Medicare subcohort identified hospitalization events using discharge diagnoses for each claim. The REGARDS-Medicare linkage strategy has been previously described [18,19,20]. The characteristics of the REGARDS-Medicare subcohort are similar to participants in the Medicare national sample [18]. While the REGARDS-Medicare subcohort contains individuals used in the original SSRS derivation, the method of severe sepsis event identification is different, entailing use of discharge diagnoses. REGARDS included individuals ≥45 years old while Medicare was limited to persons ≥65 years old; we accepted this limitation as there was no large data source characterizing insurance claims for all REGARDS participants.
The Cardiovascular Health Study (CHS) is a population-based longitudinal study of coronary heart disease and stroke in adults aged ≥65 years [21]. Since 1987 the cohort CHS recruited a total of 5888 participants from Forsyth County (North Carolina), Sacramento County (California), Washington County (Maryland) and Pittsburgh (Pennsylvania). The study conducted extensive physical and laboratory evaluations at baseline. Follow-up of participants entailed semi-annual clinic examination (enrollment through 1999) and phone contact (enrollment through present). Identification of hospitalization events in CHS occurred through semi-annual interviews, additional self-reports by participants, and periodic surveillance of Medicare records. The study obtained discharge summaries and diagnoses for all hospitalizations.
The Atherosclerosis Risk in Communities Study (ARIC) is a prospective epidemiologic study conducted in Forsyth County (North Carolina), Jackson (Mississippi), Minneapolis (Minnesota), and Washington County (Maryland) [22]. In 1987 ARIC recruited a cohort a total of 15,792 individuals aged 45–64 years, conducting an extensive examination, and determination of medical, social, and demographic data. Study personnel abstracted hospitalization records (including diagnosis and procedure codes) for events identified through annual telephone contact with study participants or surveillance in study hospitals.
We chose CHS and ARIC cohorts because of their longitudinal design, completeness of follow-up, the availability of comprehensive discharge diagnoses for all hospitalizations, and the depth of available baseline variables and biomarkers needed to calculate the SSRS for each participant.

2.3. Exposure—Calculation of REGARDS-Severe Sepsis Risk Score

The REGARDS-Severe Sepsis Risk Score (SSRS) characterizes long-term severe sepsis risk using combinations of individual sociodemographics, health behaviors, chronic medical conditions, and biomarkers. (Table 1) Derivation and internal validation of the SSRS has been previously described [12]. Briefly, for each participant reported serious infection hospitalizations, two trained abstractors independently reviewed the corresponding medical records for the first 28-h of hospitalization to confirm the presence of a serious infection, sepsis and organ dysfunction. Using Cox regression, we identified clinical characteristics associated with severe sepsis events, converting the regression model into a 0–45 point severe sepsis risk score (SSRS).
While our original derivation effort included a prediction model for sepsis events (Appendix Table A1), for this study we focused on the model identifying severe sepsis events because of its higher discrimination and clearer link with the contemporary definition of sepsis as infection plus organ dysfunction [23]. We also focused on the version of the SSRS incorporating biomarkers (hsCRP, Cystatin-C, eGFR, ACR) because of its higher discrimination.
We calculated REGARDS-SSRS scores for each participant in the REGARDS-Medicare, CHS and ARIC cohorts using baseline comorbidity data available for each participant. For CHS and ARIC, we used previously published serum creatinine adjustments in order to calibrate these values for use with the CKD-EPI equation in the calculation of estimated glomerular filtration rates (eGFR) [24]. Baseline serum creatinine values for REGARDS participants were previously IDMS standardized [25].
For the ARIC cohort, ACR and Cystatin-C were not available until participant visit (examination) No. 4 (1996–1998). Therefore, for ARIC, we used data collected at visit 4 to calculate risk score points. The visit 4 examination date was used as the baseline for all ARIC participants in calculations of time to sepsis, with individuals excluded if the visit 4 examination was not completed for any reason or if sepsis occurred before this date.

2.4. Outcomes—Identification of Severe Sepsis Events

The primary outcome was first severe sepsis hospitalization. Following the method of Angus, et al., we defined severe sepsis as hospitalizations with the concurrent presence of discharge diagnoses for a serious infection and organ dysfunction [2] (Table A2 and Appendix Table A3). We identified severe sepsis events during the following follow-up periods: REGARDS-Medicare 2003–2012, CHS 1989–2002 and ARIC 1996–2008. Although ARIC enrolled subjects starting in 1987, we limited the analysis to sepsis events identified during 1996–2008 because certain exposure variables (ACR, Cystatin-C and history of atrial fibrillation) were not available until participant examination No. 4 (1996–1998).

2.5. Data Analysis

We compared participant characteristics between the REGARDS-Medicare, CHS and ARIC cohorts. We assessed the distribution of the SSRS in each cohort using quantile plots and histograms.
To test the ability of the SSRS to detect discharge-diagnosis-identified severe sepsis events, we first internally applied the SSRS to the REGARDS-Medicare subcohort. We used the original SSRS model and point system—we did not derive a new risk score for the analysis. Using Cox proportional hazards models, we determined severe sepsis incidence and relative hazards across risk groups (very low, low, medium, high, and very high, as previously reported in the initial SSRS derivation effort). To assess calibration, we conducted a validation in the manner described by Royston [26]. We estimated predicted and observed survival functions for SSRS risk groups from Cox models, with the smoothed baseline function estimated using fractional polynomials. We assessed goodness of fit using the Groennesby and Borgan score test (Stata ‘stcoxgof’ command) and reported the observed and predicted number of events over predicted risk deciles [27]. We determined discrimination using Harrell’s C-statistic [28]. Lastly, we reported the percent positive, the true positive rate, and the true negative rate over the full ten years of follow-up.
We then externally validated application of the SSRS to the ARIC and CHS data sets. We followed a similar strategy as with the REGARDS-Medicare subcohort, examining incidence and relative hazards across risk groups, observed and predicted survival functions and events, goodness-of-fit and discrimination. For the ARIC cohort we excluded participants who died or experienced severe sepsis events prior to 1996.
In the primary analysis, we explicitly coded missing exposure variables as “normal”. We conducted all analyses using Stata 13.1 (Stata, Inc., College Station, TX, USA).

3. Results

From the REGARDS-Medicare, CHS and ARIC cohorts, we included 9522, 5888, and 11,584 participants, respectively (Table 2). While the REGARDS-Medicare and CHS cohorts included only persons ≥65 years of age, 60.4% of the ARIC cohort participants were <65 years old at baseline (Table 2). The REGARDS-Medicare cohort contained a slightly higher proportion of males and blacks. Current tobacco use was more common in ARIC participants. The REGARDS-Medicare cohort exhibited a higher comorbid burden than the CHS and ARIC cohorts. History of atrial fibrillation, coronary artery disease, diabetes, deep vein thrombosis, hypertension and stroke were most prevalent in the REGARDS-Medicare cohort. Chronic lung disease was most prevalent in the CHS cohort. Peripheral artery disease was most prevalent in ARIC. Prevalence of obesity was similar between the cohorts. Among biomarkers, CHS participants exhibited a slightly higher prevalence of abnormal hsCRP and eGFR. Elevated Cystatin-C and ACR were most common among REGARDS-Medicare participants.
Median SSRS were: REGARDS-Medicare 11 points (IQR 7–16), CHS 10 (IQR 6–15), ARIC 7 (IQR 5–10). Quantile plot and histograms patterns suggested similar SSRS distributions were between the REGARDS-Medicare and CHS cohorts (Figure 1). The ARIC cohort generally exhibited lower SSRS risk scores. The number of events and incidence rates of severe sepsis events in each cohort were: REGARDS-Medicare, n = 1593, IR 30.7 per 1000 person-years (95% CI: 29.2–32.2); CHS n = 586, IR 11.9 (10.9–12.9); ARIC n = 735, IR 6.8 (6.3–7.3) (Table 3).
The SSRS showed adequate discrimination in the REGARDS-Medicare subcohort; C-statistic 0.704 (95% CI: 0.691–0.718) (Table 3). The SSRS showed good calibration in the REGARDS-Medicare subcohort across risk groups based on plots of observed and predicted survival functions (Figure 2). The SSRS also demonstrated good fit in the REGARDS-Medicare subcohort.
In the ARIC and CHS cohorts, the hazard ratios for severe sepsis events were similar across SSRS risk categories (Table 3). The SSRS showed adequate discrimination; CHS 0.696 (0.675–0.716), ARIC 0.697 (0.677–0.716) (Table 4). In ARIC and CHS, the SSRS showed good calibration across risk groups based on plots of observed and predicted survival functions (Figure 2). For the CHS cohort, the p-value for goodness of fit was statistically significant, but the observed and predicted number of events across deciles of predicted risk were similar for all cohorts. The application of the SSRS to ARIC demonstrated good fit. The observed and predicted number of events across deciles of predicted risk were similar for the ARIC and CHS cohorts (Table 4).
Across all cohorts, less than 6% of participants in the very low risk group developed severe sepsis over ten years of follow-up, while over 20% of participants in the very high risk group had an event (Table 5). In addition, less than 7% of all participants with severe sepsis in REGARDS-Medicare and CHS were classified as very low risk. In ARIC, a higher proportion of severe sepsis cases were classified as low risk, but this may not be comparable to the other cohorts due to differences in baseline risk. Less than 15% of participants without severe sepsis were classified as very high risk across all cohorts.

4. Discussion

We previously derived and internally validated the REGARDS-SRSS using severe sepsis events identified by adjudicated review of medical records [12]. In this study, we verified the ability of the REGARDS-SSRS to predict long-term severe sepsis events identified through Medicare claims in the REGARDS-Medicare subcohort. We also externally validated the SSRS using the CHS and ARIC cohorts. Our results illustrate the robustness and generalizability of the REGARDS-SSRS, supporting its potential application in the prediction of individual and community severe sepsis risk.
The potential application of the REGARDS-SSRS is in frameworks that may be unfamiliar to acute care practitioners. Existing decision rules predict sepsis or adverse outcomes among acutely hospitalized patients [29]. In contrast, the SSRS predicts long-term severe sepsis risk among community-dwelling adults. Our study underscores the concept that risk prediction and prevention could play pivotal roles in reducing the public health burden of sepsis [10]. For example, sepsis entails infection complicated by systemic inflammation and organ dysfunction, and individuals with chronic conditions may be less able to tolerate or respond to this stress. The SSRS related severe sepsis risk to the presence of chronic conditions such as chronic lung disease, peripheral artery disease, tobacco use, coronary artery disease, obesity, hypertension, deep vein thrombosis and chronic kidney disease; optimal management of these conditions might mitigate long-term sepsis risk. Statin therapy has been proposed for reduce cardiovascular risk among individuals with elevated hsCRP [30]. Similar approaches may be possible with sepsis.
Individuals with known high SSRS risk may merit personalized approaches to prevention, acute sepsis care and even health education; for example, these individuals may receive increased emphasis on vaccines and tailored education on recognizing early signs of sepsis. The SSRS may also have important public health applications, identifying community groups or clusters for organized deployment of sepsis prevention measures such as those described above. Additional study must determine if modification of these risk factors in fact alters sepsis risk.
An important aspect of our study was the application of the REGARDS-SSRS to the detection of sepsis using hospital discharge diagnoses. We originally derived the SSRS using severe sepsis events identified through structured medical record review and adjudication. While best practices would have entailed external validation with an analogously structured data set, there are presently no cohorts with severe sepsis events identified in a similar manner. Instead we validated a pragmatic and efficient alternate approach using severe sepsis events identified through Medicare claims and discharge diagnoses. This study highlights the robust nature of the REGARDS-SSRS. National health care data sets in Denmark and Taiwan have been used to characterize sepsis epidemiology in these respective countries [31,32]. Our findings suggest that potential additional inquiry or validation of the SSRS could occur using these other claims- or discharge-diagnosis based data sets.
Experts customarily recommend that clinical decision rules exhibit a discrimination (C-statistic) value of at least 0.70 on derivation and validation [28,33]. The C-statistic values observed in this study (REGARDS-Medicare 0.704, CHS 0.696, ARIC 0.697) fell just short of this threshold. One potential explanation may entail the inherent limitations of the Angus severe sepsis criteria. Discharge diagnoses do not incorporate physiologic or laboratory measures and may be affected by biases in the documented discharge diagnoses. The Angus taxonomy presumes a clinical connection between a serious infection diagnosis and an organ dysfunction diagnosis. While the original SSRS validation identified only community-acquired severe sepsis events, discharge diagnoses cannot differentiate between community-acquired, hospital-acquired or healthcare-associated severe sepsis. In a separate effort using a hospital discharge data set with “present-on-admission” flags for each diagnosis, we found that Angus severe sepsis hospitalizations encompassed approximately 62.8% community-acquired, 25.9% hospital-acquired and 11.3% healthcare associated severe sepsis cases [34]. Additional opportunities for improving the discrimination of the SSRS may encompass the identification of additional relevant clinical variables or biomarkers.
This study has limitations. Our original efforts resulted in separate risk prediction models for both sepsis (infection + ≥2 systemic inflammatory response syndrome criteria) and severe sepsis (sepsis + ≥1 SOFA organ failure point) [12]. In the current study we focused on severe sepsis as the primary endpoint because it best aligned with the discharge data available in the test cohorts. Furthermore, current Sepsis-3 consensus guidelines advocate defining sepsis as the combination of [infection + organ dysfunction], a definition that closely aligns with the definition for severe sepsis [23]. We previously noted that the predictors of the REGARDS-SRS and SSRS were nearly identical, and that 70% of the derivation sepsis events also fulfilled severe sepsis criteria. Some cases presenting to the hospital with sepsis may have later fulfilled severe sepsis criteria. In the original derivation effort, we did not focus on septic shock due to the smaller number of events in the REGARDS cohort.
In the original REGARDS-SSRS derivation we developed models both with and without biomarker variables. For the current study, we focused on the REGARDS-SSRS with biomarker elements due its higher discriminatory power and the availability of identical biomarkers in the test cohorts.
The original REGARDS-sepsis effort identified sepsis hospitalizations through manual chart review and included only events occurring within the first 28-h of hospitalization. The current analysis utilized an alternate approach entailing ICD-9 discharge diagnoses appearing in Medicare claims, and may have encompassed sepsis events at any point of hospitalization. Comparison of the agreement between manual chart review and discharge diagnoses is the objective of a separate analysis and not the focus of the current study. Prior studies suggest the low sensitivity and high sensitivity of discharge diagnoses for identifying severe sepsis [35]. Also, changes in coding practices and reimbursement incentives may influence sepsis coding [36]. These points are evident by the higher severe sepsis incidence seen in current study. However, even with these limitations, we were able to apply and validate the SSRS using these alternate sepsis identification methods. We acknowledge that the ARIC and CHS data used for validation are 10–20 years old; validation with more recent data is an important future direction.
The current effort using the REGARDS-Medicare, CHS and ARIC cohorts illustrates potential alternate strategies for detecting sepsis events. For example, Medicare claims data may capture the most significant health events for an individual. Additional study with cohorts linked to hospital discharge data may lead to additional insights regarding sepsis event detection and classification. The original SSRS was based upon the outcome of community-acquired severe sepsis. In the current study we could not differentiate community-acquired from hospital-acquired severe sepsis because the study data sets do not indicate diagnoses that were present on admission. Additional validation with a cohort incorporating present-on-admission diagnosis flags is an important future goal. While some of the biomarkers in the SSRS are relatively new (specifically, Cystatin-C), the assay can be potentially incorporated into clinical practice.
The REGARDS cohort included only African Americans and whites ≥45 years. REGARDS also did not include nursing home patients, who may be more vulnerable to sepsis than community dwelling individuals. While we used biomarkers that were readily available in REGARDS. Other biomarkers may have potentially improved model discrimination. We could not account for changes in participant characteristics over time. We did not have information on the presence of select immunosuppressive comorbidities such as human immunodeficiency virus infection or liver disease. The risk score does not include markers of future infection risk such as prior infections, antimicrobial use, or use of urinary catheters. Re-derivation of the SSRS with a broader cohort is necessary to evaluate the impact of these factors. While we conceptualized the SSRS as having roles in primary prevention, further study is also needed to confirm if risk modification in fact leads to reduced sepsis risk. We did not account for the competing risk of death because this would have complicated the interpretation of sepsis risk.

5. Conclusions

In this study we tested the ability of the REGARDS-Severe Sepsis Risk Score to predict 10-year risk of severe sepsis when applied to the REGARDS-Medicare, CHS and ARIC cohorts. The REGARDS-SSRS may potentially play a role in community sepsis prevention or mitigation efforts.

Author Contributions

H.E.W. and J.P.D. conceived the study. H.E.W. and M.S. obtained funding. H.E.W. and M.S. oversaw data collection. J.P.D. conducted the analysis. H.E.W. drafted the manuscript and all authors contributed to its critical review. H.E.W. assumes overall responsibility for the paper.

Funding

This study was supported by award R01-NR012726 from the National Institute for Nursing Research, UL1-RR025777 from the National Center for Research Resources, as well as by grants from the Center for Clinical and Translational Science and the Lister Hill Center for Health Policy of the University of Alabama at Birmingham. The parent REGARDS study was supported by cooperative agreement U01-NS041588 from the National Institute of Neurological Disorders and Stroke, National Institutes of Health, Department of Health and Human Service. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies. Representatives of the funding agencies have been involved in the review of the manuscript but not directly involved in the collection, management, analysis or interpretation of the data. Donnelly received support from grant F31-GM122180 from the National Institute of General Medical Sciences. The Cardiovascular Health Study was supported by contracts HHSN268201200036C, HHSN268200800007C, HHSN268201800001C, N01HC55222, N01HC85079, N01HC85080, N01HC85081, N01HC85082, N01HC85083, N01HC85086, and grants U01HL080295 and U01HL130114 from the National Heart, Lung, and Blood Institute (NHLBI), with additional contribution from the National Institute of Neurological Disorders and Stroke (NINDS). Additional support was provided by R01AG023629 from the National Institute on Aging (NIA). A full list of principal CHS investigators and institutions can be found at CHS-NHLBI.org. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The Atherosclerosis Risk in Communities Study is carried out as a collaborative study supported by National Heart, Lung, and Blood Institute contracts (HHSN268201100005C, HHSN268201100006C, HHSN268201100007C, HHSN268201100008C, HHSN268201100009C, HHSN268201100010C, HHSN268201100011C, and HHSN268201100012C). The funders had no role in the design or execution of this particular paper.

Acknowledgments

The authors thank the other investigators, the staff, and the participants of the REGARDS study for their valuable contributions. A full list of participating REGARDS investigators and institutions can be found at http://www.regardsstudy.org and http://www.regardssepsis.org. The authors thank the staff and participants of the CHS and ARIC studies for their important contributions.

Conflicts of Interest

The authors declare the following potential competing interests: Safford reports the following potential conflicts of interest: Amgen—salary support to study patterns of statin use in Medicare and other large databases; diaDexus—salary support for a research grant on lipids and CHD outcomes; diaDexus—consulting to help with FDA application; NIH, AHRQ—salary support for research grants. Levitan receives research support from Amgen, serves on an advisory boards for Amgen, and provides consulting for a Novartis-sponsored research study. Wang provides methodological consultation for Shire, Inc. (Lexington, MA, USA) Donnelly, Shapiro, Dai, Zhao, Heiss, Odde and Newman do not report any related conflicts of interest.

Appendix A

The REGARDS Sepsis Risk Score (SRS) and Severe Sepsis Risk Score (SSRS) [12]. These prediction scores were previously derived using adjudicated sepsis and severe sepsis events. Validation of the SSRS Model 2 was tested in the current study.
Table A1. The REGARDS Sepsis Risk Score (SRS) and Severe Sepsis Risk Score (SSRS).
Table A1. The REGARDS Sepsis Risk Score (SRS) and Severe Sepsis Risk Score (SSRS).
SEPSIS RISK SCORE (SRS)SEVERE SEPSIS RISK SCORE (SSRS)
SRS MODEL 1 (Demographics, Health Behaviors, and Chronic Medical Conditions)SRS MODEL 2 (Add Biomarkers)SSRS MODEL 1 (Demographics, Health Behaviors, and Chronic Medical Conditions)SSRS MODEL 2 (Add Biomarkers)
VariableHR (95% CI)βSRS PointsHR (95% CI)βSRS PointsHR (95% CI)βSSRS PointsHR (95% CI)βSSRS Points
Demographics, Health Behaviors and Chronic Medical Conditions
Chronic lung disease2.18 (1.92–2.48)0.7852.09 (1.83–2.38)0.7451.93 (1.65–2.25)0.6621.84 (1.58–2.15)0.615
Age ≥75 years1.94 (1.73–2.18)0.6641.62 (1.44–1.83)0.4832.23 (1.96–2.54)0.8031.71 (1.50–1.97)0.544
Peripheral Artery Disease1.66 (1.29–2.13)0.5031.50 (1.17–1.93)0.4131.57 (1.18–2.09)0.4521.41 (1.06–1.87)0.343
Diabetes1.56 (1.40–1.75)0.4531.41 (1.25–1.58)0.3421.74 (1.53–1.98)0.5521.52 (1.33–1.73)0.423
Tobacco use (current)1.56 (1.36–1.78)0.4431.40 (1.22–1.61)0.3421.48 (1.26–1.74)0.3911.33 (1.13–1.57)0.292
White Race1.45 (1.29–1.62)0.3721.46 (1.31–1.64)0.3831.34 (1.18–1.52)0.2911.34 (1.17–1.52)0.292
Stroke1.42 (1.20–1.69)0.3521.30 (1.10–1.54)0.2721.52 (1.26–1.83)0.4221.36 (1.13–1.63)0.302
Atrial Fibrillation1.39 (1.19–1.61)0.3321.31 (1.12–1.52)0.2721.42 (1.20–1.68)0.3511.32 (1.11–1.56)0.282
Coronary artery disease1.35 (1.19–1.53)0.3021.25 (1.10–1.41)0.2221.54 (1.34–1.76)0.4321.37 (1.20–1.57)0.322
Obesity1.34 (1.21–1.50)0.3021.20 (1.07–1.34)0.1811.38 (1.21–1.56)0.3211.21 (1.06–1.38)0.191
Hypertension1.31 (1.16–1.46)0.2721.15 (1.02–1.29)0.1411.36 (1.19–1.55)0.3111.15 (1.00–1.31)0.141
Deep Vein Thrombosis1.27 (1.05–1.54)0.2411.20 (1.00–1.46)0.1911.31 (1.06–1.63)0.2711.23 (0.99–1.52)0.212
Male Sex1.19 (1.07–1.32)0.1711.23 (1.11–1.37)0.2111.43 (1.27–1.62)0.3611.49 (1.32–1.69)0.403
Biomarkers
hsCRP >3.0 mg/dL 1.48 (1.32–1.65)0.393 1.50 (1.31–1.71)0.403
Cystatin-C ≥1.11 mg/dL 1.62 (1.43–1.84)0.483 1.94 (1.66–2.26)0.665
eGFR (creatinine) <60 mL/min/1.73 m2 1.15 (1.00–1.33)0.141 1.36 (1.15–1.59)0.302
ACR ≥30 mcg/mg 1.45 (1.27–1.64)0.373 1.45 (1.26–1.67)0.373
TOTAL POINTS 32 38 20 45
Discrimination (C-statistic)0.677 (0.677–0.679)0.703 (0.702–0.706)0.705 (0.703–0.707)0.742 (0.740–0.745)
Bolded numbers reflect total possible points for each model. hsCRP = high sensitivity C-reactive protein. ACR = Albumin-to-creatinine ratio, eGFR = estimated glomerular filtration rate.
Table A2. Angus, et al. ICD-9 codes for serious infection [2].
Table A2. Angus, et al. ICD-9 codes for serious infection [2].
Infection CategoryICD-9-CM CodeICD-9-CM Code Description
Parasitic001Cholera
002Typhoid/paratyphoid fever
003Other salmonella infection
004Shigellosis
005Other food poisoning
008Intestinal infections due to Escherichia coli
008.1Intestinal infections due to Arizona group of paracolon bacillus
008.2Intestinal infections due to Aerobacter aerogenes
008.3Intestinal infections due to Proteus (mirabilis morganii)
008.4Intestinal infections due to unspecified bacteria
008.5Bacterial enteritis, unspecified
009Ill-defined intestinal infection
013CNS tuberculosis
018Miliary tuberculosis
020Plague
021Tularemia
022Anthrax
023Brucellosis
024Glanders
025Melioidosis
026Rat-bite fever
027Other bacterial zoonoses
032Diphtheria
033Whooping cough
034Streptococcal throat/scarlet fever
035Erysipelas
036Meningococcal infection
037Tetanus
038Septicemia
039Actinomycotic infections
040Other bacterial diseases
041Bacterial infection in other diseases not specified
098Gonococcal infections
100Leptospirosis
101Vincent’s angina
112Candidiasis, of mouth
112.4Candidiasis, of lung
112.5Candidiasis, disseminated
112.8Candidiasis, of other specified sites
114Coccidioidomycosis
115Histoplasmosis
116Blastomycotic infection
117Other mycoses
118Opportunistic mycoses
Nervous320Bacterial meningitis
321Cryptococcal meningitis
321.1Meningitis in other fungal diseases
324CNS abcess
325Phlebitis of intracranial sinus
360Purulent endophthalmitis
376Acute inflammation of orbit
380.14Malignant otitis externa
383Acute mastoiditis
Circulatory420.99Acute pericarditis due to other specified organisms
421Acute or subacute endocarditis
Respiratory461Acute sinusitis
462Acute pharyngitis
463Acute tonsillitis
464Acute laryngitis/tracheitis
465Acute upper respiratory infection of multiple sites/not otherwise specified
475Peritonsillar abscess
481Pneumococcal pneumonia
482Other bacterial pneumonia
485Bronchopneumonia with organism not otherwise specified
486Pneumonia, organism not otherwise specified
491.21Acute exacerbation of obstructive chronic bronchitis
494Bronchiectasis
510Empyema
513Abscess of lung and mediastinum
Digestive522.5Periapical abscess without sinus
522.7Periapical abscess with sinus
526.4Inflammatory conditions of the jaw
527.3Abscess of the salivary glands
528.3Cellulitis and abscess of oral soft tissue
540Acute appendicitis
541Appendicitis not otherwise specified
542Other appendicitis
562.01Diverticulitis of the small intestine without hemorrhage
562.03Diverticulitis of the small intestine with hemorrhage
562.11Diverticulitis of colon without hemorrhage
562.13Diverticulitis of colon with hemorrhage
566Abscess of the anal and rectal regions
567Peritonitis
569.5Intestinal abscess
569.61Infection of colostomy or enterostomy
569.83Perforation of intestine
572Abscess of liver
572.1Portal pyemia
575Acute cholecystitis
Genitourinary590Kidney infection
599Urinary tract infection not otherwise specified
601Prostatic inflammation
604Orchitis and epididymitis
614Female pelvic inflammation disease
615Uterine inflammatory disease
616.3Abscess of Bartholin’s gland
616.4Other abscess of vulva
Pregnancy634Spontaneous abortion, complicated by genital tract and pelvic infection
635Legally induced abortion, complicated by genital tract and pelvic infection
636Illegally induced abortion, complicated by genital tract and pelvic infection
637Unspecified abortion, complicated by genital tract and pelvic infection
638Failed attempted abortion, complicated by genital tract and pelvic infection
639Complications following abortion and ectopic and molar pregnancies
646.6Infections of genitourinary tract in pregnancy
658.4Infection of amniotic cavity
670Major puerperal infection
675.1Abscess of breast
Skin681Cellulitis, finger/toe
682Other cellulitis or abscess
683Acute lymphadenitis
685Pilonidal cyst, with abscess
686Other local skin infection
Musculoskeletal711Pyogenic arthritis
728.86Necrotizing fasciitis
730Osteomyelitis
Other790.7Bacteremia
958.3Posttraumatic wound infection, not elsewhere classified
996.6Infection or inflammation of device/graft
998.5Postoperative infection
999.3Infectious complication of medical care not otherwise classified
995.91Sepsis
Table A3. Angus, et al. ICD-9 codes for organ dysfunction [2].
Table A3. Angus, et al. ICD-9 codes for organ dysfunction [2].
Organ SystemICD-9 CodeDescription
Cardiovascular458Orthostatic hypotension
458.8Other specified hypotension
458.9Hypotension, unspecified
785.5Shock without mention of trauma
Hematologic286.6Defibrination syndrome
286.9Other and unspecified coagulation defects
287.4Secondary thrombocytopenia
287.5Thombocytopenia, unspecified
Hepatic570Acute and subacute necrosis of liver
573.4Hepatic infarction
Neurologic293Transient organic psychosis
348.1Anoxic brain damage
348.3Encephalopathy
Renal584Acute renal failure
Respiratory518.8Respiratory failure
786.03Apnea
799.1Respiratory arrest

References

  1. Levy, M.M.; Fink, M.P.; Marshall, J.C.; Abraham, E.; Angus, D.; Cook, D.; Cohen, J.; Opal, S.M.; Vincent, J.L.; Ramsay, G. 2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference. Intensive Care Med. 2003, 29, 530–538. [Google Scholar] [CrossRef] [PubMed]
  2. Angus, D.C.; Linde-Zwirble, W.T.; Lidicker, J.; Clermont, G.; Carcillo, J.; Pinsky, M.R. Epidemiology of severe sepsis in the United States: Analysis of incidence, outcome, and associated costs of care. Crit. Care Med. 2001, 29, 1303–1310. [Google Scholar] [CrossRef] [PubMed]
  3. Martin, G.S.; Mannino, D.M.; Eaton, S.; Moss, M. The epidemiology of sepsis in the United States from 1979 through 2000. N. Engl. J. Med. 2003, 348, 1546–1554. [Google Scholar] [CrossRef] [PubMed]
  4. Wang, H.E.; Jones, A.R.; Donnelly, J.P. Revised National Estimates of Emergency Department Visits for Sepsis in the United States. Crit. Care Med. 2017, 45, 1443–1449. [Google Scholar] [CrossRef] [PubMed]
  5. Dellinger, R.P.; Levy, M.M.; Rhodes, A.; Annane, D.; Gerlach, H.; Opal, S.M.; Sevransky, J.E.; Sprung, C.L.; Douglas, I.S.; Jaeschke, R.; et al. Surviving Sepsis Campaign: International Guidelines for Management of Severe Sepsis and Septic Shock: 2012. Crit. Care Med. 2013, 41, 580–637. [Google Scholar] [CrossRef] [PubMed]
  6. Yealy, D.M.; Kellum, J.A.; Huang, D.T.; Barnato, A.E.; Weissfeld, L.A.; Pike, F.; Terndrup, T.; Wang, H.E.; Hou, P.C.; LoVecchio, F.; et al. A randomized trial of protocol-based care for early septic shock. N. Engl. J. Med. 2014, 370, 1683–1693. [Google Scholar] [PubMed]
  7. Corday, E.; Corday, S.R. Advances in clinical management of acute myocardial infarction in the past 25 years. J. Am. Coll. Cardiol. 1983, 1, 126–132. [Google Scholar] [CrossRef]
  8. Marshall, R.S.; Mohr, J.P. Current management of ischaemic stroke. J. Neurol. Neurosurg. Psychiatry 1993, 56, 6–16. [Google Scholar] [CrossRef]
  9. Centers for Disease Control and Prevention (CDC). Ten great public health achievements—United States, 2001–2010. MMWR Morb. Mortal. Wkly. Rep. 2011, 60, 619–623. [Google Scholar]
  10. Kempker, J.A.; Wang, H.E.; Martin, G.S. Sepsis is a preventable public health problem. Crit. Care 2018, 22, 116. [Google Scholar] [CrossRef]
  11. Wang, H.E.; Shapiro, N.I.; Griffin, R.; Safford, M.M.; Judd, S.; Howard, G. Chronic medical conditions and risk of sepsis. PLoS ONE 2012, 7, e48307. [Google Scholar] [CrossRef] [PubMed]
  12. Wang, H.E.; Donnelly, J.P.; Griffin, R.; Levitan, E.B.; Shapiro, N.I.; Howard, G.; Safford, M.M. Derivation of Novel Risk Prediction Scores for Community-Acquired Sepsis and Severe Sepsis. Crit. Care Med. 2016, 44, 1285–1294. [Google Scholar] [CrossRef] [PubMed]
  13. Stiell, I.G.; Wells, G.A. Methodologic standards for the development of clinical decision rules in emergency medicine. Ann. Emerg. Med. 1999, 33, 437–447. [Google Scholar] [CrossRef]
  14. Howard, V.J.; Cushman, M.; Pulley, L.; Gomez, C.R.; Go, R.C.; Prineas, R.J.; Graham, A.; Moy, C.S.; Howard, G. The reasons for geographic and racial differences in stroke study: Objectives and design. Neuroepidemiology 2005, 25, 135–143. [Google Scholar] [CrossRef] [PubMed]
  15. Muntner, P.; Gutierrez, O.M.; Zhao, H.; Fox, C.S.; Wright, N.C.; Curtis, J.R.; McClellan, W.; Wang, H.; Kilgore, M.; Warnock, D.G.; et al. Validation study of medicare claims to identify older US adults with CKD using the Reasons for Geographic and Racial Differences in Stroke (REGARDS) Study. Am. J. Kidney Dis. 2015, 65, 249–258. [Google Scholar] [CrossRef] [PubMed]
  16. Kumamaru, H.; Judd, S.E.; Curtis, J.R.; Ramachandran, R.; Hardy, N.C.; Rhodes, J.D.; Safford, M.M.; Kissela, B.M.; Howard, G.; Jalbert, J.J.; et al. Validity of claims-based stroke algorithms in contemporary Medicare data: Reasons for geographic and racial differences in stroke (REGARDS) study linked with medicare claims. Circ. Cardiovasc. Qual. Outcomes 2014, 7, 611–619. [Google Scholar] [CrossRef] [PubMed]
  17. Aaron, K.J.; Colantonio, L.D.; Deng, L.; Judd, S.E.; Locher, J.L.; Safford, M.M.; Cushman, M.; Kilgore, M.L.; Becker, D.J.; Muntner, P. Cardiovascular Health and Healthcare Utilization and Expenditures Among Medicare Beneficiaries: The REasons for Geographic And Racial Differences in Stroke (REGARDS) Study. J. Am. Heart Assoc. 2017, 6, e005106. [Google Scholar] [CrossRef]
  18. Xie, F.; Colantonio, L.D.; Curtis, J.R.; Safford, M.M.; Levitan, E.B.; Howard, G.; Muntner, P. Linkage of a Population-Based Cohort With Primary Data Collection to Medicare Claims: The Reasons for Geographic and Racial Differences in Stroke Study. Am. J. Epidemiol. 2016, 184, 532–544. [Google Scholar] [CrossRef] [PubMed]
  19. Kent, S.T.; Safford, M.M.; Zhao, H.; Levitan, E.B.; Curtis, J.R.; Kilpatrick, R.D.; Kilgore, M.L.; Muntner, P. Optimal use of available claims to identify a Medicare population free of coronary heart disease. Am. J. Epidemiol. 2015, 182, 808–819. [Google Scholar] [CrossRef]
  20. Colantonio, L.D.; Kent, S.T.; Kilgore, M.L.; Delzell, E.; Curtis, J.R.; Howard, G.; Safford, M.M.; Muntner, P. Agreement between Medicare pharmacy claims, self-report, and medication inventory for assessing lipid-lowering medication use. Pharmacoepidemiol. Drug Saf. 2016, 25, 827–835. [Google Scholar] [CrossRef][Green Version]
  21. National Heart, Lung and Blood Institute. The Cardiovascular Health Study. Available online: https://chs-nhlbi.org/ (accessed on 10 December 2018).
  22. Atheroclerosis Risk in Communities Study. Available online: https://www2.cscc.unc.edu/aric/contact_the_coord_ctr (accessed on 10 December 2018).
  23. Singer, M.; Deutschman, C.S.; Seymour, C.W.; Shankar-Hari, M.; Annane, D.; Bauer, M.; Bellomo, R.; Bernard, G.R.; Chiche, J.D.; Coopersmith, C.M.; et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA 2016, 315, 801–810. [Google Scholar] [CrossRef] [PubMed][Green Version]
  24. Merkin, S.S.; Coresh, J.; Diez Roux, A.V.; Taylor, H.A.; Powe, N.R. Area socioeconomic status and progressive CKD: The Atherosclerosis Risk in Communities (ARIC) Study. Am. J. Kidney Dis. 2005, 46, 203–213. [Google Scholar] [CrossRef] [PubMed]
  25. Kurella Tamura, M.; Wadley, V.; Yaffe, K.; McClure, L.A.; Howard, G.; Go, R.; Allman, R.M.; Warnock, D.G.; McClellan, W. Kidney function and cognitive impairment in US adults: The Reasons for Geographic and Racial Differences in Stroke (REGARDS) Study. Am. J. Kidney Dis. 2008, 52, 227–234. [Google Scholar] [CrossRef] [PubMed]
  26. Royston, P. Tools for checking calibration of a Cox model in external validation: Prediction of population-averaged survival curves based on risk groups. Stata J. 2015, 15, 275–291. [Google Scholar] [CrossRef]
  27. Hosmer, D.W.; Lemeshow, S.; May, S. Applied Survival Analysis: Regression Modeling of Time-To-Event Data, 2nd ed.; Wiley-Interscience: Hoboken, NJ, USA, 2008. [Google Scholar]
  28. Harrell, F.E. Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis; Springer: New York, NY, USA, 2001. [Google Scholar]
  29. Shapiro, N.I.; Wolfe, R.E.; Moore, R.B.; Smith, E.; Burdick, E.; Bates, D.W. Mortality in Emergency Department Sepsis (MEDS) score: A prospectively derived and validated clinical prediction rule. Crit. Care Med. 2003, 31, 670–675. [Google Scholar] [CrossRef] [PubMed]
  30. Ridker, P.M.; Cannon, C.P.; Morrow, D.; Rifai, N.; Rose, L.M.; McCabe, C.H.; Pfeffer, M.A.; Braunwald, E.; Atorvastatin, E. Infection Therapy-Thrombolysis in Myocardial Infarction I: C-reactive protein levels and outcomes after statin therapy. N. Engl. J. Med. 2005, 352, 20–28. [Google Scholar] [CrossRef]
  31. Shen, H.-N.; Lu, C.-L.; Yang, H.-H. Epidemiologic Trend of Severe Sepsis in Taiwan from 1997 through 2006. Chest 2010, 138, 298–304. [Google Scholar] [CrossRef]
  32. Schonheyder, H.C.; Sogaard, M. Existing data sources for clinical epidemiology: The North Denmark Bacteremia Research Database. Clin. Epidemiol. 2010, 2, 171–178. [Google Scholar] [CrossRef]
  33. Steyerberg, E.W. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating; Springer: New York, NY, USA, 2009. [Google Scholar]
  34. Page, D.B.; Donnelly, J.P.; Wang, H.E. Community-, Healthcare-, and Hospital-Acquired Severe Sepsis Hospitalizations in the University HealthSystem Consortium. Crit. Care Med. 2015, 43, 1945–1951. [Google Scholar] [CrossRef][Green Version]
  35. Rhee, C.; Dantes, R.; Epstein, L.; Murphy, D.J.; Seymour, C.W.; Iwashyna, T.J.; Kadri, S.S.; Angus, D.C.; Danner, R.L.; Fiore, A.E.; et al. Incidence and Trends of Sepsis in US Hospitals Using Clinical vs. Claims Data, 2009–2014. JAMA 2017, 318, 1241–1249. [Google Scholar] [CrossRef]
  36. Lindenauer, P.K.; Bernheim, S.M.; Grady, J.N.; Lin, Z.; Wang, Y.; Wang, Y.; Merrill, A.R.; Han, L.F.; Rapp, M.T.; Drye, E.E.; et al. The performance of US hospitals as reflected in risk-standardized 30-day mortality and readmission rates for medicare beneficiaries with pneumonia. J. Hosp. Med. 2010, 5, E12–E18. [Google Scholar] [CrossRef] [PubMed]
Figure 1. Normality (panels A, C, E) and distribution (panels B, D, F) of the REGARDS Severe Sepsis Risk Score in REGARDS-Medicare, CHS and ARIC cohorts. REGARDS = Reasons for Geographic and Racial Differences in Stroke. CHS = Cardiovascular Health Study. ARIC = Atherosclerosis Risk in Communities.
Figure 1. Normality (panels A, C, E) and distribution (panels B, D, F) of the REGARDS Severe Sepsis Risk Score in REGARDS-Medicare, CHS and ARIC cohorts. REGARDS = Reasons for Geographic and Racial Differences in Stroke. CHS = Cardiovascular Health Study. ARIC = Atherosclerosis Risk in Communities.
Jcm 07 00536 g001
Figure 2. Calibration of the REGARDS Severe Sepsis Risk Score in the REGARDS-Medicare (A), CHS (B) and ARIC (C) cohorts. REGARDS = Reasons for Geographic and Racial Differences in Stroke. CHS = Cardiovascular Health Study. ARIC = Atherosclerosis Risk in Communities.
Figure 2. Calibration of the REGARDS Severe Sepsis Risk Score in the REGARDS-Medicare (A), CHS (B) and ARIC (C) cohorts. REGARDS = Reasons for Geographic and Racial Differences in Stroke. CHS = Cardiovascular Health Study. ARIC = Atherosclerosis Risk in Communities.
Jcm 07 00536 g002
Table 1. The REGARDS Severe Sepsis Risk Score (SSRS).
Table 1. The REGARDS Severe Sepsis Risk Score (SSRS).
VariableSSRS Points
Chronic lung disease5
Age ≥75 years4
Peripheral Artery Disease3
Diabetes3
Male Sex3
Tobacco use (current)2
White Race2
Stroke2
Atrial Fibrillation2
Coronary artery disease2
Deep Vein Thrombosis2
Obesity1
Hypertension1
Cystatin-C ≥1.11 mg/dL5
hsCRP >3.0 mg/dL3
ACR ≥30 mcg/mg3
eGFR (creatinine) <60 mL/min/1.73 m22
TOTAL SSRS POINTS45
hsCRP = high sensitivity C-reactive protein. ACR = Albumin-to-creatinine ratio, eGFR = estimated glomerular filtration rate.
Table 2. Baseline participant characteristics by cohort.
Table 2. Baseline participant characteristics by cohort.
CharacteristicInternal ApplicationExternal Validation
REGARDS-Medicare
(n = 9522)
CHS
(n = 5888)
ARIC
(n = 11,584)
Age
<65 years006998 (60.4)
65–746120 (64.3)3894 (66.1)4583 (39.6)
≥753402 (35.7)1994 (33.9)3 (0.0)
Gender
Male4692 (49.3)2495 (42.4)5106 (44.1)
Female4830 (50.7)3393 (57.6)6478 (55.9)
Race
White6440 (67.6)4925 (83.6)8899 (76.8)
Black3082 (32.4)963 (16.4)2685 (23.2)
Tobacco Use
Current910 (9.6)700 (11.9)1700 (14.7)
No Current Use8612 (90.4)5188 (88.1)9884 (85.3)
Chronic Medical Conditions
Atrial Fibrillation1025 (10.8)236 (4.0)324 (2.8)
Chronic Lung Disease1013 (10.6)915 (15.5)929 (8.0)
Coronary Artery Disease2364 (24.8)1,154 (19.6)965 (8.3)
Diabetes2276 (23.9)989 (16.8)1924 (16.6)
Deep Vein Thrombosis648 (6.8)312 (5.3)348 (3.0)
Hypertension6200 (65.1)2619 (44.5)5510 (47.6)
Obesity4746 (49.8)2747 (46.7)7592 (65.5)
Peripheral Artery Disease275 (2.9)120 (2.0)385 (3.3)
Stroke796 (8.4)249 (4.2)265 (2.3)
High-Sensitivity C-Reactive Protein
>3.0 mg/dL3449 (36.2)2481 (42.1)4909 (42.4)
≤3.0 mg/dL6073 (63.8)3407 (57.9)6675 (57.6)
Cystatin-C
≥1.11 mg/dL3486 (36.6)1569 (26.7)1504 (13.0)
<1.11 mg/dL6036 (63.4)4319 (73.4)10,080 (87.0)
eGFR (creatinine)
<60 mL/min/1.73 m21723 (18.1)1368 (24.1)1112 (9.6)
≥60 mL/min/1.73 m27799 (81.9)4520 (76.8)10,472 (90.4)
Albumin-to-Creatinine Ratio
≥30 mcg/mg1612 (16.9)654 (11.1)956 (8.3)
<30 mcg/mg7910 (83.1)5234 (88.9)10,628 (91.8)
SSRS Risk Group
Very Low (<6 Points)1616 (17.0)1137 (19.3)3761 (32.5)
Low (6–9 Points)2308 (24.2)1582 (26.9)4388 (37.9)
Medium (10–13 Points)2127 (22.3)1358 (23.1)2037 (17.6)
High (14–17 Points)1687 (17.7)988 (16.8)900 (7.8)
Very High (>17 Points)1784 (18.7)823 (14.0)498 (4.3)
REGARDS = Reasons for Geographic and Racial Differences in Stroke. CHS = Cardiovascular Health Study. ARIC = Atherosclerosis Risk in Communities.
Table 3. Severe sepsis incidence for each cohort, overall and stratified by REGARDS-Severe Sepsis Risk Score risk category.
Table 3. Severe sepsis incidence for each cohort, overall and stratified by REGARDS-Severe Sepsis Risk Score risk category.
Risk GroupInternal ApplicationExternal Validation
REGARDS-Medicare
(n = 9522)
CHS
(n = 5888)
ARIC
(n = 11,584)
Severe Sepsis Events (n)
IR (95% CI)
HR (95% CI)Severe Sepsis Events (n)
IR (95% CI)
HR (95% CI)Severe Sepsis Events (n)
IR (95% CI)
HR (95% CI)
Full Cohort1593
30.7 (29.2–32.2)
---586
11.9 (10.9–12.9)
---735
6.8 (6.3–7.3)
---
Stratified by REGARDS-SSRS Risk Category
Very Low (<6 Points)96
9.9 (8.1–12.1)
Reference39
3.6 (2.6–4.9)
Reference114
3.1 (2.6–3.8)
Reference
Low (6–9 Points)230
17.0 (14.9–19.3)
1.73
(1.36–2.19)
99
6.9 (5.7–8.4)
1.94
(1.34–2.82)
208
5.0 (4.3–5.7)
1.60
(1.27–2.01)
Medium (10–13 Points)299
24.8 (22.2–27.8)
2.53
(2.01–3.19)
143
12.8 (10.8–15.0)
3.71
(2.61–5.29)
179
9.7 (8.4–11.2)
3.15
(2.49–3.99)
High (14–17 Points)374
42.0 (37.9–46.5)
4.36
(3.48–5.45)
138
18.4 (15.6–21.8)
5.51
(3.86–7.86)
118
15.6 (13.0–18.7)
5.20
(4.02–6.73)
Very High (>17 Points)594
76.8 (70.9–83.3)
8.27
(6.66–10.26)
167
30.2 (25.9–35.1)
9.46
(6.67–13.41)
116
33.7 (28.1–40.4)
11.98
(9.24–15.52)
REGARDS = Reasons for Geographic and Racial Differences in Stroke. CHS = Cardiovascular Health Study. ARIC = Atherosclerosis Risk in Communities. IR = incidence rate per 1000 person-years. HR = hazard ratio. CI = confidence interval.
Table 4. Discrimination, calibration, and goodness-of-fit of the REGARDS-Severe Sepsis Risk Score for each cohort.
Table 4. Discrimination, calibration, and goodness-of-fit of the REGARDS-Severe Sepsis Risk Score for each cohort.
MeasureREGARDS-MedicareCHSARIC
Discrimination—C-statistic (95% CI)0.704 (0.691–0.718)0.696 (0.675–0.716)0.697 (0.677–0.716)
Decile of Predicted Risk (Observed/Predicted Events)
160/59.929/30.741/41.0
289/95.129/47.131/23.2
397/108.130/25.542/47.9
480/68.150/55.456/61.9
5150/147.527/28.454/56.6
6149/162.980/61.144/48.3
7179/172.378/69.554/48.2
8195/183.367/61.293/90.6
9302/298.880/83.9124/116.8
10292/297.0116/123.2196/200.4
Goodness of Fit—Grønnesby and Borgan test (Chi-square; p-value)7.66 (0.57)18.95 (0.03)5.72 (0.77)
REGARDS = Reasons for Geographic and Racial Differences in Stroke. CHS = Cardiovascular Health Study. ARIC = Atherosclerosis Risk in Communities.
Table 5. Percentage positive with severe sepsis, true positive rate, and true negative rate by REGARDS-Severe Sepsis Risk Score risk category.
Table 5. Percentage positive with severe sepsis, true positive rate, and true negative rate by REGARDS-Severe Sepsis Risk Score risk category.
Risk GroupREGARDS-MedicareCHSARIC
PP
(% with Severe Sepsis)
TPR
(% of Cases)
TNR
(% of Non-Cases)
PP
(% with Severe Sepsis)
TPR
(% of Cases)
TNR
(% of Non-Cases)
PP
(% with Severe Sepsis)
TPR
(% of Cases)
TNR
(% of Non-Cases)
Very Low
(<6 Points)
5.96.019.23.46.720.73.015.533.6
Low
(6–9 Points)
10.014.426.26.316.928.04.728.338.5
Medium
(10–13 Points)
14.118.823.110.524.422.98.824.417.1
High
(14–17 Points)
22.223.516.614.023.516.013.116.17.2
Very High
(>17 Points)
33.337.315.020.328.512.423.315.83.5
REGARDS = Reasons for Geographic and Racial Differences in Stroke. CHS = Cardiovascular Health Study. ARIC = Atherosclerosis Risk in Communities. PP = percent positive. TPR = true positive rate. TNR = true negative rate.

Share and Cite

MDPI and ACS Style

Wang, H.E.; Donnelly, J.P.; Yende, S.; Levitan, E.B.; Shapiro, N.I.; Dai, Y.; Zhao, H.; Heiss, G.; Odden, M.; Newman, A.; Safford, M. Validation of the REGARDS Severe Sepsis Risk Score. J. Clin. Med. 2018, 7, 536. https://doi.org/10.3390/jcm7120536

AMA Style

Wang HE, Donnelly JP, Yende S, Levitan EB, Shapiro NI, Dai Y, Zhao H, Heiss G, Odden M, Newman A, Safford M. Validation of the REGARDS Severe Sepsis Risk Score. Journal of Clinical Medicine. 2018; 7(12):536. https://doi.org/10.3390/jcm7120536

Chicago/Turabian Style

Wang, Henry E., John P. Donnelly, Sachin Yende, Emily B. Levitan, Nathan I. Shapiro, Yuling Dai, Hong Zhao, Gerardo Heiss, Michelle Odden, Anne Newman, and Monika Safford. 2018. "Validation of the REGARDS Severe Sepsis Risk Score" Journal of Clinical Medicine 7, no. 12: 536. https://doi.org/10.3390/jcm7120536

Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop