Next Article in Journal
Collagen Fibrillogenesis in the Mitral Valve: It’s a Matter of Compliance
Previous Article in Journal
Fat-to-Muscle Ratios and the Non-Achievement of LDL Cholesterol Targets: Analysis of the Korean Genome and Epidemiology Study
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Diagnostic Performance of Serial High-Sensitivity Cardiac Troponin Measurements in the Emergency Setting

1
Department of Pathology and Molecular Medicine, McMaster University, Hamilton, ON L8S 4L8, Canada
2
Division of Emergency Medicine, McMaster University, Hamilton, ON L8S 4L8, Canada
3
ICES McMaster, Faculty of Health Sciences, McMaster University, Hamilton, ON L8S 4K1, Canada
4
Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON L8S 4L8, Canada
5
Department of Medicine, McMaster University, Hamilton, ON L8S 4L8, Canada
6
Division of Cardiology, McMaster University, Hamilton, ON L8S 4L8, Canada
7
ICES Central, Sunnybrook Health Sciences Centre, Toronto, ON M4N 3M5, Canada
*
Author to whom correspondence should be addressed.
J. Cardiovasc. Dev. Dis. 2021, 8(8), 97; https://doi.org/10.3390/jcdd8080097
Submission received: 7 July 2021 / Revised: 5 August 2021 / Accepted: 9 August 2021 / Published: 13 August 2021

Abstract

:
Serial high-sensitivity cardiac troponin (hsTn) testing in the emergency department (ED) and the intensive cardiac care unit may assist physicians in ruling out or ruling in acute myocardial infarction (MI). There are three major algorithms proposed for high-sensitivity cardiac troponin I (hsTnI) using serial measurements while incorporating absolute concentration changes for MI or death following ED presentation. We sought to determine the diagnostic estimates of these three algorithms and if one was superior in two different Canadian ED patient cohorts with serial hsTnI measurements. An undifferentiated ED population (Cohort-1) and an ED population with symptoms suggestive of acute coronary syndrome (ACS; Cohort-2) were clinically managed with non-hsTn testing with the hsTnI testing performed in real-time with physicians blinded to these results (i.e., hsTnI not reported). The three algorithms evaluated were the European Society of Cardiology (ESC), the High-STEACS pathway, and the COMPASS-MI algorithm. The diagnostic estimates were derived for each algorithm for the 30-day MI/death outcome for the rule-out and rule-in arm in each cohort and compared to proposed diagnostic benchmarks (i.e., sensitivity ≥ 99.0% and specificity ≥ 90.0%) with 95% confidence intervals (CI). In Cohort-1 (n = 2966 patients, 15.3% had outcome) and Cohort-2 (n = 935 patients, 15.6% had outcome), the algorithm that obtained the highest sensitivity (97.8%; 95% CI: 96.0–98.9 and 98.6%; 95% CI: 95.1–99.8, respectively) in both cohorts was COMPASS-MI. Only Cohort-2 with both the ESC and COMPASS-MI algorithms exceeded the specificity benchmark (97.0%; 95% CI: 95.5–98.0 and 96.7%; 95% CI: 95.2–97.8, respectively). Patient selection for serial hsTnI testing will affect specificity estimates, with no algorithm achieving a sensitivity ≥ 99% for 30-day MI or death.

1. Introduction

Guidelines support the use of early and serial testing with high-sensitivity cardiac troponin (hsTn) in patients with symptoms suggestive of acute coronary syndrome (ACS) for the diagnosis of acute myocardial infarction (MI) [1,2,3]. Observational studies have assessed algorithms using low and high hsTn concentrations with respective minor and large differences in concentrations over a couple of hours in the emergency department (ED); these studies suggest hsTn alone may be sufficient to rule-out and rule-in MI [4,5]. However, not all studies have supported the safe rule-out or possible discharge of patients from the ED being assigned to the rule-out arms or low-risk categories with these shorter testing algorithms [6,7]. A recent randomized control trial assessing the European Society of Cardiology (ESC) 0/1 h algorithm further questions the safety of these algorithms as it did not reduce cardiovascular death or MI over the long term (2.7% composite outcome with standard protocol versus 3.3% with 0/1 h protocol; p < 0.001) [8].
The benchmark for safety estimates for ruling out MI is attributed to an international survey response from ED physicians that reported an acceptable 30-day miss-rate of major adverse cardiac events (MACE, defined as the following in the survey: “Most MACE’s are NSTEMI’s but there are also a small but significant number of others (death, cardiac arrest, cardiogenic shock, ventricular arrhythmia or AV block requiring intervention)” [9] would be ≤1% and “that clinicians may expect diagnostic strategies for the assessment of suspected ACS to achieve a sensitivity of 99% or higher for AMI or other MACE” [9]. Others have proposed predictive values be used [10], with the following diagnostic estimates being evaluated in studies: sensitivity ≥ 99% or negative predictive value (NPV) ≥ 99.5% for rule-out and specificity ≥ 90% or positive predictive value (PPV) ≥ 75% for rule-in [11,12,13]. Many algorithms have been developed and implemented based mainly on predictive values, and despite the number of published algorithms increasing, there are few head-to-head comparison studies performed in the same cohorts with even fewer algorithms validated in external populations to assess generalizability [3,10,14,15].
In the present study, we evaluated the three main published hsTnI algorithms that use serial measurements [3,5,14] in two different ED cohorts for 30-day MI or death (a common outcome reported by studies using these algorithms) [3,5,14]. In both cohorts, physicians were blinded to the hsTnI results with the cardiac troponin test being reported clinically belonging to a non-hsTn test version (note, standard, conventional or contemporary are terms that are also used to describe non-hsTn tests). The overall objective was to assess if the diagnostic estimates using these algorithms met any of the four diagnostic benchmarks (i.e., sensitivity ≥ 99%, NPV ≥ 99.5%, specificity ≥ 90%, PPV ≥ 75%). We also examined whether patients identified in the rule-out arms would also be classified as low-risk if the miss-rate for death or MI in these groups was ≤1% at 30 days.

2. Materials and Methods

The criteria from the standards for reporting diagnostic accuracy studies (STARD 2015) were followed [16], and the study was approved by the Hamilton Integrated Research Ethics Board (REB# 13-277 and 4717-D).

2.1. Study Design and Participants

In two different ED cohorts, we measured hsTnI for research purposes from the same blood samples used concurrently for testing with the non-hsTn assay that was used for clinical purposes. Both cohorts were observational studies and have been previously described using the presentation sample (i.e., the test results from the first sample measurement) [12,13,17,18]. The current retrospective analyses were confined to those participants with two hsTnI results to assess the algorithms (see Figure 1 flow of participants).
Cohort-1 consisted of consecutive patients who presented to the EDs of the Hamilton General Hospital, Juravinski Hospital, and St. Joseph’s Healthcare (Hamilton, ON, Canada) from November 2012 to February 2013 and had 2 hsTnI (blinded) results reported during their index ED visit. Patients who were not Ontario residents or covered by the Ontario Health Insurance Plan (OHIP) were excluded.
Cohort-2 was another observational cohort study (NCT01994577) conducted from May 2013 to August 2013 in patients presenting with symptoms suggestive of ACS to the ED at the same three Hamilton hospitals. Eligibility for enrollment included being an adult aged 18 or older, not being transferred from another hospital, with an ED physician having placed an order for cardiac troponin testing. Excluded participants were those patients with non-ACS symptoms (i.e., symptoms that did not include the following: chest pain/discomfort, pain or discomfort in one or both arms, pain or discomfort in the jaw, neck, back, pain or discomfort in the abdomen, shortness of breath, feeling dizzy or lightheaded, nausea and/or vomiting, diaphoresis, palpitations); or had ST-segment elevation MI at presentation; or had chest trauma, cardiac surgery or manipulation within 30-days of presentation; an MI (or pulmonary embolus) confirmed within the previous month; known active cancer or non-cardiac fatal illness; sepsis; ventricular fibrillation, or sustained ventricular tachycardia [17,19,20].

2.2. Health Outcomes

The primary outcome was 30-day MI or all-cause death [13,17]. In Cohort-1, past medical history and outcomes were obtained through linkages to administrative databases via ICES (formerly known as the Institute for Clinical Evaluative Sciences) using unique encrypted patient identifiers [13,18,21]. In Cohort-2, an emergency physician led an adjudication panel with the outcomes independently adjudicated by two members with disagreements not resolved by consensus referred to a third blinded reviewer [13,17]. From November 2012 to November 2014 in Hamilton, Ontario, the Abbott cardiac troponin I (non-hsTnI) assay was used, with the overall 99th percentile of 0.03 µg/L used to flag myocardial injury (i.e., evidence of biochemical myocardial injury present if cardiac troponin I concentrations ≥ 0.04 µg/L) [22].

2.3. Laboratory Testing and Algorithms Evaluated

The Abbott ARCHITECT hsTnI assay was measured in both cohorts with acceptable precision and performance documented for this assay [13,17]. In Cohort-1, there was no clinical recommendation on early sampling so a second sample was collected per standard care (reported median (25th to 75th) = 7.2 h (4.3 to 8.8) from presentation sample) [23]. In Cohort-2, there was a clinical recommendation to collect samples 3 h apart as per the Canadian Institutes of Health Research funded grant and study protocol [24]. As the diagnostic parameters have been published for the ESC 0/1 h algorithm cutoffs (sensitivity/NPV/specificity/PPV of 97.8%/99.4%/91.3%/50%) in Cohort-2 [24], the 0/2 h algorithm cutoffs were assessed (as the ESC recommends to use the 0/2 h algorithm as an alternative to the 0/1 h algorithm) [3], as well as the High-STEACS pathway [14] and COMPASS-MI algorithm [5] in both cohorts (see Table 1 for details of the algorithms).
For COMPASS-MI, physicians can select different cutoffs as detailed in the original publication and using the web-based calculator (https://compass-mi.com/ accessed on 26 April 2021) [5]. However, for a hospital system, a common rule-out and rule-in pathway is beneficial for achieving standardization of care. In consultation with emergency physicians and considering analytical performance, the following cutoffs and changes were used: rule-out if 1st sample hsTnI < 4 ng/L and change < 4 ng/L between the two samples and rule-in if 1st sample hsTnI ≥ 60 ng/L or change ≥ 18 ng/L between the two samples. The 4 ng/L cutoff can be measured with acceptable precision [25] and has been used in other studies for rule-out [13,17,26], with the change < 4 ng/L representing the total analytic error for hsTn at these low concentrations [27]. The 60 ng/L has been demonstrated to provide important prognostic information for 30-day cardiac events [28] and a change of 18 ng/L represents an analytically different result for assays with coefficient of variations ≤ 10% for concentrations < 60 ng/L. Applying these parameters, in the early sampling COMPASS-MI algorithm yielded the following diagnostic estimates based on the original publication [5]: for rule-out the sensitivity was 99.0% (95% CI: 98.3 to 99.5) and NPV was 99.7% (95% CI: 99.4 to 99.8), and for rule-in the specificity was 95.5% (95% CI: 95.0 to 96.0) and PPV was 75.8 (95% CI: 73.2 to 78.2). For both ESC and COMPASS-MI algorithms, there are three categories of risk: rule-out, rule-in and uncertain (observation group). For High-STEACS, there are only two categories of risk, low and high. The lower limit of detection (LoD) for the Abbott hsTnI assay ranges from 1.1 ng/L to 1.7 ng/L (see https://www.ifcc.org/media/478962/high-sensitivity-cardiac-troponin-i-and-t-assay-analytical-characteristics-designated-by-manufacturer-v042021.pdf, accessed on 5 August 2021). The lowest LoD of 1 ng/L was used in Cohort-2 with values < 1 ng/L converted to 0.9 ng/L; however, for Cohort-1 the numerical results reported from the instruments were used with the lowest concentration being 0 ng/L (no lower analytical limit or LoD was assigned to the instruments).

2.4. Statistical Analyses

We calculated non-parametric (i.e., median with 25th to 75th percentiles listed) ranges and frequencies (percentage, %) for the common variables of the two cohorts. We also calculated the diagnostic estimates sensitivity, NPV, specificity, PPV, and negative and positive likelihood ratios with 95% confidence intervals (CIs) of each cohort for both the rule-out and rule-in groups. We performed all statistical analyses using SAS 9.1.3 software (SAS Institute Inc, Cary, NC, USA), R software (R Foundation for Statistical Computing), and StatsDirect Statistical software. Acceptable performance of the algorithms was noted if the diagnostic estimates met or exceeded the stated benchmarks: sensitivity ≥ 99%, NPV ≥ 99.5%, specificity ≥ 90%, PPV ≥ 75%.

3. Results

Of the 2966 patients in Cohort-1, 454 patients (15.3%; 95% CI: 14.0 to 16.7) had the primary outcome (30-day MI or all-cause death), and of the 935 patients in Cohort-2, 146 patients (15.6%; 95% CI: 13.3 to 18.1) had the primary outcome with no significant difference in proportions (p = 0.82). There was no difference in the proportion of patients assigned to the uncertain (observation) group between the ESC and COMPASS-MI algorithms in Cohort-1 (p = 0.88) and Cohort-2 (p = 0.07). Additionally, in both cohorts, there was no difference in the frequency of women assigned to the various groups with the three algorithms; however, differences in the other variables associated with risk (i.e., clinical history) were evident (Table 2). There were, however, differences in time between samples collected between the two cohorts, with time between sample collection in Cohort-2 being shorter (median time = 3 h) (Table 3).
In Cohort-1, the High-STEACS pathway designated 1415 patients as rule-out (47%; 95% CI: 45.4 to 49.0%), with 4.0% (95% CI: 3.0 to 5.1) of this group having the primary outcome, whereas the COMPASS-MI algorithm designated the fewest patients (n = 697) as rule-out (23.6%; 95% CI: 22.0 to 25.1), with 1.4% (95% CI: 0.7 to 2.6) of this group having the primary outcome (Table 3). The proportion of patients in the rule-in group having the primary outcome was nearly identical between the ESC (39.2%; 95% CI: 25.9 to 42.6) and COMPASS-MI (39.6%; 95% CI: 36.2 to 43.0) algorithms and higher as compared to the High-STEACS pathway (25.7%; 95% CI: 23.5 to 27.9). Cohort-2 yielded similar estimates and trends (Table 3).
In Cohort-1, the highest sensitivity and NPV were observed with the COMPASS-MI rule-out arm (97.8%; 95% CI: 96.0 to 98.9 and 98.6%; 95% CI: 97.4 to 99.3, respectively), with the smallest negative likelihood ratio (0.08; 95% CI: 0.03 to 0.13) (Table 4). The highest specificity (80.3; 95% CI: 78.6 to 81.8), PPV (39.6%; 95% CI: 36.2 to 43.0), and positive likelihood ratio (3.63; 95% CI: 3.27 to 3.98) were also observed with the COMPASS-MI rule-in arm. In Cohort-2, the COMPASS-MI rule-out arm yielded the highest sensitivity (98.6%; 95% CI: 95.1 to 99.8), NPV (99.4%; 95% CI: 97.7 to 99.9), and lowest negative likelihood ratio (0.03; 95% CI: 0.01 to 0.13), with both the ESC and COMPASS-MI rule-in arms yielding estimates for specificity (>96%), PPV (>76%), and positive likelihood ratios (>17) (Table 4).

4. Interpretation

High-sensitivity cardiac troponin testing is the gold standard for detecting myocardial injury and aiding clinicians in making the diagnosis of MI [1,2,3]. An undetectable or low hsTn level within the normal reference interval is associated with a good prognosis and identifies patients at low risk for future cardiac events [29,30]. However, using a single low hsTn level to rule-out or discharge patients home may miss patients who are at high risk and have not reliably achieved a sensitivity ≥ 99.0%. Additional tools and laboratory tests could improve this performance (examples being HEART Score and Clinical Chemistry Score) [7,17]. Serial testing with hsTn may close this gap by identifying patients with evolving injury. This is the premise of the various algorithms that assess a low level with minimal change in concentration as being a good prognostic indicator and a high level with a large change as a poor prognostic indicator. However, these approaches are affected by analytical issues, thus requiring close collaboration between clinicians and the clinical laboratory to mitigate patient harm [31,32,33,34].
Our findings from assessing three well characterized algorithms demonstrate that for patients with symptoms suggestive of ACS, both the ESC and COMPASS-MI algorithms can identify high-risk patients as rule-in. However, we have also found that serial hsTnI alone cannot safely rule-out patients based on these proposed diagnostic benchmarks. The algorithm that yielded the highest sensitivity was COMPASS-MI, but the estimates (Cohort-1 sensitivity = 97.8% and Cohort-2 sensitivity = 98.6%) were slightly below 99.0% and below the derived estimates from the original publication [5]. These findings reaffirm the importance of assessing clinical signs and symptoms when using hsTn for early rule-out. On a population level, initial data on hsTn testing in Ontario (April 2013 to March 2017) were associated with lower MI, angina, and all-cause hospitalization during a time before the algorithms tested in this study were widely used [21]. These findings in Ontario are in agreement with the RAPID-TnT findings, where the standard protocol missed fewer outcomes as compared to the 0/1 h algorithm [8].

4.1. Limitations

First, our serial sampling occurred later than 2 h after presentation; however, recent data suggest a continuous linear release of hsTn levels in patients with MI [35]; thus, testing these algorithms with later samples (as performed in this analyses) would only improve the sensitivity estimates. Second, our findings are from a single urban city; however, the prevalence of outcomes is similar to other centers within North America [36]. Third, outcomes were assessed differently (i.e., adjudication versus databases); however, administrative databases, as used in Cohort-1, can be a robust source [37]. Fourth, different lower limits of reporting for hsTnI were utilized (0 ng/L in Cohort-1 and 0.9 ng/L in Cohort-2), leading to possible differences of 0.9 ng/L at the low end between the cohorts. However, acceptable variation of testing at the low end has been reported to be 0.8 ng/L, so differences of 1 ng/L could be based on different lower limits or the imprecision of testing alone [27].

4.2. Conclusions

For patients with symptoms suggestive of ACS, both the ESC and COMPASS-MI algorithms can identify high-risk patients as rule-in (see Figure 2 for summary). However, no hsTnI algorithm alone can achieve the sensitivity or NPV estimates required for safely discharging patients as “rule-out MI”. Interventional studies assessing the addition of other laboratory or clinical variables in conjunction with hsTn are needed to determine the safest method of discharging patients home from the ED.

Author Contributions

Conceptualization, P.A.K., M.K.H., S.E.M., A.W.; methodology, P.A.K.; software, P.A.K., J.O.C., J.M.; validation, P.A.K., A.W.; formal analysis, P.A.K., J.O.C., J.M.; investigation, P.A.K., A.W.; resources, P.A.K., A.W., N.C., L.E.G., R.P., H.S., D.T.K.; data curation, P.A.K., J.O.C., J.M., N.C., D.T.K.; writing—original draft preparation, P.A.K.; writing—review and editing, J.O.C., D.T.K., J.M., M.K.H., S.E.M., N.C., M.M., L.E.G., R.P., H.S., C.A., A.W.; visualization, P.A.K.; supervision, P.A.K., A.W., H.S., D.T.K.; project administration, P.A.K., A.W., R.P., H.S., D.T.K.; funding acquisition, P.A.K., A.W. All authors have read and agreed to the published version of the manuscript.

Funding

This study was supported by a Canadian Institutes of Health Research (funding reference number 155964) grant with reagent support from Abbott Laboratories to perform high-sensitivity cardiac troponin I testing in both cohorts.

Institutional Review Board Statement

Ethics approval was obtained by the Hamilton Integrated Research Ethics Board (REB# 13-277 and 4717-D). Patients in Cohort-1 were part of a database study (no consent required, REB 4717-D) and patients in Cohort-2 were enrolled in NCT01994577 and agreed to follow-up (REB#13-277).

Informed Consent Statement

Patients enrolled in Cohort-2 agreed to follow-up per Ethics approved studies (#13-277, 4717-D). Patient consent was waived for Cohort-1 as part of database study (no consent required, REB 4717-D).

Data Availability Statement

The studies were conducted before data sharing processes were in place, and thus individual data are not available. The data are not publicly available due to privacy.

Acknowledgments

This study was supported by ICES, which is funded by an annual grant from the Ontario Ministry of Health (MOH) and the Ministry of Long-Term Care (MLTC). Content enclosed is based on data and information compiled and provided by CIHI. Analyses, conclusions, opinions, and statements expressed herein are those of the authors and not necessarily those of CIHI; no endorsement is intended or should be inferred. We thank IQVIA Solutions Canada Inc. for use of their Drug Information File.

Conflicts of Interest

Kavsak has received grants/reagents/consultant/advisor/honoraria from Abbott Laboratories, Beckman Coulter, Ortho Clinical Diagnostics, Quidel, Randox Laboratories, Roche Diagnostics, Siemens Healthcare Diagnostics, and Thermo Fisher Scientific. McMaster University has filed patents with Kavsak and Worster listed as an inventor in the acute cardiovascular biomarker field; in particular, a European patent has been granted on the following patent: “METHOD OF DETERMINING RISK OF AN ADVERSE CARDIAC EVENT” EP3341723A4. No other disclosures were reported.

Trial Registration

ClinicalTrials.gov Identifiers: NCT01994577.

Sample Availability

Not available.

References

  1. Wu, A.H.B.; Christenson, R.H.; Greene, D.N.; Jaffe, A.S.; Kavsak, P.A.; Ordonez-Llanos, J.; Apple, F.S. Clinical Laboratory Practice Recommendations for the Use of Cardiac Troponin in Acute Coronary Syndrome: Expert Opinion from the Academy of the American Association for Clinical Chemistry and the Task Force on Clinical Applications of Cardiac Bio-Markers of the International Federation of Clinical Chemistry and Laboratory Medicine. Clin. Chem. 2018, 64, 645–655. [Google Scholar]
  2. Thygesen, K.; Alpert, J.S.; Jaffe, A.S.; Chaitman, B.R.; Bax, J.J.; Morrow, D.A.; White, H.D.; Executive Group on behalf of the Joint European Society of Cardiology (ESC)/American College of Cardiology (ACC)/American Heart Association (AHA)/World Heart Federation (WHF) Task Force for the Universal Definition of Myocardial Infarction. Fourth Universal Definition of Myocardial Infarction (2018). Circulation 2018, 138, e618–e651. [Google Scholar] [CrossRef]
  3. Collet, J.P.; Thiele, H.; Barbato, E.; Barthélémy, O.; Bauersachs, J.; Bhatt, D.L.; Dendale, P.; Dorobantu, M.; Edvardsen, T.; Folliguet, T.; et al. 2020 ESC Guidelines for the management of acute coronary syndromes in patients presenting without persistent ST-segment elevation. Eur. Heart J. 2021, 42, 1289–1367. [Google Scholar] [CrossRef]
  4. Reichlin, T.; Twerenbold, R.; Wildi, K.; Gimenez, M.R.; Bergsma, N.; Haaf, P.; Druey, S.; Puelacher, C.; Moehring, B.; Freese, M.; et al. Prospective validation of a 1-hour algorithm to rule-out and rule-in acute myocardial infarction using a high-sensitivity cardiac troponin T assay. CMAJ Can. Med. Assoc. J. 2015, 187, E243–E252. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  5. Neumann, J.T.; Twerenbold, R.; Ojeda, F.; Sörensen, N.A.; Chapman, A.R.; Shah, A.S.V.; Anand, A.; Boeddinghaus, J.; Nestelberger, T.; Badertscher, P.; et al. Application of High-Sensitivity Troponin in Suspected Myocardial Infarction. N. Engl. J. Med. 2019, 380, 2529–2540. [Google Scholar] [CrossRef] [PubMed]
  6. Pickering, J.W.; Greenslade, J.H.; Cullen, L.; Flaws, D.; Parsonage, W.; Aldous, S.; George, P.; Worster, A.; Kavsak, P.A.; Than, M.P. Assessment of the European Society of Cardiology 0-Hour/1-Hour Algorithm to Rule-Out and Rule-In Acute Myocardial Infarction. Circulation 2016, 134, 1532–1541. [Google Scholar] [CrossRef]
  7. Allen, B.R.; Christenson, R.H.; Cohen, S.A.; Nowak, R.; Wilkerson, R.G.; Mumma, B.; Madsen, T.; McCord, J.; Huis In’t Veld, M.; Massoomi, M.; et al. Diagnostic Performance of High-Sensitivity Cardiac Troponin T Strategies and Clinical Variables in a Multisite US Cohort. Circulation 2021, 143, 1659–1672. [Google Scholar] [CrossRef] [PubMed]
  8. Lambrakis, K.; Papendick, C.; French, J.K.; Quinn, S.; Blyth, A.; Seshadri, A.; Edmonds, M.J.R.; Chuang, A.; Khan, E.; Nelson, A.J.; et al. Late Outcomes of the RAPID-TnT Randomized Controlled Trial: 0/1-Hour High-Sensitivity Troponin T Protocol in Suspected ACS. Circulation 2021, 144, 113–125. [Google Scholar] [CrossRef]
  9. Than, M.; Herbert, M.; Flaws, D.; Cullen, L.; Hess, E.; Hollander, J.E.; Diercks, D.; Ardagh, M.W.; Kline, J.A.; Munro, Z.; et al. What is an acceptable risk of major adverse cardiac event in chest pain patients soon after discharge from the Emergency Department? A clinical survey. Int. J. Cardiol. 2013, 166, 752–754. [Google Scholar] [CrossRef] [PubMed]
  10. Shah, A.S.; Anand, A.; Sandoval, Y.; Lee, K.K.; Smith, S.W.; Adamson, P.D.; Chapman, A.R.; Langdon, T.; Sandeman, D.; Vaswani, A.; et al. High-sensitivity cardiac troponin I at presentation in patients with suspected acute coronary syndrome: A cohort study. Lancet 2015, 386, 2481–2488. [Google Scholar] [CrossRef] [Green Version]
  11. Than, M.P.; Pickering, J.W.; Sandoval, Y.; Shah, A.S.V.; Tsanas, A.; Apple, F.S.; Blankenberg, S.; Cullen, L.; Mueller, C.; Neumann, J.T.; et al. Machine Learning to Predict the Likelihood of Acute Myocardial Infarction. Circulation 2019, 140, 899–909. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  12. Kavsak, P.A.; Cerasuolo, J.O.; Ko, D.T.; Ma, J.; Sherbino, J.; Mondoux, S.E.; Perez, R.; Seow, H.; Worster, A. High-Sensitivity Cardiac Troponin I vs a Clinical Chemistry Score for Predicting All-Cause Mortality in an Emergency Department Population. CJC Open 2020, 2, 296–302. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  13. Kavsak, P.A.; Cerasuolo, J.O.; Ko, D.T.; Ma, J.; Sherbino, J.; Mondoux, S.E.; Clayton, N.; Hill, S.A.; McQueen, M.; Griffith, L.E.; et al. Using the clinical chemistry score in the emergency department to detect adverse cardiac events: A diagnostic accuracy study. CMAJ Open 2020, 8, E676–E684. [Google Scholar] [CrossRef]
  14. Anand, A.; Lee, K.K.; Chapman, A.R.; Ferry, A.V.; Adamson, P.D.; Strachan, F.E.; Berry, C.; Findlay, I.; Cruickshank, A.; Reid, A.; et al. High-Sensitivity Cardiac Troponin on Presentation to Rule Out Myocardial Infarction: A Stepped-Wedge Cluster Randomized Controlled Trial. Circulation 2021, 143, 2214–2224. [Google Scholar] [CrossRef]
  15. Pickering, J.W. The Need to Improve Derivation and Description of Algorithms to Rule-Out Patients with Possible Myocardial Infarction. Circulation 2019, 139, 1351–1353. [Google Scholar] [CrossRef] [PubMed]
  16. Bossuyt, P.M.; Reitsma, J.B.; Bruns, D.E.; Gatsonis, C.A.; Glasziou, P.P.; Irwig, L.; Lijmer, J.G.; Moher, D.; Rennie, D.; de Vet, H.C.; et al. STARD 2015: An updated list of essential items for reporting diagnostic accuracy studies. BMJ 2015, 351, h5527. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  17. Kavsak, P.A.; Neumann, J.T.; Cullen, L.; Than, M.; Shortt, C.; Greenslade, J.H.; Pickering, J.W.; Ojeda, F.; Ma, J.; Clayton, N.; et al. Clinical chemistry score versus high-sensitivity cardiac troponin I and T tests alone to identify patients at low or high risk for myocardial infarction or death at presentation to the emergency department. CMAJ Can. Med. Assoc. J. 2018, 190, E974–E984. [Google Scholar] [CrossRef] [Green Version]
  18. Kavsak, P.A.; Cerasuolo, J.O.; Mondoux, S.E.; Sherbino, J.; Ma, J.; Hoard, B.K.; Perez, R.; Seow, H.; Ko, D.T.; Worster, A. Risk Stratification for Patients with Chest Pain Discharged Home from the Emergency Department. J. Clin. Med. 2020, 9, 2948. [Google Scholar] [CrossRef]
  19. Shortt, C.; Ma, J.; Clayton, N.; Sherbino, J.; Whitlock, R.; Pare, G.; Hill, S.A.; McQueen, M.; Mehta, S.R.; Devereaux, P.J.; et al. Rule-In and Rule-Out of Myocardial Infarction Using Cardiac Troponin and Glycemic Biomarkers in Patients with Symptoms Suggestive of Acute Coronary Syndrome. Clin. Chem. 2017, 63, 403–414. [Google Scholar] [CrossRef]
  20. Kavsak, P.A.; Worster, A.; Ma, J.; Shortt, C.; Clayton, N.; Sherbino, J.; Hill, S.A.; McQueen, M.; Mehta, S.R.; Devereaux, P.J. High-Sensitivity Cardiac Troponin Risk Cutoffs for Acute Cardiac Outcomes at Emergency Department Presentation. Can. J. Cardiol. 2017, 33, 898–903. [Google Scholar] [CrossRef] [PubMed]
  21. Lau, G.; Koh, M.; Kavsak, P.A.; Schull, M.J.; Armstrong, D.W.J.; Udell, J.A.; Austin, P.C.; Wang, X.; Ko, D.T. Clinical outcomes for chest pain patients discharged home from emergency departments using high-sensitivity versus conventional cardiac troponin assays. Am. Heart J. 2020, 221, 84–94. [Google Scholar] [CrossRef]
  22. Kavsak, P.A.; Pardhan, A.; Krizmanich, W.; Worster, A. Hospital Admission and Myocardial Injury Prevalence after the Clinical Introduction of a High-Sensitivity Cardiac Troponin I Assay. Clin. Chem. 2015, 61, 1209–1210. [Google Scholar] [CrossRef] [Green Version]
  23. Kufaishi, H.; Pardhan, A.; Krizmanich, W.; Worster, A.; Hill, S.; Kavsak, P.A. Adopting ‘ng/L’ as the units for high-sensitivity cardiac troponin assays and commitment by the entire health-care team could be the key for adopting recommendations. Ann. Clin. Biochem. 2016, 53 Pt 4, 516–517. [Google Scholar] [CrossRef] [Green Version]
  24. Kavsak, P.A.; Mondoux, S.E.; Ma, J.; Sherbino, J.; Hill, S.A.; Clayton, N.; Mehta, S.R.; Griffith, L.E.; McQueen, M.; Devereaux, P.J.; et al. Comparison of two biomarker only algorithms for early risk stratification in patients with suspected acute coronary syndrome. Int. J. Cardiol. 2020, 319, 140–143. [Google Scholar] [CrossRef]
  25. Hickman, P.E.; Koerbin, G.; Badrick, T.; Oakman, C.; Potter, J.M. The importance of low level QC for high sensitivity troponin assays. Clin. Biochem. 2018, 58, 60–63. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  26. Wildi, K.; Boeddinghaus, J.; Nestelberger, T.; Haaf, P.; Koechlin, L.; Ayala Lopez, P.; Walter, J.; Badertscher, P.; Ratmann, P.D.; Miró, Ò.; et al. External validation of the clinical chemistry score. Clin. Biochem. 2021, 91, 16–25. [Google Scholar] [CrossRef]
  27. Kavsak, P.A.; Jaffe, A.S.; Greene, D.N.; Christenson, R.H.; Apple, F.S.; Wu, A.H.B. Total Analytic Error for Low Cardiac Troponin Concentrations (≤10 ng/L) by Use of a High-Sensitivity Cardiac Troponin Assay. Clin. Chem. 2017, 63, 1043–1045. [Google Scholar] [CrossRef]
  28. Duceppe, E.; Borges, F.; Tiboni, M.; Pearse, R.; Chan, M.; Srinathan, S.; Kavsak, P.; Szalay, D.; Garg, A.; Sessler, D.; et al. Association between high-sensitivity troponin I and major cardiovascular events after non-cardiac surgery. J. Am. Coll. Cardiol. 2020, 75 (Suppl. 1), 110. [Google Scholar] [CrossRef]
  29. Pickering, J.W.; Than, M.P.; Cullen, L.; Aldous, S.; Ter Avest, E.; Body, R.; Carlton, E.W.; Collinson, P.; Dupuy, A.M.; Ekelund, U.; et al. Rapid rule-out of acute myocardial Infarction with a single high-sensitivity cardiac troponin T measurement below the limit of detection: A Collaborative Meta-analysis. Ann. Intern. Med. 2017, 166, 715–724. [Google Scholar] [CrossRef] [PubMed]
  30. Chapman, A.R.; Lee, K.K.; McAllister, D.A.; Cullen, L.; Greenslade, J.H.; Parsonage, W.; Worster, A.; Kavsak, P.A.; Blankenberg, S.; Neumann, J.; et al. Association of high-sensitivity cardiac troponin I concentration with cardiac outcomes in patients with suspected acute coronary syndrome. JAMA J. Am. Med. Assoc. 2017, 318, 1913–1924. [Google Scholar] [CrossRef] [PubMed]
  31. Hammarsten, O.; Jacobsson, C.E.; Widegren, M.; Danylchenko, T.; Jaffe, A.S. Long-time quality assessment of the Elecsys Troponin T hs assay. Clin. Biochem. 2013, 46, 1055–1057. [Google Scholar] [CrossRef] [PubMed]
  32. Clifford-Mobley, O. Using the European Society of Cardiology 1-h algorithm for ruling out non-ST-segment elevated myocardial infarction to define acceptable analytical performance limits for a cardiac troponin T assay. Ann. Clin. Biochem. 2021, 58, 157. [Google Scholar] [CrossRef] [PubMed]
  33. Kavsak, P.A.; Clark, L.; Martin, J.; Mark, C.T.; Paré, G.; Mondoux, S.; Chetty, V.T.; Ainsworth, C.; Worster, A. Acute Phase Response and Non-Reproducible Elevated Concentrations with a High-Sensitivity Cardiac Troponin I Assay. J. Clin. Med. 2021, 10, 1014. [Google Scholar] [CrossRef]
  34. Kavsak, P.A.; Mondoux, S.E.; Martin, J.; Hewitt, M.K.; Clark, L.; Caruso, N.; Mark, C.T.; Chetty, V.T.; Ainsworth, C.; Worster, A. Disagreement between Cardiac Troponin Tests Yielding a Higher Incidence of Myocardial Injury in the Emergency Setting. J. Cardiovasc. Dev. Dis. 2021, 8, 31. [Google Scholar] [CrossRef]
  35. Sandoval, Y.; Chapman, A.R.; Mills, N.L.; Than, M.; Pickering, J.W.; Worster, A.; Kavsak, P.; Apple, F.S. Sex-Specific Kinetics of High-Sensitivity Cardiac Troponin I and T following Symptom Onset and Early Presentation in Non-ST-Segment Elevation Myocardial Infarction. Clin. Chem. 2021, 67, 321–324. [Google Scholar] [CrossRef]
  36. Sandoval, Y.; Nowak, R.; deFilippi, C.R.; Christenson, R.H.; Peacock, W.F.; McCord, J.; Limkakeng, A.T.; Sexter, A.; Apple, F.S. Myocardial Infarction Risk Stratification with a Single Measurement of High-Sensitivity Troponin I. J. Am. Coll. Cardiol. 2019, 74, 271–282. [Google Scholar] [CrossRef] [PubMed]
  37. Meah, M.N.; Denvir, M.A.; Mills, N.L.; Norrie, J.; Newby, D.E. Clinical endpoint adjudication. Lancet 2020, 395, 1878–1882. [Google Scholar] [CrossRef]
Figure 1. Flow diagram of the two cohorts.
Figure 1. Flow diagram of the two cohorts.
Jcdd 08 00097 g001
Figure 2. Summary figure of the performance of three different hsTnI algorithms for 30-day all-cause death or MI in undifferentiated ED patients with serial hsTnI measurements (Cohort-1) and ED patients with symptoms suggestive of ACS with serial hsTnI measurements (Cohort-2).
Figure 2. Summary figure of the performance of three different hsTnI algorithms for 30-day all-cause death or MI in undifferentiated ED patients with serial hsTnI measurements (Cohort-1) and ED patients with symptoms suggestive of ACS with serial hsTnI measurements (Cohort-2).
Jcdd 08 00097 g002
Table 1. Algorithms used for rule-out and rule-in with hsTnI testing.
Table 1. Algorithms used for rule-out and rule-in with hsTnI testing.
ESC 0/2 h algorithmRule-out: if 1st hsTnI < 6 ng/L and absolute change between 2nd and 1st sample < 2 ng/LRule-in: if 1st ≥ 64 ng/L or absolute change ≥ 15 ng/L between 2nd and 1st sample
COMPASS-MI algorithmRule-out: if 1st hsTnI < 4 ng/L and absolute change between 2nd and 1st sample < 4 ng/LRule-in: if 1st ≥ 60 ng/L or absolute change ≥ 18 ng/L between 2nd and 1st sample
High-STEACSRule-out: if 1st hsTnI ≤ sex-specific URLs and absolute change between 2nd and 1st sample < 3 ng/LRule-in: if 1st hsTnI > sex-specific URLs or absolute change between 2nd and 1st sample ≥ 3 ng/L
Table 2. Demographics and clinical history of the two ED cohorts characterized by the different rule-out and rule-in hsTnI algorithms.
Table 2. Demographics and clinical history of the two ED cohorts characterized by the different rule-out and rule-in hsTnI algorithms.
VariableValueCohort-1 (n = 2966)Cohort-2 (n = 935)
UncertainRule-In GroupRule-Out Groupp-ValueUncertainRule-In GroupRule-Out Groupp-Value
ESC absolute criteria
NumberCount1264837865-401109425-
Age (years)Median (IQR)77 (65–84)78 (66–86)60 (50–71)<0.00177 (64–85)77 (63–85)59 (48–70)<0.001
SexFemale (%)612 (48.4%)397 (47.4%)453 (52.4%)0.09210 (52.4%)49 (45.0%)237 (55.8%)0.12
History of arrhythmiaCount (%)311 (24.6%)203 (24.3%)81 (9.4%)<0.001131 (32.7%)39 (35.8%)56 (13.2%)<0.001
History of heart failureCount (%)417 (33.0%)307 (36.7%)93 (10.8%)<0.001114 (28.4%)40(36.7%)27 (6.4%)<0.001
History of diabetesCount (%)509 (40.3%)356 (42.5%)220 (25.4%)<0.001137 (34.2%)45 (41.3%)92 (21.7%)<0.001
History of myocardial infarctionCount (%)217 (17.2%)167 (20.0%)61 (7.1%)<0.001175 (43.6%)54 (49.5%)109 (25.7%)<0.001
COMPASS-MI absolute criteria
NumberCount1448821697-488111336-
Age (years)Median (IQR)76 (65–84)78 (66–86)57 (48–67)<0.00176 (63–84)77 (64–85)56 (46–67)<0.001
SexFemale (%)711 (49.1%)386 (47.0%)365 (52.4%)0.113266 (54.5%)50 (45.1%)180 (53.6%)0.191
History of arrhythmiaCount (%)340 (23.5%)201 (24.5%)54 (7.7%)<0.001142 (29.1%)42 (37.8%)42 (12.5%)<0.001
History of heart failureCount (%)464 (32.0%)300 (36.5%)53 (7.6%)<0.001122 (25.0%)43 (38.7%)16 (4.8%)<0.001
History of diabetesCount (%)566 (39.1%)352 (42.9%)167 (24.0%)<0.001156 (32.0%)45 (40.5%)73 (21.7%)0.001
History of myocardial infarctionCount (%)238 (16.4%)157 (19.1%)50 (7.2%)<0.001210 (43.0%)53 (47.8%)75 (22.3%)<0.001
High-STEACS absolute criteria
NumberCount 15511415- 287648-
Age (years)Median (IQR) 78 (66–85)66 (54–78)<0.001 78 (64–85)64 (53–77)<0.001
SexFemale (%) 771 (49.7%)691 (48.8%)0.634 160 (55.8%)336 (51.9%)0.271
History of arrhythmiaCount (%) 374 (24.1%)221 (15.6%)<0.001 99 (34.5%)127 (19.6%)<0.001
History of heart failureCount (%) 550 (35.5%)267 (18.9%)<0.001 97 (33.8%)84 (13.0%)<0.001
History of diabetesCount (%) 646 (41.7%)439 (31.0%)<0.001 102 (35.5%)172 (26.5%)0.015
History of myocardial infarctionCount (%) 287 (18.5%)158 (11.2%)<0.001 136 (47.4%)202 (31.2%)<0.001
Table 3. The hs-cTnI levels and 30-day outcomes of the two ED cohorts characterized by the three hsTnI algorithms.
Table 3. The hs-cTnI levels and 30-day outcomes of the two ED cohorts characterized by the three hsTnI algorithms.
VariableValueCohort-1 (n = 2966)Cohort-2 (n = 935)
UncertainRule-In GroupRule-Out Groupp-ValueUncertainRule-In GroupRule-Out Groupp-Value
ESC absolute criteria
NumberCount1264837865-401109425-
Time elapsed (hours) between 1st and 2nd hsTnIMedian (IQR)8 (6–13)8 (7–11)6 (4–9)<0.0013 (3–3)3 (3–3)3 (3–3)<0.001
1st hsTnI (ng/L)Median (IQR)13 (8–26)89 (35–235)3 (2–4)<0.00112 (7–23)99 (41–229)2 (1–4)<0.001
2nd hsTnI (ng/L)Median (IQR)14 (9–26)137 (67–492)3 (2–4)<0.00113 (8–5)188 (80–695)2 (1–4)<0.001
Absolute difference hsTnI (ng/L) between samplesMedian (IQR)3 (1–6)56 (21–250)1 (0–1)<0.0012 (1–3)52 (15–288)0 (0–1)<0.001
All-cause mortality or MI at 30 daysCount (%)108 (8.5%)328 (39.2%)18 (2.1%)<0.00155 (13.7%)85 (78.0%)6 (1.4%)<0.001
COMPASS-MI absolute criteria
NumberCount1448821697-488111336-
Time elapsed (hours) between 1st and 2nd hsTnIMedian (IQR)8 (6–12)8 (7–10)6 (4–9)<0.0013 (3–3)3 (3–3)3 (3–3)<0.001
1st hsTnI (ng/L)Median (IQR)11 (6–23)93 (39–241)2 (1–3)<0.00110 (5–19)90 (50–229)2 (1–3)<0.001
2nd hsTnI (ng/L)Median (IQR)12 (7–23)142 (72–506)2 (2–3)<0.00111 (6–21)182 (77–695)2 (1–3)<0.001
Absolute difference hsTnI (ng/L) between samplesMedian (IQR)2 (1–5)56 (22–255)1 (0–1)<0.0011 (0–3)51 (11–288)0 (0–1)<0.001
All-cause mortality or MI at 30 daysCount (%)119 (8.2%)325 (39.6%)10 (1.4%)<0.00159 (12.1%)85 (76.6%)2 (0.6%)<0.001
High-STEACS absolute criteria
NumberCount 15511415- 287648-
Time elapsed (hours) between 1st and 2nd hsTnIMedian (IQR) 8 (7–12)7 (4–10)<0.001 3 (3–3)3 (3–3)<0.001
1st hsTnI (ng/L)Median (IQR) 38 (16–100)4 (2–8)<0.001 30 (15–66)3 (2–7)<0.001
2nd hsTnI (ng/L)Median (IQR) 48 (21–157)5 (2–8)<0.001 38 (21–100)4 (2–7)<0.001
Absolute difference hsTnI (ng/L) between samplesMedian (IQR) 14 (5–64)1 (0–1)<0.001 6 (3–22)1 (0–1)<0.001
All-cause mortality or MI at 30 daysCount (%) 398 (25.7%)56 (4.0%)<0.001 124 (43.2%)22 (3.4%)<0.001
Table 4. Diagnostic performance of the different algorithms for death or MI at 30 days in the two ED cohorts.
Table 4. Diagnostic performance of the different algorithms for death or MI at 30 days in the two ED cohorts.
GroupSensitivitySpecificityPPVNPVPositive LRNegative LRSensitivitySpecificityPPVNPVPositive LRNegative LR
Cohort-1 ESC absolute criteriaCohort-2 ESC absolute criteria
Rule-out96.0%
(93.8–97.6%)
33.7%
(31.9–35.6%)
20.8%
(19.0–22.6%)
97.9%
(96.7–98.8%)
1.45
(1.40–1.50)
0.12
(0.06–0.17)
95.9%
(91.3–98.5%)
53.1%
(49.6–56.6%)
27.5%
(25.9–29.1%)
98.6%
(97.0–99.4%)
2.04
(1.88–2.22)
0.08
(0.04–0.17)
Rule-in72.2%
(67.9–76.3%)
79.7%
(78.1–81.3%)
39.2%
(35.9–42.6%)
94.1%
(93.0–95.0%)
3.57
(3.22–3.91)
0.35
(0.30–0.40)
58.2%
(49.8–66.3%)
97.0%
(95.5–98.0%)
78.0%
(70.0–84.3%)
92.6%
(91.2–93.3%)
19.14
(12.61–29.05)
0.43
(0.36–0.52)
Cohort-1 COMPASS-MI absolute criteriaCohort-2 COMPASS-MI absolute criteria
Rule-out97.8%
(96.0–98.9%)
27.3%
(25.6–29.1%)
19.6%
(18.0–21.3%)
98.6%
(97.4–99.3%)
1.35
(1.31–1.38)
0.08
(0.03–0.13)
98.6%
(95.1–99.8%)
42.3%
(38.9–45.9%)
24.0%
(22.9–25.2%)
99.4%
(97.7–99.9%)
1.71
(1.61–1.82)
0.03
(0.01–0.13)
Rule-in71.6%
(67.2–75.7%)
80.3%
(78.6–81.8%)
39.6%
(36.2–43.0%)
94.0%
(92.9–95.0%)
3.63
(3.27–3.98)
0.35
(0.30–0.41)
58.2%
(49.8–66.3%)
96.7%
(95.2–97.8%)
76.6%
(68.6–83.0%)
92.6%
(91.2–93.8%)
17.67
(11.82–26.41)
0.43
(0.36–0.52)
Cohort-1 High-STEACS absolute criteriaCohort-2 High-STEACS absolute criteria
Rule-out and Rule-in87.7%
(84.2–90.5%)
54.1%
(52.1–56.1%)
25.7%
(23.5–27.9%)
96.0%
(94.9–97.0%)
1.91
(1.81–2.01)
0.23
(0.17–0.29)
84.9%
(78.1–90.3%)
79.3%
(76.4–82.1%)
43.2%
(39.5–47.0%)
96.6%
(95.1–97.7%)
4.11
(3.53–4.79)
0.19
(0.13–0.28)
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Kavsak, P.A.; Hewitt, M.K.; Mondoux, S.E.; Cerasuolo, J.O.; Ma, J.; Clayton, N.; McQueen, M.; Griffith, L.E.; Perez, R.; Seow, H.; et al. Diagnostic Performance of Serial High-Sensitivity Cardiac Troponin Measurements in the Emergency Setting. J. Cardiovasc. Dev. Dis. 2021, 8, 97. https://doi.org/10.3390/jcdd8080097

AMA Style

Kavsak PA, Hewitt MK, Mondoux SE, Cerasuolo JO, Ma J, Clayton N, McQueen M, Griffith LE, Perez R, Seow H, et al. Diagnostic Performance of Serial High-Sensitivity Cardiac Troponin Measurements in the Emergency Setting. Journal of Cardiovascular Development and Disease. 2021; 8(8):97. https://doi.org/10.3390/jcdd8080097

Chicago/Turabian Style

Kavsak, Peter A., Mark K. Hewitt, Shawn E. Mondoux, Joshua O. Cerasuolo, Jinhui Ma, Natasha Clayton, Matthew McQueen, Lauren E. Griffith, Richard Perez, Hsien Seow, and et al. 2021. "Diagnostic Performance of Serial High-Sensitivity Cardiac Troponin Measurements in the Emergency Setting" Journal of Cardiovascular Development and Disease 8, no. 8: 97. https://doi.org/10.3390/jcdd8080097

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop