Next Article in Journal
Presence of Antibodies Binding to Negative Elongation Factor E in Sarcoidosis
Previous Article in Journal
Analysis of Cervical Spine Alignment and its Relationship with Other Spinopelvic Parameters after Laminoplasty in Patients with Degenerative Cervical Myelopathy
Previous Article in Special Issue
Additional Value of Machine-Learning Computed Tomographic Angiography-Based Fractional Flow Reserve Compared to Standard Computed Tomographic Angiography
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
JCM
Volume 9
Issue 3
10.3390/jcm9030714
jcm-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Comparison of Machine Learning Computed Tomography-Based Fractional Flow Reserve and Coronary CT Angiography-Derived Plaque Characteristics with Invasive Resting Full-Cycle Ratio

by
Stefan Baumann
1,2,*,
Markus Hirt
1,2,
Christina Rott
1,2,
Gökce H. Özdemir
1,2,
Christian Tesche
3,
Tobias Becher
1,2,4,
Christel Weiss
5,
Svetlana Hetjens
5,
Ibrahim Akin
1,2,
Stefan O. Schoenberg
6,
Martin Borggrefe
1,2,
Sonja Janssen
6,
Daniel Overhoff
6 and
Dirk Lossnitzer
1,2
1
First Department of Medicine-Cardiology, University Medical Centre Mannheim, and DZHK (German Centre for Cardiovascular Research), partner site Heidelberg/Mannheim, Mannheim 68167, Germany
2
European Center for Angioscience, Mannheim 68167, Germany
3
Department of Internal Medicine, St. Johannes-Hospital, Dortmund 44137, Germany
4
Laboratory of Molecular Metabolism, The Rockefeller University, New York NY 10065, USA
5
Medical Faculty Mannheim, Department of Medical Statistics and Biomathematics, University Medical Center Mannheim, Heidelberg University, Mannheim 68167, Germany
6
Institute of Clinical Radiology and Nuclear Medicine, University Medical Center Mannheim, Faculty of Medicine Mannheim, Heidelberg University, Mannheim 68167, Germany
*
Author to whom correspondence should be addressed.
J. Clin. Med. 2020, 9(3), 714; https://doi.org/10.3390/jcm9030714
Submission received: 19 January 2020 / Revised: 26 February 2020 / Accepted: 3 March 2020 / Published: 6 March 2020
(This article belongs to the Special Issue Coronary Physiology in Clinical Practice: Fractional Flow Reserve and Beyond)
Browse Figure
Versions Notes

Abstract

:
Background: The aim is to compare the machine learning-based coronary-computed tomography fractional flow reserve (CT-FFRML) and coronary-computed tomographic morphological plaque characteristics with the resting full-cycle ratio (RFRTM) as a novel invasive resting pressure-wire index for detecting hemodynamically significant coronary artery stenosis. Methods: In our single center study, patients with coronary artery disease (CAD) who had a clinically indicated coronary computed tomography angiography (cCTA) and subsequent invasive coronary angiography (ICA) with pressure wire-measurement were included. On-site prototype CT-FFRML software and on-site CT-plaque software were used to calculate the hemodynamic relevance of coronary stenosis. Results: We enrolled 33 patients (70% male, mean age 68 ± 12 years). On a per-lesion basis, the area under the receiver operating characteristic curve (AUC) of CT-FFRML (0.90) was higher than the AUCs of the morphological plaque characteristics length/minimal luminal diameter4 (LL/MLD4; 0.80), minimal luminal diameter (MLD; 0.77), remodeling index (RI; 0.76), degree of luminal diameter stenosis (0.75), and minimal luminal area (MLA; 0.75). Conclusion: CT-FFRML and morphological plaque characteristics show a significant correlation to detected hemodynamically significant coronary stenosis. Whole CT-FFRML had the best discriminatory power, using RFRTM as the reference standard.

1. Introduction

Assessment of coronary artery stenosis with coronary computed tomographic angiography (cCTA) and additional computed tomographic morphological plaque characteristics derived from cCTA datasets leads to a more detailed evaluation of suspicious stenosis [1]. Two studies published recently showed that the joined assessment of some computed tomographic morphological plaque characteristics and cCTA could improve the detection of hemodynamically relevant stenosis [2,3]. In addition, the functional relevance of a lesion may be assessed by cCTA-based fractional flow reserve (CT-FFR) [4]. Several studies have shown a significant correlation between non-invasive CT-FFR and invasive fractional flow reserve (FFR) [2,5,6]. Using an on-site prototype with a machine-learning algorithm (CT-FFRML) to reduce computation time was the consequent next step in technical development [7,8,9,10]. CT-FFRML was compared in previously published studies to the invasive gold standards invasive FFR [8] and instantaneous wave-free ratio (iwFR) [11] and showed promising results.
Invasive diagnostic tools currently used to assess the hemodynamic relevance of coronary artery stenosis during invasive coronary angiography (ICA) are invasive FFR and iwFR [12,13]. These two methods are both recommended by the 2018 European Society of Cardiology (ESC) guidelines on myocardial revascularization as a class IA recommendation [12,14,15].
Resting full-cycle ratio (RFRTM) is a recently developed invasive parameter to determine the hemodynamic relevance of coronary artery stenosis. Compared to invasive FFR, it is not necessary to induce hyperemia for RFRTM, resulting in improved patient comfort. The advantage of this novel resting index approach may consist of detecting lower values during systole, while iwFR measures values during diastole. RFRTM showed an excellent correlation to the gold standard iwFR in recent studies [16,17].
The aim of our study is to compare the machine learning-based coronary-computed tomography fractional flow reserve (CT-FFRML) and coronary-computed tomographic morphological plaque characteristics calculated based on coronary-computed tomography angiography (cCTA) with the resting full-Cycle ratio (RFRTM).

2. Materials and Methods

2.1. Study Design and Patient Population

The local Institutional Review Board approved the study protocol (No. 2015-583N-MA). Written informed consent was obtained from all patients. Patients with suspected coronary artery disease (CAD) underwent a clinically indicated cCTA and ICA with pressure wire measurement. The CAD consortium clinical score was applied to determine the pre-test probability for CAD and estimate the probability based on age, sex, symptoms, and cardiovascular risk factors [18]. The acquired cCTA datasets were used to calculate CT-FFRML and computed tomographic morphological plaque characteristics. Four patients had to be excluded due to inadequate image quality. Patients were included if they had at least one coronary artery with stenosis of undetermined hemodynamic relevance, defined as a 40% to 70% diameter stenosis by visual assessment [15]. These lesions were further assessed by pressure wire measurements (VerrataTM pressure wire, Volcano Corporation, Koninklijke Philips N.V. Amsterdam, The Netherlands) [14]. After the procedures, the raw data of the iwFR measurement were extracted and analyzed with the RFRTM algorithm by an external core lab (Abbott GmbH & Co. KG; Wiesbaden, Germany). Exclusion criteria for cCTA were electrocardiographic signs of acute myocardial ischemia, unstable angina with elevated serum cardiac biomarkers, patients with renal insufficiency, and known contrast agent allergies. The exclusion criteria for CT-FFRML calculation and computed tomographic plaque characteristics analysis included left main or complex bifurcation stenosis, aneurysms, severe diffuse disease, chronic total occlusion, previous percutaneous coronary stent implantation or coronary artery bypass graft (CABG), or inadequate image quality. Data for baseline characteristics were obtained from electronic medical records.

2.2. Acquisition and Analysis of cCTA Datasets

A third-generation dual-source CT (Siemens Somatom FORCE, Siemens Healthineers, Forchheim, Germany) was used for imaging. A specified regime of medication consisting of sublingual nitroglycerin (0.8 mg) and intravenous beta-blockers were given prior to the scan if deemed necessary by a radiologist. Initially, 80 mL iodinated contrast material (Iomeron 400; Bracco Imaging S.p.A., Milan, Italy) was administered using a power injector (Stellant D; Medrad, Warrendale, PA, USA) at a flow rate of 5 mL/s followed by a 50 mL saline chaser. Coronary artery stenosis was analyzed and quantified using on-site software (Coronary Plaque Analysis 2.0. syngo. via FRONTIER, Siemens Healthineers). The grading was done in accordance with the guidelines of the Society of Cardiovascular Computed Tomography [19].

2.3. Analysis of Machine Learning Computed Tomography-Based Fractional Flow Reserve and Computed Tomographic Morphological Plaque Characteristics

The Leiden-risk score that combines different plaque characteristics to optimize risk stratification was calculated to characterize the complexity and severity of CAD [20]. To quantify coronary calcification, the Agatston score was calculated using a software application, according to the Agatston scoring convention (CaScore, Siemens Healthineers) [21]. The cCTA datasets were used for morphological plaque analysis calculated by on-site software (Coronary Plaque Analysis 2.0.). The target lesion was defined as an area with atherosclerotic changes between non-affected proximal and distal sections without any atherosclerotic changes. Discrepancies in plaque density measured in Hounsfield units (HU) were used to analyze the plaque morphology [2]. The software automatically calculated the parameters lesion length (LL), vessel volume (VV), and total plaque volume (TPV) [3]. LL was defined as the dimension of the plaque between areas free of atherosclerotic plaque. Minimal luminal area (MLA) was measured automatically by the on-site prototype software as the lowest value for the luminal area within the lesion. Corrected coronary opacification (CCO) was defined as the difference between the lowest mean HU attenuation at the proximal extent of stenosis and the lowest mean HU attenuation in the coronary artery distal to the lesion and was calculated manually [22]. The minimal luminal diameter (MLD) was measured, although manually, as the area with the narrowest luminal diameter in the lesion [2]. The remodeling index (RI) was calculated as the ratio of the smallest cross-sectional area of the atherosclerotic lesion over the respective proximal luminal area [23]. On-site CT-FFR prototype software based on a machine-learning algorithm (Siemens cFFR, version 3.1; Siemens Healthineers, currently not commercially available) was used and installed on a regular workstation (Syngo VE36A; Siemens Healthineers) for the CT-FFRML calculation. An experienced cardiovascular radiologist checked the semi-automatically calculated vessel-specific centerlines for the accurate recognition of suspected coronary artery stenosis. The exact percentage of manual centerline corrections was initially not documented. However, the algorithm has captured excellently in most cases, and the rate of manual corrections was fewer than 5%. The on-site prototype software calculated blood flow and showed the hyperemic state in the coronary vessels based on patient-specific physiological conditions and flow dynamic models. The technical specification of the used CT-FFRML algorithm is described in detail in a previously published study [8]. After on-site calculation, the CT-FFRML software created a patient-specific anatomic color-coded 3-dimensional mesh of the coronary artery tree and aortic root. A CT-FFRML cut off value of ≤ 0.80 was used for the detection of hemodynamically relevant coronary artery stenosis [24].

2.4. Invasive Coronary Angiography and Resting Full Cycle Ratio Measurement

A coronary-pressure guide wire (VerrataTM pressure wire, Volcano Corporation) was introduced in the respective coronary artery [12,15]. The pressure sensor was placed distal to the stenosis. The optimal diastolic interval with minimized and constant microvascular resistance for pressure measurement was calculated by dedicated software (Volcano Corporation). IwFR values ≤ 0.89 were considered as diagnostic relevant for lesion-specific ischemia [12,15,25]. The raw data of the iwFR measurement was extracted from the console and an external core lab (Abbott GmbH & Co. KG) retrospectively calculated the RFRTM with the new algorithm, using a hyperemia-free resting measurement for the coronary pressure at the lowest point of resting diastolic-pressure-to-aortic-pressure ratio. The maximal relative pressure difference in the whole cardiac cycle (either in systole or diastole) in five consecutive cardiac cycles was measured. The cut-off value for hemodynamic relevance for RFRTM was equal to the value for iwFR ≤ 0.89 [16,17].

2.5. Statistical Analysis

SAS (version 9.4, SAS Institute Inc., Cary, NC, USA) was used for all analyses. For quantitative variables, mean values and standard deviations were calculated. Categorical variables are presented as percentages, whereas continuous variables are presented as either mean ± standard deviation (SD) or median with interquartile range (IQR) and were analyzed with the independent sample t-test. For qualitative factors, absolute and relative frequencies are presented. In order to quantify the degree to which two measurement methods agree, interclass correlation (ICC) was estimated using a two-way ANOVA with one of the methods being equivalent to 100% consistency. Additionally, Kappa coefficients were calculated to assess the degree of agreement for binary factors. Sensitivity, specificity, positive predictive value, and negative predictive value were evaluated on a per-lesion and per-patient level for CT-FFRML and coronary-computed tomographic morphological plaque characteristics calculated based cCTA, using RFRTM (≤0.89) as the reference standard to detect lesion-specific ischemia. The area under the receiver operating characteristics curve (AUC) was determined and compared for cCTA, CT-FFRML, and RFRTM as a metric of overall diagnostic performance. For computed tomographic morphological plaque characteristics that showed statistically significant correlation with the RFRTM values, sensitivity, specificity, positive predictive value, and negative predictive value were calculated. A p-value of ≤0.05 was considered statistically significant. The AUC was determined for the significant CT-FFRML and morphological plaque characteristics as a metric of overall diagnostic performance.

3. Results

We enrolled 33 patients (70% male, mean age 68 ± 12 years) with acquired cCTA datasets, CT-FFRML and morphological plaque characteristics calculations, and RFRTM measurements. On average, we calculated a pretest probability of 57% ± 19%, which was calculated with the CAD consortium clinical score [18]. Baseline characteristics of patients are presented in Table 1.
Forty-four vessel lesions in 33 patients were analyzed. Fourteen (32%) were identified as hemodynamically relevant stenoses by RFRTM (RFRTM ≤ 0.89), whereas CT-FFRML classified only 13 (30%) as hemodynamically significant coronary artery stenoses (CT-FFRML ≤ 0.80). The mean calculation time for the CT-FFRML was 11 ± 2 min, and the mean calculation time for the analysis of the morphological plaque markers was 15 ± 5 min. An example case is presented in Figure 1.
The cCTA-derived quantitative markers, LL/MLD4, MLD, the degree of luminal diameter stenosis, MLA, CT-FFRML, and RI, showed statistically significant differences between obstructive and non-obstructive lesions compared to the reference standard RFRTM (Table 2).
Lesion length (LL), total plaque volume (TPV), vessel volume (VV), and corrected coronary opacification (CCO) did not achieve statistical significance and were unable to detect lesion-specific ischemia (Table 2). We further assessed the sensitivity, specificity, positive predictive values and negative predictive values and accuracy for CT-FFRML and morphological plaque characteristics with statistical significance as well as the area under the receiver operating characteristic curve (AUC).
On a per-lesion and per-patient level, for CT-FFRML, the parameter with the highest discriminatory power, the sensitivity, specificity, positive predictive value, and a negative predictive value are demonstrated in Table 3.
Accordingly, LL/MLD4, as the best representative coronary computed tomographic morphological plaque characteristic, on a per-lesion and per-patient basis yielded a sensitivity of 71% (95% CI: 42–92%) and 75% (95% CI: 73–94%), a specificity of 77% (95% CI: 58–90%) and 71% (95% CI: 48–84%), a positive predictive value of 59% (95% CI: 33–82%) and 60% (95% CI: 32–84%), and a negative predictive value of 85% (95% CI: 66–96%) and 83% (95% CI: 59–96%), respectively. In comparison, using RFRTM as the reference standard, the sensitivity, specificity, positive predict value, negative predict value and accuracy for MLD, the degree of luminal diameter stenosis, MLA, RI, cCTA (>50%), and cCTA (>70%) are lower than CT-FFRML and LL/MLD4 (Table 3) for detecting lesion-specific ischemia.
When compared to RFRTM, the diagnostic accuracy for detecting hemodynamically significant coronary artery stenoses of CT-FFRML and LL/MLD4 on a per-lesion and a per-patient level was 93% (95% CI: 81–98%) and 91% (95% CI: 76–98%) and 75% (95% CI: 60–87%) and 73% (95% CI: 55–87%), respectively. For MLD, the degrees of accuracy of luminal diameter stenosis, MLA, RI, cCTA (>50%), and cCTA (>70%) are also listed in Table 3.
The ROC curve analysis on a per lesion level for CT-FFRML and coronary-computed tomographic morphological plaque characteristics resulted in an AUC of 0.90 for CT-FFRML, 0.80 for LL/MLD4, 0.77 for MLD, 0.75 for the degree of luminal diameter stenosis, 0.77 for MLA, 0.76 for RI, 0.69 for cCTA (<50%), and likewise, 0.69 for cCTA (<70%). The AUCs on a per-patient level are outlined in Table 3.

4. Discussion

In this study, the aim is to compare CT-FFRML and coronary computed tomographic morphological plaque characteristics calculated with the RFRTM as a novel invasive resting pressure-derived index for detecting hemodynamically significant coronary artery stenosis.
Based on the comparatively moderate specificity of cCTA, a tendency may exist to overestimate the severity of coronary artery stenosis by cCTA examination [26,27,28]. In order to compensate for this disadvantage of cCTA, which is caused by its low specificity for the detection of hemodynamically relevant stenosis, techniques have been developed in recent years to improve the quantification of suspicious coronary artery stenosis.
The coronary computed tomographic morphological plaque characteristics calculated based on cCTA were able to prove in various studies and the information gained from plaque parameters led to an improvement in the diagnostic accuracy of the cCTA for the detection of hemodynamically relevant stenosis [2,3,22]. Our results demonstrate that cCTA morphological plaque characteristics, including LL/MLD4, MLD, the degree of luminal diameter stenosis, MLA, and RI, have discriminatory power to differentiate between hemodynamically relevant and non-relevant coronary artery stenoses. In line with previous studies, LL/MLD4, MLD, the degree of luminal diameter stenosis, and MLA were also identified as having significant discriminatory power in our study [3,22]. The present analysis supports findings from Tesche et al. [3], Wang et al. [22], and Baumann et al. [29], that LL, TPV, and VV do not show statistically relevant differences between flow obstructing and non-flow obstructing coronary artery stenoses. However, our findings concerning the discriminatory power of RI and CCO are discordant with the results of the studies mentioned above regarding cCTA plaque characteristics. We could demonstrate that the RI belongs to the morphological plaque characteristics with significant discriminatory power. Previous studies have described the opposite results when examining the discriminatory power of RI [3,22]. Even when using CCO, our results differed from the studies mentioned. We could further demonstrate that CCO did not show statistically significant differences between relevant and non-relevant coronary artery lesions. In contrast to these studies mentioned above, with the exception of one study, which used the invasive FFR as the reference standard, we utilized the RFRTM as a novel invasive resting index and to detect hemodynamically significant coronary artery stenosis. We could demonstrate that LL/MLD4, an anatomic plaque marker, had the best diagnostic power to determine the hemodynamic relevance of coronary lesions on a per-lesion level (AUC: 0.80; 95% CI: 0.64–0.96%). Our results are in accordance with the findings of a previous study, including Baumann et al. [29] (AUC: 0.84) and Wang et al. [22] (AUC: 0.90) that also demonstrated the utility of LL/MLD4. We are also able to support the findings in terms of MLD (AUC: 0.81 and AUC: 0.80). In addition to our previously published results [29], it can be assumed that some plaque markers derived from regular anatomical cCTA may assist in the detection of hemodynamically relevant coronary stenosis. Specifically, LL/MLD4, a morphological plaque marker, with its high specificity and accuracy, may resemble a strong predictor.
In addition to morphological plaque characteristics calculated by cCTA, suspicious coronary lesions may be further assessed by functional cCTA predictors. As an example, CT-FFR is gaining importance for non-invasive assessment of suspicious coronary artery stenosis. Previous trials, including DISCOVER-FLOW, DeFACTO, and NXT, have demonstrated a good correlation between CT-FFR and invasive FFR [24,30,31]. These results led to the approval of the CT-FFR off-site algorithm by the US Food and Drug Administration (FDA) in 2015.
Because of the time-consuming calculation time by the off-site CT-FFR algorithm, an on-site CT-FFRML algorithm was developed [7]. This on-site CT-FFRML algorithm shortened calculation time and has been successfully validated since [10]. In our study, we utilized this on-site CT-FFRML algorithm, which is currently not commercially available. In addition, both off-site and on-site CT-FFR algorithms were compared to invasive gold standard FFR in the above-mentioned studies. We could demonstrate similar results in terms of diagnostic accuracy (accuracy on a per-lesion level: 93%, 95% CI: 81–98%) and a high diagnostic performance for the CT-FFRML (AUC: 0.90, 95% CI: 0.75–1.00%) using the RFRTM as a novel invasive resting pressure-derived index for detecting hemodynamically significant coronary artery stenosis.
Instead of using the invasive gold standard FFR, the novel resting pressure-derived index RFRTM was used as the reference standard to classify suspicious coronary artery stenosis. Initial studies have demonstrated that both CT-FFRML and morphological plaque characteristics compare well to iwFR as the reference standard for detecting hemodynamically significant coronary artery stenosis regarding their diagnostic accuracy [11,29,32]. Thus, we used the rest index RFRTM as the reference standard, as RFRTM demonstrated an excellent correlation to iwFR [16]. Therefore, this is the first study that compares CT-FFRML and morphological plaque characteristics based on cCTA with the RFRTM as a novel invasive resting pressure-derived index for detecting hemodynamically significant coronary artery stenosis.
Our findings are to be evaluated and considered in light of the following limitations. First, it should be noted that our monocentric study cohort of a total number of 33 enrolled patients with 44 lesions is relatively small. Second, we did not perform any comparisons to conventional invasive FFR as the invasive gold standard.
Third, all patients had suspected but chronic coronary syndrome with an indication for cCTA, which is just a subgroup of patients who could possibly have an advantage of improved diagnostic tools. We treated patients according to existing guidelines, stating that invasive pressure wire measuring is only indicated in patients lacking evidence of ischemia. Finally, there are no follow up data regarding any adverse events or survival rates stratified by diagnostic parameter or treatment.

5. Conclusions

In contrast to prior studies using invasive FFR or iwFR as the reference standard, this is the first investigation to our knowledge to compare CT-FFRML and morphological plaque characteristics against the invasive reference standard RFRTM as a novel resting index without the use of intra-arterial adenosine administration. On-site CT-FFRML and computed tomography morphological plaque characteristics (LL/MLD4, RI, MLD, MLA, degree of luminal diameter stenosis) showed a significant correlation to detected hemodynamically significant coronary artery stenosis. Whole CT-FFRML had the best discriminatory power using RFRTM as the reference standard. CCO, LL, TPV, and VV failed to detect hemodynamically relevant coronary artery stenosis.

Author Contributions

Conceptualization, S.B. and D.L.; Data curation, S.B., M.H., C.R., G.H.Ö. and D.L.; Formal analysis, C.W. and S.H.; Funding acquisition, S.B. and D.L.; Investigation, S.B., M.H. and D.L.; Methodology, S.B. and D.L.; Project administration, S.B. and D.L.; Resources, S.B. and D.L.; Software, S.B., M.H., G.H.Ö., D.O. and D.L.; Supervision, S.B. and D.L.; Validation, S.B. and D.L.; Visualization, S.B. and M.H.; Writing – original draft, S.B., M.H., C.R. and D.L.; Writing – review & editing, G.H.Ö., C.T., T.B., C.W., S.H., I.A., S.O.S., M.B., S.J. and D.O. All Authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Acknowledgments

Supported by Siemens Healthineers for providing CT-FFRML software and morphological plaque software for research purposes, which is currently not commercially available. We acknowledge financial support by the Baden-Württemberg Ministry of Science, Research and the Arts and by Ruprecht-Karls-Universität Heidelberg.

Conflicts of Interest

S.B. receives research support from Philips Volcano. C.T. has received research support and speaker honoraria from Siemens. D.L. receives consultant honoraria from Actelion. All other authors declare that they have no financial disclosures. The presented CT-FFRML software and morphological plaque software is provided by Siemens and is currently not commercially available.

References

  1. Chang, H.J.; Lin, F.Y.; Lee, S.E.; Andreini, D.; Bax, J.; Cademartiri, F.; Chinnaiyan, K.; Chow, B.J.W.; Conte, E.; Cury, R.C.; et al. Coronary Atherosclerotic Precursors of Acute Coronary Syndromes. J. Am. Coll. Cardiol. 2018, 71, 2511–2522. [Google Scholar] [CrossRef] [PubMed]
  2. Wang, R.; Renker, M.; Schoepf, U.J.; Wichmann, J.L.; Fuller, S.R.; Rier, J.D.; Bayer, R.R., 2nd; Steinberg, D.H.; De Cecco, C.N.; Baumann, S. Diagnostic value of quantitative stenosis predictors with coronary CT angiography compared to invasive fractional flow reserve. Eur. J. Radiol. 2015, 84, 1509–1515. [Google Scholar] [CrossRef] [PubMed]
  3. Tesche, C.; De Cecco, C.N.; Caruso, D.; Baumann, S.; Renker, M.; Mangold, S.; Dyer, K.T.; Varga-Szemes, A.; Baquet, M.; Jochheim, D.; et al. Coronary CT angiography derived morphological and functional quantitative plaque markers correlated with invasive fractional flow reserve for detecting hemodynamically significant stenosis. J. Cardiovasc. Comput. Tomogr. 2016, 10, 199–206. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  4. Renker, M.; Schoepf, U.J.; Becher, T.; Krampulz, N.; Kim, W.; Rolf, A.; Mollmann, H.; Hamm, C.W.; Henzler, T.; Borggrefe, M.; et al. Computed tomography in patients with chronic stable angina: Fractional flow reserve measurement. Herz 2017, 42, 51–57. [Google Scholar] [CrossRef]
  5. Baumann, S.; Renker, M.; Schoepf, U.J. Coronary Computed Tomography-Based Fractional Flow Reserve: A Rapidly Developing Field. Jama Cardiol. 2018, 3, 87. [Google Scholar] [CrossRef]
  6. Baumann, S.; Wang, R.; Schoepf, U.J.; Steinberg, D.H.; Spearman, J.V.; Bayer, R.R., 2nd; Hamm, C.W.; Renker, M. Coronary CT angiography-derived fractional flow reserve correlated with invasive fractional flow reserve measurements--initial experience with a novel physician-driven algorithm. Eur. Radiol. 2015, 25, 1201–1207. [Google Scholar] [CrossRef]
  7. Coenen, A.; Kim, Y.H.; Kruk, M.; Tesche, C.; De Geer, J.; Kurata, A.; Lubbers, M.L.; Daemen, J.; Itu, L.; Rapaka, S.; et al. Diagnostic Accuracy of a Machine-Learning Approach to Coronary Computed Tomographic Angiography-Based Fractional Flow Reserve: Result From the MACHINE Consortium. Circ. Cardiovasc. Imaging 2018, 11, e007217. [Google Scholar] [CrossRef] [Green Version]
  8. Tesche, C.; De Cecco, C.N.; Baumann, S.; Renker, M.; McLaurin, T.W.; Duguay, T.M.; Bayer, R.R., 2nd; Steinberg, D.H.; Grant, K.L.; Canstein, C.; et al. Coronary CT Angiography-derived Fractional Flow Reserve: Machine Learning Algorithm versus Computational Fluid Dynamics Modeling. Radiology 2018, 288, 64–72. [Google Scholar] [CrossRef] [Green Version]
  9. Yu, M.; Lu, Z.; Li, W.; Wei, M.; Yan, J.; Zhang, J. CT morphological index provides incremental value to machine learning based CT-FFR for predicting hemodynamically significant coronary stenosis. Int. J. Cardiol. 2018, 265, 256–261. [Google Scholar] [CrossRef]
  10. Renker, M.; Schoepf, U.J.; Wang, R.; Meinel, F.G.; Rier, J.D.; Bayer, R.R., 2nd; Mollmann, H.; Hamm, C.W.; Steinberg, D.H.; Baumann, S. Comparison of diagnostic value of a novel noninvasive coronary computed tomography angiography method versus standard coronary angiography for assessing fractional flow reserve. Am. J. Cardiol. 2014, 114, 1303–1308. [Google Scholar] [CrossRef]
  11. Baumann, S.; Hirt, M.; Schoepf, U.J.; Rutsch, M.; Tesche, C.; Renker, M.; Golden, J.W.; Buss, S.J.; Becher, T.; Bojara, W.; et al. Correlation of machine learning computed tomography-based fractional flow reserve with instantaneous wave free ratio to detect hemodynamically significant coronary stenosis. Clin. Res. Cardiol. Off. J. Ger. Card. Soc. 2019. [Google Scholar] [CrossRef] [PubMed]
  12. Gotberg, M.; Christiansen, E.H.; Gudmundsdottir, I.J.; Sandhall, L.; Danielewicz, M.; Jakobsen, L.; Olsson, S.E.; Ohagen, P.; Olsson, H.; Omerovic, E.; et al. Instantaneous Wave-free Ratio versus Fractional Flow Reserve to Guide PCI. New Engl. J. Med. 2017, 376, 1813–1823. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  13. Xaplanteris, P.; Fournier, S.; Pijls, N.H.J.; Fearon, W.F.; Barbato, E.; Tonino, P.A.L.; Engstrom, T.; Kaab, S.; Dambrink, J.H.; Rioufol, G.; et al. Five-Year Outcomes with PCI Guided by Fractional Flow Reserve. New Engl. J. Med. 2018, 379, 250–259. [Google Scholar] [CrossRef] [PubMed]
  14. Neumann, F.J.; Sousa-Uva, M.; Ahlsson, A.; Alfonso, F.; Banning, A.P.; Benedetto, U.; Byrne, R.A.; Collet, J.P.; Falk, V.; Head, S.J.; et al. 2018 ESC/EACTS Guidelines on myocardial revascularization. Eur. Heart J. 2018. [Google Scholar] [CrossRef] [Green Version]
  15. Davies, J.E.; Sen, S.; Dehbi, H.M.; Al-Lamee, R.; Petraco, R.; Nijjer, S.S.; Bhindi, R.; Lehman, S.J.; Walters, D.; Sapontis, J.; et al. Use of the Instantaneous Wave-free Ratio or Fractional Flow Reserve in PCI. New Engl. J. Med. 2017, 376, 1824–1834. [Google Scholar] [CrossRef] [Green Version]
  16. Svanerud, J.; Ahn, J.M.; Jeremias, A.; van ’t Veer, M.; Gore, A.; Maehara, A.; Crowley, A.; Pijls, N.H.J.; De Bruyne, B.; Johnson, N.P.; et al. Validation of a novel non-hyperaemic index of coronary artery stenosis severity: The Resting Full-cycle Ratio (VALIDATE RFR) study. EuroIntervention 2018, 14, 806–814. [Google Scholar] [CrossRef]
  17. Lee, J.M.; Choi, K.H.; Park, J.; Hwang, D.; Rhee, T.M.; Kim, J.; Park, J.; Kim, H.Y.; Jung, H.W.; Cho, Y.K.; et al. Physiological and Clinical Assessment of Resting Physiological Indexes. Circulation 2019, 139, 889–900. [Google Scholar] [CrossRef]
  18. Bittencourt, M.S.; Hulten, E.; Polonsky, T.S.; Hoffman, U.; Nasir, K.; Abbara, S.; Di Carli, M.; Blankstein, R. European Society of Cardiology-Recommended Coronary Artery Disease Consortium Pretest Probability Scores More Accurately Predict Obstructive Coronary Disease and Cardiovascular Events Than the Diamond and Forrester Score: The Partners Registry. Circulation 2016, 134, 201–211. [Google Scholar] [CrossRef] [Green Version]
  19. Leipsic, J.; Abbara, S.; Achenbach, S.; Cury, R.; Earls, J.P.; Mancini, G.J.; Nieman, K.; Pontone, G.; Raff, G.L. SCCT guidelines for the interpretation and reporting of coronary CT angiography: A report of the Society of Cardiovascular Computed Tomography Guidelines Committee. J. Cardiovasc. Comput. Tomogr. 2014, 8, 342–358. [Google Scholar] [CrossRef]
  20. van Rosendael, A.R.; Shaw, L.J.; Xie, J.X.; Dimitriu-Leen, A.C.; Smit, J.M.; Scholte, A.J.; van Werkhoven, J.M.; Callister, T.Q.; DeLago, A.; Berman, D.S.; et al. Superior Risk Stratification With Coronary Computed Tomography Angiography Using a Comprehensive Atherosclerotic Risk Score. Jacc Cardiovasc. Imaging 2019. [Google Scholar] [CrossRef] [Green Version]
  21. Agatston, A.S.; Janowitz, W.R.; Hildner, F.J.; Zusmer, N.R.; Viamonte, M., Jr.; Detrano, R. Quantification of coronary artery calcium using ultrafast computed tomography. J. Am. Coll Cardiol. 1990, 15, 827–832. [Google Scholar] [CrossRef] [Green Version]
  22. Wang, R.; Baumann, S.; Schoepf, U.J.; Meinel, F.G.; Rier, J.D.; Morris, J.Z.; Mollmann, H.; Hamm, C.W.; Steinberg, D.H.; Renker, M. Comparison of quantitative stenosis characteristics at routine coronary computed tomography angiography with invasive fractional flow reserve for assessing lesion-specific ischemia. J. Cardiovasc. Comput. Tomogr. 2015, 9, 546–552. [Google Scholar] [CrossRef] [PubMed]
  23. Achenbach, S.; Ropers, D.; Hoffmann, U.; MacNeill, B.; Baum, U.; Pohle, K.; Brady, T.J.; Pomerantsev, E.; Ludwig, J.; Flachskampf, F.A.; et al. Assessment of coronary remodeling in stenotic and nonstenotic coronary atherosclerotic lesions by multidetector spiral computed tomography. J. Am. Coll. Cardiol. 2004, 43, 842–847. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  24. Norgaard, B.L.; Leipsic, J.; Gaur, S.; Seneviratne, S.; Ko, B.S.; Ito, H.; Jensen, J.M.; Mauri, L.; De Bruyne, B.; Bezerra, H.; et al. Diagnostic performance of noninvasive fractional flow reserve derived from coronary computed tomography angiography in suspected coronary artery disease: The NXT trial (Analysis of Coronary Blood Flow Using CT Angiography: Next Steps). J. Am. Coll. Cardiol. 2014, 63, 1145–1155. [Google Scholar] [CrossRef] [Green Version]
  25. Escaned, J.; Echavarria-Pinto, M.; Garcia-Garcia, H.M.; van de Hoef, T.P.; de Vries, T.; Kaul, P.; Raveendran, G.; Altman, J.D.; Kurz, H.I.; Brechtken, J.; et al. Prospective Assessment of the Diagnostic Accuracy of Instantaneous Wave-Free Ratio to Assess Coronary Stenosis Relevance: Results of ADVISE II International, Multicenter Study (ADenosine Vasodilator Independent Stenosis Evaluation II). Jacc Cardiovasc. Interv. 2015, 8, 824–833. [Google Scholar] [CrossRef] [Green Version]
  26. Meijboom, W.B.; Meijs, M.F.; Schuijf, J.D.; Cramer, M.J.; Mollet, N.R.; van Mieghem, C.A.; Nieman, K.; van Werkhoven, J.M.; Pundziute, G.; Weustink, A.C.; et al. Diagnostic accuracy of 64-slice computed tomography coronary angiography: A prospective, multicenter, multivendor study. J. Am. Coll. Cardiol. 2008, 52, 2135–2144. [Google Scholar] [CrossRef] [Green Version]
  27. Patel, M.R.; Peterson, E.D.; Dai, D.; Brennan, J.M.; Redberg, R.F.; Anderson, H.V.; Brindis, R.G.; Douglas, P.S. Low diagnostic yield of elective coronary angiography. New Engl. J. Med. 2010, 362, 886–895. [Google Scholar] [CrossRef] [Green Version]
  28. Patel, M.R.; Dai, D.; Hernandez, A.F.; Douglas, P.S.; Messenger, J.; Garratt, K.N.; Maddox, T.M.; Peterson, E.D.; Roe, M.T. Prevalence and predictors of nonobstructive coronary artery disease identified with coronary angiography in contemporary clinical practice. Am. Heart J. 2014, 167, 846–852.e842. [Google Scholar] [CrossRef]
  29. Baumann, S.; Ozdemir, G.H.; Tesche, C.; Schoepf, U.J.; Golden, J.W.; Becher, T.; Hirt, M.; Weiss, C.; Renker, M.; Akin, I.; et al. Coronary CT angiography derived plaque markers correlated with invasive instantaneous flow reserve for detecting hemodynamically significant coronary stenoses. Eur. J. Radiol. 2019, 122, 108744. [Google Scholar] [CrossRef] [Green Version]
  30. Min, J.K.; Leipsic, J.; Pencina, M.J.; Berman, D.S.; Koo, B.K.; van Mieghem, C.; Erglis, A.; Lin, F.Y.; Dunning, A.M.; Apruzzese, P.; et al. Diagnostic accuracy of fractional flow reserve from anatomic CT angiography. Jama 2012, 308, 1237–1245. [Google Scholar] [CrossRef]
  31. Koo, B.K.; Erglis, A.; Doh, J.H.; Daniels, D.V.; Jegere, S.; Kim, H.S.; Dunning, A.; DeFrance, T.; Lansky, A.; Leipsic, J.; et al. Diagnosis of ischemia-causing coronary stenoses by noninvasive fractional flow reserve computed from coronary computed tomographic angiograms. Results from the prospective multicenter DISCOVER-FLOW (Diagnosis of Ischemia-Causing Stenoses Obtained Via Noninvasive Fractional Flow Reserve) study. J. Am. Coll. Cardiol. 1997, 58, 1989–1997. [Google Scholar] [CrossRef] [Green Version]
  32. Driessen, R.S.; de Waard, G.A.; Stuijfzand, W.J.; Raijmakers, P.G.; Danad, I.; Bom, M.J.; Min, J.K.; Leipsic, J.A.; Ahmadi, A.; van de Ven, P.M.; et al. Adverse Plaque Characteristics Relate More Strongly With Hyperemic Fractional Flow Reserve and Instantaneous Wave-Free Ratio Than With Resting Instantaneous Wave-Free Ratio. Jacc Cardiovasc. Imaging 2019. [Google Scholar] [CrossRef] [PubMed]
Jcm 09 00714 g001 550
Figure 1. (A) The acquired cCTA shows non-relevant stenosis of the LAD with color-coded automated lesion quantification by the plaque tool. The mixed plaque in the LAD demonstrated a remodeling index of 0.91 and was calculated as (C) the target-lesion cross-sectional area (marked in orange). (B) divided by proximal reference-cross-sectional area (marked blue). (D) A 3-dimensional color-coded reconstruction calculated CT-FFRML value of 0.94 (CT-FFRML cut-off value ≤ 0.80) is presented (arrow). (E) This stenosis (arrow) was visualized by ICA and was invasively measured using RFRTM (0.94; RFRTM cut-off value ≤ 0.89), indicating no hemodynamic relevance. cCTA = coronary computed tomography angiography; CT-FFRML = fractional flow reserve derived from coronary computed tomography angiography based on machine learning algorithm; ICA = invasive coronary angiography; LAD = left anterior descending artery; RFRTM = resting full-cycle ratio.
Figure 1. (A) The acquired cCTA shows non-relevant stenosis of the LAD with color-coded automated lesion quantification by the plaque tool. The mixed plaque in the LAD demonstrated a remodeling index of 0.91 and was calculated as (C) the target-lesion cross-sectional area (marked in orange). (B) divided by proximal reference-cross-sectional area (marked blue). (D) A 3-dimensional color-coded reconstruction calculated CT-FFRML value of 0.94 (CT-FFRML cut-off value ≤ 0.80) is presented (arrow). (E) This stenosis (arrow) was visualized by ICA and was invasively measured using RFRTM (0.94; RFRTM cut-off value ≤ 0.89), indicating no hemodynamic relevance. cCTA = coronary computed tomography angiography; CT-FFRML = fractional flow reserve derived from coronary computed tomography angiography based on machine learning algorithm; ICA = invasive coronary angiography; LAD = left anterior descending artery; RFRTM = resting full-cycle ratio.
Jcm 09 00714 g001
Table
Table 1. Baseline characteristics (patients n = 33 and lesions π n = 44) and findings of cCTA, CT-FFRML, and ICA.
Table 1. Baseline characteristics (patients n = 33 and lesions π n = 44) and findings of cCTA, CT-FFRML, and ICA.
ParameterMean Value ± Standard Deviation, Frequency (%) or Interquartile Range
Age (years)68 ± 12
Men23 (70%)
Body mass index (kg/m2) 29 ± 5
Pretest probability (%) +57 ± 19
Cardiovascular Risk Factors
Hypertension *26 (78%)
Hyperlipidemia †18 (54%)
Diabetes mellitus 8 (24%)
Family history of coronary artery disease8 (24%)
Current smoker4 (12%)
Coronary Computed Tomography
Agatston score800 (35–3608)
Luminal stenosis > 70%14 (32%)
Leiden cCTA risk score12 ± 5
CT-FFRML ≤ 0.80 π13 (30%)
Invasive Coronary Catheter Angiography
Left anterior descending coronary artery π26 (59%)
Left circumflex coronary artery π10 (23%)
Right coronary artery π8 (18%)
RFRTM ≤ 0.89 π14 (32%)
Unless otherwise specified, data are numbers of patients with percentages in parentheses. Data are mean ± standard deviation (SD) or interquartile range. + Pretest probability calculated with the CAD consortium clinical score [18]. * Defined as blood pressure > 140 mmHg systolic, >90 mmHg diastolic, or use of an antihypertensive medication. † Defined as a total cholesterol level of >200 mg/dL or use of antilipidemic medication; cCTA = coronary computed tomography angiography; CT-FFRML = fractional flow reserve derived from coronary computed tomography angiography based on machine learning algorithm; RFRTM = resting full-cycle ratio.
Table
Table 2. Comparison of CT-FFRML and plaque characteristics in stenosis with and without hemodynamic relevance stenosis using RFRTM as reference.
Table 2. Comparison of CT-FFRML and plaque characteristics in stenosis with and without hemodynamic relevance stenosis using RFRTM as reference.
ParameterAll LesionsLesions RFRTM > 0.89LesionsRFRTM ≤ 0.89p-Value
Number of Lesions443014-
Morphological computed tomographic morphological plaque characteristics
LL/MLD411.0 ± 6.28.8 ± 4.015.7 ± 7.60.0016 *
MLD (mm)1.8 ± 0.82.0 ± 0.81.4 ± 0.70.0038 *
Degree of luminal diameter stenosis (%)50.3 ± 28.642.9 ± 26.366.4 ± 27.40.0063 *
MLA (mm2)4.8 ± 3.55.5 ± 3.43.5 ± 3.30.0078 *
cCTA stenosis > 50%2614120.0141 *
cCTA stenosis > 70%14680.0341 *
TPV (mm3)116.1 ± 76.2101.4 ± 66.3147.5 ± 88.50.0989
VV (mm3)200.7 ± 117.0189.0 ± 115.7225.7 ± 120.00.3710
LL (mm)17.0 ± 7.416.5 ± 8.118.2 ± 5.70.4421
Functional computed tomographic morphological plaque characteristics
CT-FFRML0.87 ± 0.140.94 ± 0.050.72 ± 0.15<0.0001 **
RI1.20 ± 0.281.12 ± 0.231.38 ± 0.290.0062 *
CCO0.13 ± 0.120.14 ± 0.130.13 ± 0.080.6770
cCTA = coronary computed tomography angiography; CCO = corrected coronary opacification; CT-FFRML = fractional flow reserve derived from coronary computed tomography angiography based on machine learning algorithm; LL = lesion length; MLA = minimal luminal area; MLD = minimal luminal diameter; RFRTM = resting full-cycle ratio; RI = remodeling index; TPV = total plaque volume; VV = vessel volume; ** highly significant (p ≤ 0.001), * significant (p ≤ 0.05).
Table
Table 3. Diagnostic performance of fractional flow reserve derived from coronary-computed tomography angiography based on machine learning algorithm and coronary-computed tomographic plaque characteristic on a per-lesion and per-patient level using RFRTM as the reference standard.
Table 3. Diagnostic performance of fractional flow reserve derived from coronary-computed tomography angiography based on machine learning algorithm and coronary-computed tomographic plaque characteristic on a per-lesion and per-patient level using RFRTM as the reference standard.
Per Lesion (n = 44)
Sensitivity (%) (95% CI)Specificity (%) (95% CI)PPV (%) (95% CI)NPV (%) (95% CI)Accuracy (%) (95% CI)AUC (95% CI)
Morphological Computed Tomographic Morphological Plaque Characteristics
LL/MLD471 (42–92)77 (58–90)59 (33–82)85 (66–96)75 (60–87)0.80 (0.64–0.96)
MLD (mm2)64 (35–87)80 (61–92)60 (32–84)83 (64–94)75 (59–87)0.77 (0.62–0.93)
Degree of luminal diameter stenosis (%)71 (42–92)77 (58–90)59 (33–82)85 (66–96)75 (59–87)0.75 (0.58–0.94)
MLA (mm2)71 (42–92)80 (61–92)62 (35–85)86 (67–96)77 (62–88)0.75 (0.57–0.94)
cCTA (<50%)86 (57–98)53 (34–72)46 (27–67)89 (65–98)63 (48–78)0.69 (0.56–0.83)
cCTA (<70%)57 (29–82)80 (61–92)57 (29–82)80 (61–92)73 (57–85)0.69 (0.53–0.84)
Functional Computed Tomographic Morphological Plaque Characteristics
CT-FFRML86 (57–98)97 (88–99)92 (74–99)94 (79–99)93 (81–98)0.90 (0.75–1.00)
RI71 (42–92)67 (47–83)50 (27–73)83 (63–95)68 (52–81)0.76 (0.61–0.91)
Per Patient (n = 33)
Morphological Computed Tomographic Morphological Plaque Characteristics
LL/MLD475 (73–94)71 (48–84)60 (32–84)83 (59–96)73 (55–87)0.78 (0.57–0.99)
MLA (mm2)67 (35–90)71 (48–89)57 (29–82)79 (54–94)70 (51–84)0.70 (0.48–0.93)
MLD (mm2)58 (28–85)71 (48–89)54 (25–81)75 (51–91)67 (48–82)0.70 (0.50–0.90)
Degree of luminal diameter stenosis (%)67 (35–93)67 (43–85)53 (27–79)78 (52–94)67 (48–82)0.68 (0.46–0.89)
cCTA (<50%)83 (52–98)62 (39–82)44 (23–65)80 (44–98)55 (36–72)0.60 (0.45–0.76)
cCTA (<70%)50 (21–79)71 (48–89)50 (21–79)71 (48–88)64 (45–80)0.60 (0.42–0.78)
Functional Computed Tomographic Morphological Plaque Characteristics
CT-FFRML83 (52–98)95 (76–99)91 (59–99)91 (71–99)91 (76–98)0.87 (0.70–1.00)
RI67 (35–90)52 (30–74)44 (21–69)73 (45–92)58 (39–74)0.70 (0.51–0.89)
AUC = area under the receiver operating characteristic curve; cCTA = coronary computed tomography angiography; CI= confidence interval; CT-FFRML = fractional flow reserve derived from coronary computed tomography angiography based on machine learning algorithm; MLA = minimal luminal area; MLD = minimal luminal diameter; NPV = negative predictive value; PPV = positive predictive value; RFRTM = resting full-cycle ratio; RI = remodeling index.

Share and Cite

MDPI and ACS Style

Baumann, S.; Hirt, M.; Rott, C.; Özdemir, G.H.; Tesche, C.; Becher, T.; Weiss, C.; Hetjens, S.; Akin, I.; Schoenberg, S.O.; et al. Comparison of Machine Learning Computed Tomography-Based Fractional Flow Reserve and Coronary CT Angiography-Derived Plaque Characteristics with Invasive Resting Full-Cycle Ratio. J. Clin. Med. 2020, 9, 714. https://doi.org/10.3390/jcm9030714

AMA Style

Baumann S, Hirt M, Rott C, Özdemir GH, Tesche C, Becher T, Weiss C, Hetjens S, Akin I, Schoenberg SO, et al. Comparison of Machine Learning Computed Tomography-Based Fractional Flow Reserve and Coronary CT Angiography-Derived Plaque Characteristics with Invasive Resting Full-Cycle Ratio. Journal of Clinical Medicine. 2020; 9(3):714. https://doi.org/10.3390/jcm9030714

Chicago/Turabian Style

Baumann, Stefan, Markus Hirt, Christina Rott, Gökce H. Özdemir, Christian Tesche, Tobias Becher, Christel Weiss, Svetlana Hetjens, Ibrahim Akin, Stefan O. Schoenberg, and et al. 2020. "Comparison of Machine Learning Computed Tomography-Based Fractional Flow Reserve and Coronary CT Angiography-Derived Plaque Characteristics with Invasive Resting Full-Cycle Ratio" Journal of Clinical Medicine 9, no. 3: 714. https://doi.org/10.3390/jcm9030714

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop