Next Article in Journal
Long-Term Swallowing Outcome and Dysphagia in Advanced Staged Head and Neck Squamous Cell Carcinomas after Radiotherapy
Next Article in Special Issue
The Association between Elevated Hematocrit and Retinal Artery Occlusion in Adult Patients
Previous Article in Journal
Distribution of Cerebrovascular Phenotypes According to Variants of the ENG and ACVRL1 Genes in Subjects with Hereditary Hemorrhagic Telangiectasia
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Risk Assessment of CHD Using Retinal Images with Machine Learning Approaches for People with Cardiometabolic Disorders

1
Division of Biostatistics, The Jockey Club School of Public Health and Primary Care, Faculty of Medicine, The Chinese University of Hong Kong, Hong Kong SAR, China
2
Department of Acupuncture and Moxibustion, Shenzhen Traditional Chinese Medicine Hospital, Shenzhen 518005, China
3
Department of Cardiovascular Disease, Shenzhen Traditional Chinese Medicine Hospital, Shenzhen 518005, China
4
Diabetes Research Group, Swansea University, Swansea SA2 8PP, UK
5
Clinical Trials and Biostatistics Lab, CUHK Shenzhen Research Institute, Shenzhen 518057, China
*
Author to whom correspondence should be addressed.
These authors contributed equally to this work.
Current address: School of Population Medicine and Public Health, Chinese Academy of Medical Sciences & Peking Union Medical College, Beijing 100730, China.
J. Clin. Med. 2022, 11(10), 2687; https://doi.org/10.3390/jcm11102687
Submission received: 15 April 2022 / Revised: 5 May 2022 / Accepted: 6 May 2022 / Published: 10 May 2022
(This article belongs to the Special Issue Retinal Microvasculature in Health and Disease)

Abstract

:
Background: Coronary heart disease (CHD) is the leading cause of death worldwide, constituting a growing health and social burden. People with cardiometabolic disorders are more likely to develop CHD. Retinal image analysis is a novel and noninvasive method to assess microvascular function. We aim to investigate whether retinal images can be used for CHD risk estimation for people with cardiometabolic disorders. Methods: We have conducted a case–control study at Shenzhen Traditional Chinese Medicine Hospital, where 188 CHD patients and 128 controls with cardiometabolic disorders were recruited. Retinal images were captured within two weeks of admission. The retinal characteristics were estimated by the automatic retinal imaging analysis (ARIA) algorithm. Risk estimation models were established for CHD patients using machine learning approaches. We divided CHD patients into a diabetes group and a non-diabetes group for sensitivity analysis. A ten-fold cross-validation method was used to validate the results. Results: The sensitivity and specificity were 81.3% and 88.3%, respectively, with an accuracy of 85.4% for CHD risk estimation. The risk estimation model for CHD with diabetes performed better than the model for CHD without diabetes. Conclusions: The ARIA algorithm can be used as a risk assessment tool for CHD for people with cardiometabolic disorders.

1. Introduction

Globally, coronary heart disease (CHD) is the leading cause of death and affected 110 million people and accounted for 8.9 million deaths in 2015 [1]. Coronary heart disease caused approximately one third of all deaths in persons older than 35 years of age [2]. The prevalence and mortality of CHD vary according to risk factors among countries and regions and change markedly over time. Although CHD prevalence has been decreasing in many developed countries, the overall disease burden is still increasing, especially in low- and middle-income countries, due to rapid socioeconomic development and aging [3,4,5,6,7,8,9]. Approximately three fourths of the global mortality and 82% of the global disability-adjusted life years (DALYs) of CHD occurred in low- and middle-income countries [10]. China has undergone a rapid health transition in recent decades, with rapid economic growth and social change [11]. A crude prevalence of 4.42% was reported in rural China in 2019 [12]. The global burden of disease 2015 study showed that the age-standardized CHD mortality rate increased by 13.3% from 1990 to 2015 in China [13]. The mortality rate was 113.46 per 100,000 for Chinese urban residents and 118.74 per 100,000 for rural residents according to the “China Health and Family Planning Statistical Yearbook 2017” [14]. A study predicted a 69% increase in CHD events and a 64% increase in CHD-related deaths during 2020–2029 compared to 2000–2009 [15]. Among the CHD patients who died within one month after the onset of symptoms, approximately two thirds died before they reached a hospital after suffering a CHD event [16].
Cardiometabolic disorders, including hypertension, type 2 diabetes, overweightness or obesity, and dyslipidemia, are documented preclinical factors associated with cardiovascular diseases [17,18,19]. People suffering from cardiometabolic disorders are at higher risk of developing CHD in the future than the general population [20,21,22,23,24,25,26]. Therefore, risk assessment of CHD among these patients has more clinical significance.
The tools used for the diagnostic assessment of CHD, such as coronary angiography and computed tomography (CT), are reliable and well established. Sophisticated imaging modalities can determine the presence and severity of the diseases with a high degree of sensitivity and specificity. However, these diagnostic tools are technically challenging and expensive to operate, and thus unsuitable as a screening tool in the general population, especially in rural areas. Thus, feasible and cost-efficient tests must be developed to enable the diagnostic identification of individuals with a high risk of CHD.
Many risk equations have been developed in recent years to predict the risk of CHD in the general population, such as those described in the Framingham Study, the Atherosclerosis Risk in Communities (ARIC) Study, the Prospective Cardiovascular Münster (PROCAM) Study, the CUORE cohort, and the Taiwan cohort study [27,28,29,30,31,32,33,34]. Risk equations for diabetes patients were also established, such as the Cardiac Risk Calculator from the UKPDS study [35]. These risk equations included similar clinical characteristics which substantially vary in disease risk. However, a proportion of disease morbidity and mortality cannot be explained by these risk factors [36,37,38,39]. Additionally, these models’ practice value in primary care settings is limited since they require information from questionnaires, invasive blood tests, and physical examinations [28,29,30,31,40]. Moreover, it is more challenging to develop risk estimation models for people with cardiometabolic disorders since risk factors used in traditional risk equations such as blood pressure, blood glucose, and blood lipid levels were also abnormal for these people.
Thus, there is a pressing need for new, convenient, and inexpensive risk factors for CHD assessment and prevention for people with cardiometabolic disorders. We noted that retinal vessels are the only directly visible vessels in the body and microvascular retinopathy features are associated with coronary artery abnormalities [41,42,43]. Retinal vascular changes are a summary marker of a patient’s lifetime exposure to risk factors and can reflect cumulative vascular damage [44]. Furthermore, many studies found retinal changes associated with CHD and CHD mortality [45,46,47,48,49,50,51]. The analysis of retinal characteristics may provide a supplementary way for risk estimation of CHD.

2. Materials and Methods

2.1. Study Subjects

This cross-sectional study was conducted in the Cardiology Department of Shenzhen Traditional Chinese Medicine Hospital from December 2017 to September 2019. The Shenzhen Traditional Chinese Medicine Hospital Ethics Committee (Ref. No.: K2019-005-01) and the Joint Chinese University of Hong Kong—New Territories East Cluster Clinical Research Ethics Committee (Ref. No.: 2020.093) approved the study. Informed consent was obtained from each participant. The inclusion criteria used in the study included good health status and ability to sit on a chair for taking a retinal image, diagnosis with a cardiometabolic disorder, and having a clear disease diagnosis. Subjects with cataracts or other eye diseases that affected retinal imaging required close clinical monitoring or who were unable or unwilling to comply with disease examination were excluded.
Baseline demographic and medical information was collected from participants’ hospital medical records. The nurses measured height and weight and calculated body mass index (BMI; calculated as weight in kilograms divided by height in meters squared) for each participant. Diabetes mellitus (DM) was defined as a fasting blood glucose concentration > 7.0 mmol/L or an HbA1c value > 6.5%; patients with a history of diabetes were also labelled as having DM. Hypertension was defined as a systolic blood pressure ≥ 140 mmHg, diastolic blood pressure ≥ 90 mmHg, or a patient with a history of hypertension. Dyslipidemia was defined as total cholesterol (TC) ≥ 240 mg/L, triglyceride (TG) ≥ 200 mg/dL, high-density lipoprotein cholesterol (HDL-C) < 40 mg/dL, low-density lipoprotein cholesterol (LDL-C) ≥ 160 mg/dL, or a patient with a history of dyslipidemia. Obese status was defined as a BMI of 28 or higher, overweight status as a BMI of 24.0 to 28, and normal weight (including underweight) status as a BMI of less than 24.0. Cardiometabolic disorders, including hypertension, diabetes, dyslipidemia, and overweight or obese status, were measured based on the hospital clinical protocol. Patients with at least one of these conditions were recruited. CHD was diagnosed according to standard recommendations from international guidelines [52,53,54]. Patients were initially suspected of CHD based on their symptoms and electrocardiogram results. Coronary computed tomography or coronary angiography were performed for accurate diagnosis in accordance with clinical practice guidelines. The presence of 50% diameter stenosis of any coronary a with a diameter ≥ 2.0mm has been used as the threshold value for diagnosis.
Three hundred sixteen subjects were included in this study: 188 CHD patients and 128 control with cardiometabolic disorders.

2.2. Retinal Imaging Acquisition and Analysis

Retinal images of both eyes were taken within two weeks from the hospital admission using a non-mydriatic fundus camera (Canon CR2). Each participant obtained one retinal image centered at the fovea of each eye. The spatial resolution of each retinal image was 3648 by 2432 pixels, and the images were stored without compression before analysis.
A fully automatic retinal image analysis for CHD risk (ARIA-CHD) was developed using R (version 3.6.0, R Foundation for Statistical Computing, Vienna, Austria) and Matlab (Version 2020a, The Math Works, Inc., Natick, MA, USA) computer software to estimate retinal microvascular characteristics and incorporate machine learning techniques to estimate an overall risk of CHD. The detailed methods of the automatic retinal imaging analysis method have been reported [55]. The methods include fractal analysis, high order spectra analysis, and statistical texture analysis incorporating a machine learning approach. The characteristics include retinal vessel measurements, arteriole–venous nicking, arteriole occlusion, hemorrhage, exudates, tortuosity, bifurcation coefficients (BC), asymmetry of branches, bifurcation angles, and other machine-learning generated factors.

2.3. Statistical Analysis

For univariate analysis, independent t-test and chi-square test were used to evaluate if the retinal and clinical variables were significantly different between the CHD and control groups. Variables with a p-value of less than 0.05 were considered statistically significant.
For the classification analysis, we randomly selected 70% for the training of the classification model. The other 30% were used for an internal validation process. We used machine learning and deep learning techniques. We extracted the texture/fractal/spectrum-related features (such as high order spectra and fractal dimensions) associated with CHD by using the automatic retinal image analysis (ARIA) algorithm written in Matlab [44]. We then used the glmnet approach to select the most important subset of features based on the penalized maximum likelihood using R and Matlab [55]. Finally, we translated the features extracted from the machine learning approaches to commonly used retinal characteristics measured from the images using ImageJ to gain further insights. We have previously applied this method and validated results in different disease cohorts [56,57,58].
For the validation, we applied a 10-fold cross-validation method using the Support Vector Machine (SVM) algorithm to test the datasets that have not been used in the model’s training [59,60]. This is done by partitioning the dataset and using 90% of the data to train the algorithm and the remaining 10% of data for testing. Because cross-validation does not use all of the data to build a model, it is a commonly used method to prevent overfitting during training.

2.4. Sample Size Estimation

To obtain sensitivity and specificity values of 0.85 or higher with a lower bound of the 95% confidence intervals of at least 0.7, we require more than 50 subjects to estimate sensitivity and specificity for each subgroup [61,62].

3. Results

The univariate analysis showed that many characteristics were significantly different between the CHD patients and the control group (Table 1). The CHD group are older and has a higher rate of female participants than the cardiometabolic disorders group (p < 0.001). In addition, the percentage of patients with hypertension, diabetes, and dyslipidemia was higher in the CHD group than in the control group (p < 0.005).
The univariate analysis also revealed differences in many retinal characteristics between the CHD patients and people with cardiometabolic disorders (Table 2). We found significantly more tortuosity (p = 0.020 for average tortuosity, p = 0.013 for arteriole tortuosity, p < 0.001 for venule tortuosity, respectively), more exudates (p = 0.001 for left eyes), and more arteriole occlusion (p = 0.032 for left eyes) in the CHD group. While the central retinal artery equivalent (CRAE, p = 0.028 for right eyes), central retinal vein equivalent (CRVE, p = 0.002 for left eyes), mean bifurcation coefficient of venules (MBCV, p = 0.014 for left eyes), mean asymmetry index of arterioles (MAasymmetry, p < 0.001 for left eyes), the mean asymmetry index of venules (MVasymmetry, p = 0.008 for left eyes), and mean bifurcation angles of arterioles (MAangle, p = 0.007 for right eyes) were smaller in the CHD group.
After the univariate analysis, we randomly selected 70%—132 CHD and 90 controls—for training the classification model. The remaining 30%—56 CHD and 38 control—were used for internal validation. The results of this classification model are our primary analysis. Thirty-three controls (86.8%) and 52 CHD patients (92.9%) were correctly classified in the internal validation set. The ROC curve was displayed in Figure 1, with an AUC of 0.96. The overall accuracy was 90.4% (Figure 2).
We further divided CHD patients into diabetes and non-diabetes groups for subgroup analysis. The same training and testing procedures were conducted for the two subgroup analyses. The results showed that for classification between CHD without diabetes and control, the model had a sensitivity of 84.6% and a specificity of 89.5%, with an overall accuracy of 87.5%. For classification between CHD with diabetes and control, the model had a sensitivity of 90.0% and a specificity of 94.7%, with an overall accuracy of 92.6% (Figure 3).
To avoid overfitting and test the robustness of the models, we performed a 10-fold cross-validation analysis by using an SVM algorithm for testing datasets that were not used in the training of the model. This was performed by 10-fold partitioning the dataset and repeating the training and testing ten times. Each analysis used a subset of 90% of the data to train the algorithm and the remaining 10% for testing. This process was repeated ten times until all ten folds of the data were tested. We then calculated the overall sensitivity and specificity for the cross-validation analysis based on all ten independent analyses. The sensitivity and specificity results from the cross-validation analysis were 88.3% and 81.3%, respectively, for the overall CHD model; 90.6% and 84.1%, respectively, for CHD with diabetes; 89.1% and 89.0%, respectively, for CHD without diabetes.

4. Discussion

WHO has recommended cost-effective interventions that can be applied in low-resource settings to prevent CHD. The population-level prevention strategies include comprehensive tobacco control; reducing intake of foods high in fat, sugar, and salt; encouraging physical activity; reducing alcohol consumption; and promoting healthy diet habits. Individual-level prevention requires targeting people at high risk. These diseases can be prevented by addressing the above behavioral risk factors to target the high-risk population. Thus, early screening with appropriate tools is urgently needed.
The relationship between retinal images and cardiovascular diseases has been studied previously. Many retinal characteristics are associated with CHD risk factors, disease presence, and prognosis. For instance, retinal vessel atherosclerosis was correlated strongly with the risk factors and severity of CHD, and retinal arteriolar endothelial dysfunction predicts major adverse cardiovascular events in patients with CHD or cardiovascular risk factors [63,64,65]. Automatic retinal vessel analysis may provide added benefit to traditional risk factors in stratifying patients at risk for CHD.
Retinal vessel calibers are the most studied retinal characteristics. The association of CRAE, CRVE, and CHD have been reported previously, though the results were not consistent [66,67,68]. Our result showed that CHD patients had smaller CRAE and CRVE than the control group. Retinal vascular tortuosity quantifies the frequency with which a vessel crosses a low dimensional spline [69]. The tortuosity of vessels constitutes a physiological mechanism that increases metabolic tissue input [70]. Several studies have reported positive associations between retinal vessel tortuosity and cyanotic CHD, intracranial artery atherosclerosis, diabetes, and cerebrovascular events. However, one research study reported a negative association with ischemic heart disease death [68,71,72,73,74]. Our study result demonstrated a positive association between retinal vessel tortuosity and CHD. For retinal fractal dimension characteristics, we found a negative association of MAangle, MAasymmetry, and MVasymmetry with CHD. Similar associations have been found with cerebrovascular events and stroke previously [74]. We have also found larger occlusion and more exudates in the CHD group than the control group in this study, consistent with previous studies [75,76,77].
Our study shows that retinal images can be used as a risk assessment tool for CHD in people with cardiometabolic disorders. The sensitivity and specificity for the classification model were 92.9% and 86.8%, respectively. After dividing CHD patients into diabetes and non-diabetes groups, the model performed even better in the diabetes subgroup. The results of cross-validation were also robust.
Recently, one study using data from South Korea, Singapore, and the UK has shown that a deep learning retinal coronary artery calcium (CAC) score is comparable to CT-scan-measured CAC in predicting cardiovascular events [78]. Another study has established a risk assessment model using fundus photographs for a 10-year risk assessment of ischemic cardiovascular diseases in China, which proved the utility of retinal images as a risk assessment tool in the general population [79]. Compared with these studies, we chose people with cardiometabolic disorders as the control group since they are at higher risk of developing cardiovascular diseases. Though these people were more difficult to classify from CHD patients, the performance of our model was just as good, if not better.
However, there were several limitations to this research. First, although the retinal images were captured prospectively, this is still a retrospective study since CHD patients may have their disease developed when admitted to the hospital. Second, all the participants in this study were patients from the Shenzhen Traditional Chinese Medicine Hospital, and no community controls were recruited. Thus, we cannot eliminate potential case-selection bias in this research. Third, we did not have a separate and large enough data set for model testing for further external validation.
For the future direction of the study, we aim to explore the ability of automatically calculated retinal images to prospectively evaluate the risk of CHD by incorporating retinal images captured in prospective community-based cohorts. Population with cardiometabolic disorders can have regular retinal image acquisition and automated analysis, and subjects with a high burden of developing CHD can undergo further evaluation.

Author Contributions

Conceptualization, Y.Q. and B.C.-Y.Z.; Data curation, Y.Q. and J.J.-W.L.; Formal analysis, J.J.-W.L. and B.C.-Y.Z.; Funding acquisition, Y.Z. and B.C.-Y.Z.; Investigation, Y.Q., Y.Z. and S.L.; Methodology, J.J.-W.L., R.L.T., D.R.O. and B.C.-Y.Z.; Writing—original draft, Y.Q. and B.C.-Y.Z.; Writing—review & editing, Y.Q. and B.C.-Y.Z. All authors have read and agreed to the published version of the manuscript.

Funding

This study was supported by the General Research Fund (GRF) of the Research Grant Council Hong Kong (RGC Ref No. 14139116) and the Shenzhen Science and Technology Innovation Commission (No: KCXFZ20201221173208024).

Institutional Review Board Statement

Shenzhen Traditional Chinese Medicine Hospital Ethics Committee (Ref. No.: K2019-005-01) and the Joint Chinese University of Hong Kong—New Territories East Cluster Clinical Research Ethics Committee (Ref. No.: 2020.093).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Datasets are available upon request to the corresponding author.

Conflicts of Interest

The authors declare that they have no conflict of interest.

References

  1. Wang, H.; Naghavi, M.; Allen, C.; Barber, R.M.; Bhutta, Z.A.; Carter, A.; Casey, D.C.; Charlson, F.J.; Chen, A.Z.; Coates, M.M. Global, regional, and national life expectancy, all-cause mortality, and cause-specific mortality for 249 causes of death, 1980–2015: A systematic analysis for the Global Burden of Disease Study 2015. Lancet 2016, 388, 1459–1544. [Google Scholar] [CrossRef] [Green Version]
  2. Sanchis-Gomar, F.; Perez-Quilis, C.; Leischik, R.; Lucia, A. Epidemiology of coronary heart disease and acute coronary syndrome. Ann. Transl. Med. 2016, 4, 256. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  3. Guilbert, J.J. The world health report 2002—Reducing risks, promoting healthy life. Educ. Health 2003, 16, 230. [Google Scholar] [CrossRef]
  4. Bhatnagar, P.; Wickramasinghe, K.; Wilkins, E.; Townsend, N. Trends in the epidemiology of cardiovascular disease in the UK. Heart 2016, 102, 1945–1952. [Google Scholar] [CrossRef]
  5. Dalen, J.E.; Alpert, J.S.; Goldberg, R.J.; Weinstein, R.S. The epidemic of the 20(th) century: Coronary heart disease. Am. J. Med. 2014, 127, 807–812. [Google Scholar] [CrossRef]
  6. Gupta, R.; Mohan, I.; Narula, J. Trends in Coronary Heart Disease Epidemiology in India. Ann. Glob. Health 2016, 82, 307–315. [Google Scholar] [CrossRef] [Green Version]
  7. Zhu, K.F.; Wang, Y.M.; Zhu, J.Z.; Zhou, Q.Y.; Wang, N.F. National prevalence of coronary heart disease and its relationship with human development index: A systematic review. Eur. J. Prev. Cardiol. 2016, 23, 530–543. [Google Scholar] [CrossRef]
  8. Centers for Disease Control and Prevention. Prevalence of coronary heart disease—United States, 2006–2010. MMWR Morb. Mortal. Wkly. Rep. 2011, 60, 1377–1381. [Google Scholar]
  9. Ferreira-González, I. The Epidemiology of Coronary Heart Disease. Rev. Española Cardiol. Engl. Ed. 2014, 67, 139–144. [Google Scholar] [CrossRef]
  10. Gaziano, T.A.; Bitton, A.; Anand, S.; Abrahams-Gessel, S.; Murphy, A. Growing epidemic of coronary heart disease in low- and middle-income countries. Curr. Probl. Cardiol. 2010, 35, 72–115. [Google Scholar] [CrossRef] [Green Version]
  11. Zhang, G.; Yu, C.; Zhou, M.; Wang, L.; Zhang, Y.; Luo, L. Burden of Ischaemic heart disease and attributable risk factors in China from 1990 to 2015: Findings from the global burden of disease 2015 study. BMC Cardiovasc. Disord. 2018, 18, 18. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  12. Wang, Y.; Li, Y.; Liu, X.; Zhang, H.; Abdulai, T.; Tu, R.; Tian, Z.; Qian, X.; Jiang, J.; Qiao, D.; et al. Prevalence and Influencing Factors of Coronary Heart Disease and Stroke in Chinese Rural Adults: The Henan Rural Cohort Study. Front. Public Health 2020, 7, 411. [Google Scholar] [CrossRef] [PubMed]
  13. Parish, S.; Arnold, M.; Clarke, R.; Du, H.; Wan, E.; Kurmi, O.; Chen, Y.; Guo, Y.; Bian, Z.; Collins, R.; et al. Assessment of the Role of Carotid Atherosclerosis in the Association Between Major Cardiovascular Risk Factors and Ischemic Stroke Subtypes. JAMA 2019, 2, e194873. [Google Scholar] [CrossRef] [PubMed]
  14. National Health and Family Planning Commission. China Health and Family Planning Statistical Yearbook 2017; Peking Union Medical College: Beijing, China, 2017. [Google Scholar]
  15. Moran, A.; Zhao, D.; Gu, D.; Coxson, P.; Chen, C.-S.; Cheng, J.; Liu, J.; He, J.; Goldman, L. The future impact of population growth and aging on coronary heart disease in China: Projections from the Coronary Heart Disease Policy Model-China. BMC Public Health 2008, 8, 394. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  16. Lown, B.; Wolf, M. Approaches to sudden death from coronary heart disease. Circulation 1971, 44, 130–142. [Google Scholar] [CrossRef] [Green Version]
  17. Yang, B.Y.; Hu, L.W.; Jalaludin, B.; Knibbs, L.D.; Markevych, I.; Heinrich, J.; Bloom, M.S.; Morawska, L.; Lin, S.; Jalava, P.; et al. Association Between Residential Greenness, Cardiometabolic Disorders, and Cardiovascular Disease Among Adults in China. JAMA 2020, 3, e2017507. [Google Scholar] [CrossRef]
  18. Forouzanfar, M.H.; Alexander, L.; Anderson, H.R.; Bachman, V.F.; Biryukov, S.; Brauer, M.; Burnett, R.; Casey, D.; Coates, M.M.; Cohen, A.; et al. Global, regional, and national comparative risk assessment of 79 behavioural, environmental and occupational, and metabolic risks or clusters of risks in 188 countries, 1990–2013: A systematic analysis for the Global Burden of Disease Study 2013. Lancet 2015, 386, 2287–2323. [Google Scholar] [CrossRef] [Green Version]
  19. Gerdts, E.; Regitz-Zagrosek, V. Sex differences in cardiometabolic disorders. Nat. Med. 2019, 25, 1657–1666. [Google Scholar] [CrossRef]
  20. Kokubo, Y.; Matsumoto, C. Hypertension Is a Risk Factor for Several Types of Heart Disease: Review of Prospective Studies. Adv. Exp. Med. Biol. 2017, 956, 419–426. [Google Scholar] [CrossRef]
  21. Xu, G.; You, D.; Wong, L.; Duan, D.; Kong, F.; Zhang, X.; Zhao, J.; Xing, W.; Han, L.; Li, L. Risk of all-cause and CHD mortality in women versus men with type 2 diabetes: A systematic review and meta-analysis. Eur. J. Endocrinol. 2019, 180, 243–255. [Google Scholar] [CrossRef]
  22. Reaven, G.M. Multiple CHD risk factors in type 2 diabetes: Beyond hyperglycaemia. Diabetes Obes. Metab. 2002, 4 (Suppl. 1), S13–S18. [Google Scholar] [CrossRef] [PubMed]
  23. Temple, N.J. Fat, Sugar, Whole Grains and Heart Disease: 50 Years of Confusion. Nutrients 2018, 10, 39. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  24. DiNicolantonio, J.J.; Lucan, S.C.; O’Keefe, J.H. The Evidence for Saturated Fat and for Sugar Related to Coronary Heart Disease. Prog. Cardiovasc. Dis. 2016, 58, 464–472. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  25. Tziomalos, K.; Athyros, V.G.; Karagiannis, A.; Mikhailidis, D.P. Dyslipidemia as a risk factor for ischemic stroke. Curr. Top. Med. Chem. 2009, 9, 1291–1297. [Google Scholar] [CrossRef]
  26. Yu, J.N.; Cunningham, J.A.; Thouin, S.R.; Gurvich, T.; Liu, D. Hyperlipidemia. Prim. Care 2000, 27, 541–587. [Google Scholar] [CrossRef]
  27. Zhang, X.-F.; Attia, J.; D’Este, C.; Yu, X.-H.; Wu, X.-G. A risk score predicted coronary heart disease and stroke in a Chinese cohort. J. Clin. Epidemiol. 2005, 58, 951–958. [Google Scholar] [CrossRef]
  28. Wilson, P.W.F.; D’Agostino, R.B.; Levy, D.; Belanger, A.M.; Silbershatz, H.; Kannel, W.B. Prediction of Coronary Heart Disease Using Risk Factor Categories. Circulation 1998, 97, 1837–1847. [Google Scholar] [CrossRef] [Green Version]
  29. Anderson, K.M.; Wilson, P.W.; Odell, P.M.; Kannel, W.B. An updated coronary risk profile. A statement for health professionals. Circulation 1991, 83, 356–362. [Google Scholar] [CrossRef] [Green Version]
  30. Anderson, K.M.; Odell, P.M.; Wilson, P.W.; Kannel, W.B. Cardiovascular disease risk profiles. Am. Heart J. 1991, 121, 293–298. [Google Scholar] [CrossRef]
  31. Liu, J.; Hong, Y.; D’Agostino, R.B., Sr.; Wu, Z.; Wang, W.; Sun, J.; Wilson, P.W.; Kannel, W.B.; Zhao, D. Predictive value for the Chinese population of the Framingham CHD risk assessment tool compared with the Chinese Multi-Provincial Cohort Study. JAMA 2004, 291, 2591–2599. [Google Scholar] [CrossRef]
  32. Chambless, L.E.; Folsom, A.R.; Sharrett, A.R.; Sorlie, P.; Couper, D.; Szklo, M.; Nieto, F.J. Coronary heart disease risk prediction in the Atherosclerosis Risk in Communities (ARIC) study. J. Clin. Epidemiol. 2003, 56, 880–890. [Google Scholar] [CrossRef]
  33. Assmann, G.; Cullen, P.; Schulte, H. Simple scoring scheme for calculating the risk of acute coronary events based on the 10-year follow-up of the prospective cardiovascular Münster (PROCAM) study. Circulation 2002, 105, 310–315. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  34. Ferrario, M.; Chiodini, P.; Chambless, L.E.; Cesana, G.; Vanuzzo, D.; Panico, S.; Sega, R.; Pilotto, L.; Palmieri, L.; Giampaoli, S. Prediction of coronary events in a low incidence population. Assessing accuracy of the CUORE Cohort Study prediction equation. Int. J. Epidemiol. 2005, 34, 413–421. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  35. Stevens, R.J.; Kothari, V.; Adler, A.I.; Stratton, I.M. The UKPDS risk engine: A model for the risk of coronary heart disease in Type II diabetes (UKPDS 56). Clin. Sci. 2001, 101, 671–679. [Google Scholar] [CrossRef]
  36. Pekkanen, J.; Tervahauta, M.; Nissinen, A.; Karvonen, M.J. Does the predictive value of baseline coronary risk factors change over a 30-year follow-up? Cardiology 1993, 82, 181–190. [Google Scholar] [CrossRef] [PubMed]
  37. Benfante, R.; Reed, D. Is elevated serum cholesterol level a risk factor for coronary heart disease in the elderly? JAMA 1990, 263, 393–396. [Google Scholar] [CrossRef] [PubMed]
  38. Menotti, A.; Blackburn, H.; Kromhout, D.; Nissinen, A.; Karvonen, M.; Aravanis, C.; Dontas, A.; Fidanza, F.; Giampaoli, S. The inverse relation of average population blood pressure and stroke mortality rates in the seven countries study: A paradox. Eur. J. Epidemiol. 1997, 13, 379–386. [Google Scholar] [CrossRef]
  39. Wong, T.Y.; Klein, R.; Klein, B.E.K.; Tielsch, J.M.; Hubbard, L.; Nieto, F.J. Retinal Microvascular Abnormalities and their Relationship with Hypertension, Cardiovascular Disease, and Mortality. Surv. Ophthalmol. 2001, 46, 59–80. [Google Scholar] [CrossRef]
  40. Wu, Y.; Liu, X.; Li, X.; Li, Y.; Zhao, L.; Chen, Z.; Li, Y.; Rao, X.; Zhou, B.; Detrano, R.; et al. Estimation of 10-year risk of fatal and nonfatal ischemic cardiovascular diseases in Chinese adults. Circulation 2006, 114, 2217–2225. [Google Scholar] [CrossRef] [Green Version]
  41. Patton, N.; Aslam, T.; Macgillivray, T.; Pattie, A.; Deary, I.J.; Dhillon, B. Retinal vascular image analysis as a potential screening tool for cerebrovascular disease: A rationale based on homology between cerebral and retinal microvasculatures. J. Anat. 2005, 206, 319–348. [Google Scholar] [CrossRef]
  42. Cheng, L.; Barlis, P.; Gibson, J.; Colville, D.; Hutchinson, A.; Gleeson, G.; Lamoureux, E.; VanGaal, W.; Savige, J. Microvascular retinopathy and angiographically-demonstrated coronary artery disease: A cross-sectional, observational study. PLoS ONE 2018, 13, e0192350. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  43. Tabatabaee, A.; Asharin, M.R.; Dehghan, M.H.; Pourbehi, M.R.; Nasiri-Ahmadabadi, M.; Assadi, M. Retinal vessel abnormalities predict coronary artery diseases. Perfusion 2013, 28, 232–237. [Google Scholar] [CrossRef] [PubMed]
  44. Moss, H.E. Retinal Vascular Changes are a Marker for Cerebral Vascular Diseases. Curr. Neurol. Neurosci. Rep. 2015, 15, 40. [Google Scholar] [CrossRef] [PubMed]
  45. Zhuo, Y.; Yu, H.; Yang, Z.; Zee, B.; Lee, J.; Kuang, L. Prediction Factors of Recurrent Stroke among Chinese Adults Using Retinal Vasculature Characteristics. J. Stroke Cerebrovasc. Dis. 2017, 26, 679–685. [Google Scholar] [CrossRef] [PubMed]
  46. Wang, J.; Leng, F.; Li, Z.; Tang, X.; Qian, H.; Li, X.; Zhang, Y.; Chen, X.; Du, H.; Liu, P. Retinal vascular abnormalities and their associations with cardiovascular and cerebrovascular diseases: A Study in rural southwestern Harbin, China. BMC Ophthalmol. 2020, 20, 136. [Google Scholar] [CrossRef]
  47. Juutilainen, A.; Lehto, S.; Rönnemaa, T.; Pyörälä, K.; Laakso, M. Retinopathy predicts cardiovascular mortality in type 2 diabetic men and women. Diabetes Care 2007, 30, 292–299. [Google Scholar] [CrossRef] [Green Version]
  48. Wang, J.J.; Liew, G.; Wong, T.Y.; Smith, W.; Klein, R.; Leeder, S.R.; Mitchell, P. Retinal vascular calibre and the risk of coronary heart disease-related death. Heart Br. Card. Soc. 2006, 92, 1583–1587. [Google Scholar] [CrossRef] [PubMed]
  49. Klein, R.; Klein, B.E.; Moss, S.E.; Wong, T.Y. Retinal vessel caliber and microvascular and macrovascular disease in type 2 diabetes: XXI: The Wisconsin Epidemiologic Study of Diabetic Retinopathy. Ophthalmology 2007, 114, 1884–1892. [Google Scholar] [CrossRef]
  50. Doubal, F.N.; de Haan, R.; MacGillivray, T.J.; Cohn-Hokke, P.E.; Dhillon, B.; Dennis, M.S.; Wardlaw, J.M. Retinal arteriolar geometry is associated with cerebral white matter hyperintensities on magnetic resonance imaging. Int. J. Stroke Off. J. Int. Stroke Soc. 2010, 5, 434–439. [Google Scholar] [CrossRef]
  51. Witt, N.; Wong, T.Y.; Hughes, A.D.; Chaturvedi, N.; Klein, B.E.; Evans, R.; McNamara, M.; Thom, S.A.; Klein, R. Abnormalities of retinal microvascular structure and risk of mortality from ischemic heart disease and stroke. Hypertension 2006, 47, 975–981. [Google Scholar] [CrossRef] [Green Version]
  52. Fihn, S.D.; Blankenship, J.C.; Alexander, K.P.; Bittl, J.A.; Byrne, J.G.; Fletcher, B.J.; Fonarow, G.C.; Lange, R.A.; Levine, G.N.; Maddox, T.M.; et al. 2014 ACC/AHA/AATS/PCNA/SCAI/STS focused update of the guideline for the diagnosis and management of patients with stable ischemic heart disease: A report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines, and the American Association for Thoracic Surgery, Preventive Cardiovascular Nurses Association, Society for Cardiovascular Angiography and Interventions, and Society of Thoracic Surgeons. J. Thorac. Cardiovasc. Surg. 2015, 149, e5–e23. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  53. Amsterdam, E.A.; Wenger, N.K.; Brindis, R.G.; Casey, D.E.; Ganiats, T.G.; Holmes, D.R.; Jaffe, A.S.; Jneid, H.; Kelly, R.F.; Kontos, M.C.; et al. 2014 AHA/ACC Guideline for the Management of Patients with Non–ST-Elevation Acute Coronary Syndromes: A Report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. J. Am. Coll. Cardiol. 2014, 64, e139–e228. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  54. Ibanez, B.; James, S.; Agewall, S.; Antunes, M.J.; Bucciarelli-Ducci, C.; Bueno, H.; Caforio, A.L.P.; Crea, F.; Goudevenos, J.A.; Halvorsen, S.; et al. 2017 ESC Guidelines for the management of acute myocardial infarction in patients presenting with ST-segment elevation: The Task Force for the management of acute myocardial infarction in patients presenting with ST-segment elevation of the European Society of Cardiology (ESC). Eur. Heart J. 2018, 39, 119–177. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  55. Zee, B.C.-y.; Lee, J.J.-w.; Li, E.Q. Method and Device for Retinal Image Analysis. U.S. Patent 8,787,638, 22 July 2014. [Google Scholar]
  56. Lai, M.; Lee, J.; Chiu, S.; Charm, J.; So, W.Y.; Yuen, F.P.; Kwok, C.; Tsoi, J.; Lin, Y.; Zee, B. A machine learning approach for retinal images analysis as an objective screening method for children with autism spectrum disorder. eClinicalMedicine 2020, 28, 100588. [Google Scholar] [CrossRef] [PubMed]
  57. Guo, V.Y.; Cao, B.; Wu, X.; Lee, J.J.W.; Zee, B.C. Prospective Association between Diabetic Retinopathy and Cardiovascular Disease-A Systematic Review and Meta-analysis of Cohort Studies. J. Stroke Cerebrovasc. Dis. 2016, 25, 1688–1695. [Google Scholar] [CrossRef] [PubMed]
  58. Guo, V.Y.; Chan, J.C.; Chung, H.; Ozaki, R.; So, W.; Luk, A.; Lam, A.; Lee, J.; Zee, B.C. Retinal Information is Independently Associated with Cardiovascular Disease in Patients with Type 2 diabetes. Sci. Rep. 2016, 6, 19053. [Google Scholar] [CrossRef] [Green Version]
  59. Fan, R.-E.; Chen, P.-H.; Lin, C.-J.; Joachims, T. Working set selection using second order information for training support vector machines. J. Mach. Learn. Res. 2005, 6, 1889–1918. [Google Scholar]
  60. Schölkopf, B.; Smola, A.J.; Bach, F. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond; MIT Press: Cambridge, MA, USA, 2002. [Google Scholar]
  61. Newcombe, R.G. Two-sided confidence intervals for the single proportion: Comparison of seven methods. Stat. Med. 1998, 17, 857–872. [Google Scholar] [CrossRef]
  62. Lau, A.Y.; Mok, V.; Lee, J.; Fan, Y.; Zeng, J.; Lam, B.; Wong, A.; Kwok, C.; Lai, M.; Zee, B. Retinal image analytics detects white matter hyperintensities in healthy adults. Ann. Clin. Transl. Neurol. 2019, 6, 98–105. [Google Scholar] [CrossRef]
  63. Tedeschi-Reiner, E.; Strozzi, M.; Skoric, B.; Reiner, Z. Relation of Atherosclerotic Changes in Retinal Arteries to the Extent of Coronary Artery Disease. Am. J. Cardiol. 2005, 96, 1107–1109. [Google Scholar] [CrossRef]
  64. Tedeschi-Reiner, E.; Reiner, Z.; Sonicki, Z. Atherosclerosis of retinal arteries in men: Role of serum lipoproteins and apoproteins. Croat. Med. J. 2004, 45, 333–337. [Google Scholar] [PubMed]
  65. Theuerle, J.D.; Al-Fiadh, A.H.; Amirul Islam, F.M.; Patel, S.K.; Burrell, L.M.; Wong, T.Y.; Farouque, O. Impaired retinal microvascular function predicts long-term adverse events in patients with cardiovascular disease. Cardiovasc. Res. 2021, 117, 1949–1957. [Google Scholar] [CrossRef] [PubMed]
  66. McGeechan, K.; Liew, G.; Macaskill, P.; Irwig, L.; Klein, R.; Klein, B.E.; Wang, J.J.; Mitchell, P.; Vingerling, J.R.; Dejong, P.T.; et al. Meta-analysis: Retinal vessel caliber and risk for coronary heart disease. Ann. Intern. Med. 2009, 151, 404–413. [Google Scholar] [CrossRef] [PubMed]
  67. Wang, N.; Liang, C. Relationship of Gensini score with retinal vessel diameter and arteriovenous ratio in senile CHD. Open Life Sci. 2021, 16, 737–745. [Google Scholar] [CrossRef] [PubMed]
  68. Cordina, R.; Leaney, J.; Golzan, M.; Grieve, S.; Celermajer, D.S.; Graham, S.L. Ophthalmological consequences of cyanotic congenital heart disease: Vascular parameters and nerve fibre layer. Clin. Exp. Ophthalmol. 2015, 43, 115–123. [Google Scholar] [CrossRef]
  69. Hart, W.E.; Goldbaum, M.; Côté, B.; Kube, P.; Nelson, M.R. Measurement and classification of retinal vascular tortuosity. Int. J. Med. Inform. 1999, 53, 239–252. [Google Scholar] [CrossRef]
  70. Vilela, M.A.; Amaral, C.E.; Ferreira, M.A.T. Retinal vascular tortuosity: Mechanisms and measurements. Eur. J. Ophthalmol. 2021, 31, 1497–1506. [Google Scholar] [CrossRef]
  71. Kim, B.J.; Kim, S.M.; Kang, D.W.; Kwon, S.U.; Suh, D.C.; Kim, J.S. Vascular tortuosity may be related to intracranial artery atherosclerosis. Int. J. Stroke 2015, 10, 1081–1086. [Google Scholar] [CrossRef]
  72. Cheung, C.Y.; Zheng, Y.; Hsu, W.; Lee, M.L.; Lau, Q.P.; Mitchell, P.; Wang, J.J.; Klein, R.; Wong, T.Y. Retinal vascular tortuosity, blood pressure, and cardiovascular risk factors. Ophthalmology 2011, 118, 812–818. [Google Scholar] [CrossRef]
  73. Sasongko, M.B.; Wong, T.Y.; Nguyen, T.T.; Cheung, C.Y.; Shaw, J.E.; Wang, J.J. Retinal vascular tortuosity in persons with diabetes and diabetic retinopathy. Diabetologia 2011, 54, 2409–2416. [Google Scholar] [CrossRef]
  74. Sandoval-Garcia, E.; McLachlan, S.; Price, A.H.; MacGillivray, T.J.; Strachan, M.W.J.; Wilson, J.F.; Price, J.F. Retinal arteriolar tortuosity and fractal dimension are associated with long-term cardiovascular outcomes in people with type 2 diabetes. Diabetologia 2021, 64, 2215–2227. [Google Scholar] [CrossRef] [PubMed]
  75. Duncan, B.B.; Wong, T.Y.; Tyroler, H.A.; Davis, C.E.; Fuchs, F.D. Hypertensive retinopathy and incident coronary heart disease in high risk men. Br. J. Ophthalmol. 2002, 86, 1002–1006. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  76. Hu, Y.H.; Pan, X.R.; Liu, P.A.; Li, G.W.; Howard, B.V.; Bennett, P.H. Coronary heart disease and diabetic retinopathy in newly diagnosed diabetes in Da Qing, China: The Da Qing IGT and Diabetes Study. Acta Diabetol. 1991, 28, 169–173. [Google Scholar] [CrossRef] [PubMed]
  77. Martin, S.C.; Butcher, A.; Martin, N.; Farmer, J.; Dobson, P.M.; Bartlett, W.A.; Jones, A.F. Cardiovascular risk assessment in patients with retinal vein occlusion. Br. J. Ophthalmol. 2002, 86, 774–776. [Google Scholar] [CrossRef] [Green Version]
  78. Rim, T.H.; Lee, C.J.; Tham, Y.C.; Cheung, N.; Yu, M.; Lee, G.; Kim, Y.; Ting, D.S.W.; Chong, C.C.Y.; Choi, Y.S.; et al. Deep-learning-based cardiovascular risk stratification using coronary artery calcium scores predicted from retinal photographs. Lancet Digit. Health 2021, 3, e306–e316. [Google Scholar] [CrossRef]
  79. Ma, Y.; Xiong, J.; Zhu, Y.; Ge, Z.; Hua, R.; Fu, M.; Li, C.; Wang, B.; Dong, L.; Zhao, X.; et al. Deep learning algorithm using fundus photographs for 10-year risk assessment of ischemic cardiovascular diseases in China. Sci. Bull. 2022, 67, 17–20. [Google Scholar] [CrossRef]
Figure 1. The ROC curve of the classification model for CHD.
Figure 1. The ROC curve of the classification model for CHD.
Jcm 11 02687 g001
Figure 2. The classification model for CHD in box plot.
Figure 2. The classification model for CHD in box plot.
Jcm 11 02687 g002
Figure 3. Subgroup analysis with classification models for CHD patients (A) without and (B) without diabetes in box plot.
Figure 3. Subgroup analysis with classification models for CHD patients (A) without and (B) without diabetes in box plot.
Jcm 11 02687 g003
Table 1. Patient characteristics between coronary heart disease (CHD) and cardiometabolic disorders.
Table 1. Patient characteristics between coronary heart disease (CHD) and cardiometabolic disorders.
Basic CharacteristicsControl
n = 128
CHD
n = 188
p
Age (years)52.13 ± 11.7863.89 ± 11.40<0.001
Sex n, (%) <0.001
Male42(32.81%)103(55.79%)
Female86(67.19%)85(45.21%)
Smoking n, (%) 0.891
No115(89.84%)168(89.36%)
Yes13(10.16%)20(10.64%)
Drinking n, (%) 0.100
No114(89.06%)177(94.15%)
Yes14(10.94%)11(5.85%)
BMI group 0.200
<2470(54.69%)89(47.34%)
≥2458(45.31%)99(52.66%)
Diabetes n, (%) <0.001
No97(75.78%)95(50.53%)
Yes31(24.22%)93(49.47%)
HbA1c (%)6.25 ± 1.416.66 ± 1.260.019
Fasting glucose (mmol/L)5.16 ± 2.165.63 ± 1.950.050
Hypertension n, (%) <0.001
No49(38.28%)34(18.09%)
Yes79(61.72%)154(81.91%)
SBP (mmHg)135.39 ± 22.05133.87 ± 20.260.529
DBP (mmHg)85.53 ± 14.3980.64 ± 13.470.002
Dyslipidemia n, (%) 0.043
No50(39.06%)53(28.19%)
Yes78(60.94%)135(71.81%)
TG (mmol/L)1.85 ± 1.341.90 ± 1.900.791
TC (mmol/L)4.56 ± 0.984.32 ± 1.290.076
HDL-C (mmol/L)1.20 ± 0.331.13 ± 0.310.073
LDL-C (mmol/L)2.85 ± 0.902.67 ± 1.100.119
Table 2. Comparison of retinal characteristics between CHD and Control.
Table 2. Comparison of retinal characteristics between CHD and Control.
Retinal CharacteristicsControl
n = 128
CHD
n = 188
p
lCRVE18.34 ± 0.3618.21 ± 0.380.002
lMBCV1.21 ± 0.031.20 ± 0.030.014
lMAasymmetry0.85 ± 0.010.85 ± 0.01<0.001
lMVasymmetry0.75 ± 0.010.74 ± 0.010.008
lAocclusion0.13 ± 0.080.16 ± 0.090.032
lExudates0.23 ± 0.070.26 ± 0.080.001
lTortuosity_av0.20 ± 0.070.22 ± 0.080.020
lTortuosity_a0.14 ± 0.060.16 ± 0.070.013
lTortuosity_v0.15 ± 0.060.18 ± 0.08<0.001
rCRAE11.17 ± 0.2611.10 ± 0.250.028
rMBCV1.20 ± 0.021.20 ± 0.020.015
rMAangle76.76 ± 1.4476.32 ± 1.440.007
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Qu, Y.; Lee, J.J.-W.; Zhuo, Y.; Liu, S.; Thomas, R.L.; Owens, D.R.; Zee, B.C.-Y. Risk Assessment of CHD Using Retinal Images with Machine Learning Approaches for People with Cardiometabolic Disorders. J. Clin. Med. 2022, 11, 2687. https://doi.org/10.3390/jcm11102687

AMA Style

Qu Y, Lee JJ-W, Zhuo Y, Liu S, Thomas RL, Owens DR, Zee BC-Y. Risk Assessment of CHD Using Retinal Images with Machine Learning Approaches for People with Cardiometabolic Disorders. Journal of Clinical Medicine. 2022; 11(10):2687. https://doi.org/10.3390/jcm11102687

Chicago/Turabian Style

Qu, Yimin, Jack Jock-Wai Lee, Yuanyuan Zhuo, Shukai Liu, Rebecca L. Thomas, David R. Owens, and Benny Chung-Ying Zee. 2022. "Risk Assessment of CHD Using Retinal Images with Machine Learning Approaches for People with Cardiometabolic Disorders" Journal of Clinical Medicine 11, no. 10: 2687. https://doi.org/10.3390/jcm11102687

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop