Next Article in Journal
The Role of Ultrasound in Epicutaneo-Caval Catheter Insertion in Neonates: Systematic Review, Meta-Analysis and Future Perspectives
Previous Article in Journal
Precision in Dermatology: Developing an Optimal Feature Selection Framework for Skin Lesion Classification
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Predicting Future Incidences of Cardiac Arrhythmias Using Discrete Heartbeats from Normal Sinus Rhythm ECG Signals via Deep Learning Methods

1
Synergy A.I. Co., Ltd., Seoul 07573, Republic of Korea
2
Department of Cardiology, Ewha Womans University Mokdong Hospital, Seoul 07985, Republic of Korea
3
Tomocube Inc., Daejeon 34141, Republic of Korea
4
Department of Urology, Ewha Womans University Mokdong Hospital, Seoul 07985, Republic of Korea
*
Authors to whom correspondence should be addressed.
These authors contributed equally to this work.
Diagnostics 2023, 13(17), 2849; https://doi.org/10.3390/diagnostics13172849
Submission received: 28 July 2023 / Revised: 29 August 2023 / Accepted: 1 September 2023 / Published: 3 September 2023
(This article belongs to the Section Machine Learning and Artificial Intelligence in Diagnostics)

Abstract

:
This study aims to compare the effectiveness of using discrete heartbeats versus an entire 12-lead electrocardiogram (ECG) as the input for predicting future occurrences of arrhythmia and atrial fibrillation using deep learning models. Experiments were conducted using two types of inputs: a combination of discrete heartbeats extracted from 12-lead ECG and an entire 12-lead ECG signal of 10 s. This study utilized 326,904 ECG signals from 134,447 patients and categorized them into three groups: true–normal sinus rhythm (T-NSR), atrial fibrillation–normal sinus rhythm (AF-NSR), and clinically important arrhythmia–normal sinus rhythm (CIA-NSR). The T-NSR group comprised patients with at least three normal rhythms in a year and no atrial fibrillation or arrhythmias history. Clinically important arrhythmia included atrial fibrillation, atrial flutter, atrial premature contraction, atrial tachycardia, ventricular premature contraction, ventricular tachycardia, right and left bundle branch block, and atrioventricular block over the second degree. The AF-NSR group included normal sinus rhythm paired with atrial fibrillation or atrial flutter within 14 days, and the CIA-NSR group comprised normal sinus rhythm paired with CIA occurring within 14 days. Three deep learning models, ResNet-18, LSTM, and Transformer-based models, were utilized to distinguish T-NSR from AF-NSR and T-NSR from CIA-NSR. The experiments demonstrated the potential of using discrete heartbeats in predicting future arrhythmia and atrial fibrillation incidences extracted from 12-lead electrocardiogram (ECG) signals alone, without any additional patient information. The analysis reveals that these discrete heartbeats contain subtle patterns that deep learning models can identify. Focusing on discrete heartbeats may lead to more timely and accurate diagnoses of these conditions, improving patient outcomes and enabling automated diagnosis using ECG signals as a biomarker.

1. Introduction

Cardiac arrhythmias, encompassing conditions such as atrial fibrillation, are among the leading causes of concern in cardiovascular health. The insidious nature of these conditions, often manifesting asymptomatically or with minimal symptoms, renders them particularly elusive to standard detection methods [1,2,3]. The stakes of such undetected irregularities are alarmingly high, with potential outcomes ranging from debilitating strokes to heart failures and, in the most severe instances, culminating in sudden cardiac death [4].
The gravity of atrial fibrillation, a prominent subtype of arrhythmias, lies in its strong correlation with intensified risks of both stroke and heart failure [5,6,7,8,9]. This association highlights the pressing need for effective early detection mechanisms and prompt interventions. Despite the profound clinical implications, these conditions often go undetected, only becoming evident when they result in more severe outcomes.
The motivation behind our study is to bridge this detection gap. Recognizing the challenges faced in identifying these conditions early, we delved into innovative methods aimed at enhancing the screening process. By refining the current diagnostic paradigms, we believe we can bring a robust solution that aids in the proactive management of atrial fibrillation and other arrhythmias [10,11]. Through this endeavor, our motivation is clear: to mitigate the potential complications and enhance the quality of life for patients across the globe.
Electrocardiogram (ECG) recording includes 1-dimensional time series data that measure the heart’s electrical activity, and it is a valuable tool for diagnosing and monitoring arrhythmias and atrial fibrillation. Recent studies have demonstrated the potential of deep learning techniques in predicting the future incidence of arrhythmias and atrial fibrillation using ECG signals [12,13,14,15,16]. Previous approaches to ECG analysis have mainly concentrated on using whole 12-lead EKG recordings as the input for deep learning models due to the popularity of two-dimensional CNNs in analyzing various data types, such as auditory signals that were transformed into two-dimensional image data. However, applying the same approach to ECG signals may not be optimal due to the challenges posed by the complex and noisy nature of the electrical signals generated by the heart, which are superimposed on various noise sources, such as muscle movement, respiration, and electrical interference from other equipment.
The utilization of discrete heartbeats as input data has been identified as a more optimal approach for detecting subtle abnormalities indicating future incidences of atrial fibrillation and other arrhythmias [17,18], compared to using whole 12-lead ECG recordings as the input. This approach enables the detection of critical temporal events, improving the performance of predictive models. Compared to using complete 12-lead ECG recordings as the input data, this approach allows for more focused analysis and reduces the need for larger datasets (Figure 1). Adopting this method facilitates the identification of the key indicators of potential cardiac issues, enhancing the accuracy of predictions.
To further optimize the approach for identifying subtle abnormalities in ECG signals, our methodology for predicting future cardiac events from normal sinus rhythm relies exclusively on the ECG signal, without using additional patient data, such as electronic medical records, that contain potentially sensitive or private information, such as age, gender, medical history, family history of heart disease, medication use, lifestyle factors (smoking and alcohol consumption), and comorbid conditions (hypertension or diabetes). While utilizing such data could enhance the accuracy of ECG prediction algorithms, deep learning models may focus more on medical records than ECG signals, leading to biased prediction results.
The primary aim of our study is to demonstrate that utilizing discrete heartbeats extracted from 10-s 12-lead sinus rhythm ECGs as inputs yields superior results compared to using an entire 12-lead heartbeat as the input for predicting future incidences of cardiac arrhythmias and atrial fibrillation. We conducted two distinct experiments: one for predicting the future incidence of atrial fibrillation, and another for predicting arrhythmias with prediction windows of 14 days. The reason for conducting separate experiments for arrhythmia and atrial fibrillation, despite atrial fibrillation being a type of arrhythmia, is to precisely analyze and understand the distinct characteristics and patterns associated with each condition. Isolating atrial fibrillation as a separate experiment allows for a more focused investigation into the unique features and predictive factors specific to it. The chosen prediction windows were aligned with the typical duration of wearing cardiac event monitors, which ranged up to 14 days. Evaluating the effectiveness of our approach in predicting clinically important arrhythmias within these windows provides insight into its potential usefulness in clinical practice. Moreover, our approach’s reliance on the ECG signal makes it a practical and feasible solution for clinical implementation, given that ECGs are routinely performed in clinical settings. Our study results indicate that using discrete heartbeats as the input yielded superior results compared to the conventional approach and could be a valuable tool for healthcare providers in predicting future cardiac arrhythmias from normal sinus rhythm and improving patient care and disease management.

2. Materials and Methods

2.1. Data Information and Study Population

We included 134,447 patients with 326,904 ECGs acquired from two Ewha Womans University Hospitals in Mokdong and Seoul, Republic of Korea, between May 2017 and May 2022. Raw ECGs were obtained from Philips (236,645 ECGs) and General Electric (90,259 ECGs) ECG machines in XML format. Philips ECGs are standard 10-s, 12-lead ECGs with a sampling rate of 500 Hz. GE ECGs are 10-s, 8- or 12-lead ECGs with a sampling rate of 500 Hz. The 8-lead ECGs from the General Electric ECG machine were reconstructed to 12 leads using Einthoven’s law and Goldberger’s equations [19].

2.2. Study Group Selection

We categorized 326,904 ECG datasets into three groups: true–normal sinus rhythm (“T-NSR”), atrial fibrillation–normal sinus rhythm (“AF-NSR”), and clinically important arrhythmia–normal sinus rhythm (“CIA-NSR”). We defined arrhythmias based on several criteria, which included atrial fibrillation, atrial flutter, atrial arrhythmia, premature ventricular contraction, right and left bundle branch block, and any atrioventricular block exceeding the second degree. Each of these conditions holds clinical significance and necessitates medical intervention.

2.2.1. Study Group Selection with Automated Labels

For the T-NSR group, we considered patients who recorded a minimum of three ECGs displaying normal sinus rhythm over 12 months and who had no documented history of atrial fibrillation or other arrhythmias. From this pool of T-NSR ECGs, we randomly selected one ECG per patient. The AF-NSR group comprised ECGs explicitly labeled as normal sinus rhythm and which had a corresponding ECG showing atrial fibrillation or atrial flutter within the subsequent 14 days. The CIA-NSR group included normal sinus rhythm paired with ECGs that showed CIA occurring within 14 days after the initial normal sinus rhythm reading, as illustrated in Figure 2.
To ensure the integrity of our dataset, we omitted any ECGs that were flawed or had missing or inconclusive interpretations from the T-NSR, AF-NSR, and CIA-NSR groups. To uphold consistency in our findings and focus on the adult demographic, we also excluded the ECG records of individuals younger than 18.

2.2.2. Study Group Selection with Manual Labels

Each ECG interpretation, thus far, was determined by automatic symptom analysis reports from the Philips and GE ECG machines. To ensure the accuracy of our data, we converted all selected ECGs into waveform images. We asked trained practitioners with more than five years of experience in cardiology to manually annotate them. Any ECGs with discrepancies between the automatic diagnosis and manual annotations were excluded from the study. To ensure robust model evaluation and simulate real-world scenarios, we partitioned the dataset based on the dates of the ECG scans and the patients who underwent them. The training and validation set spanned from 23 May 2017 to 10 June 2021, while the test set covered 11 June 2021 to 23 May 2022. For convenience in the learning context, train and validation set separation was performed on the ECG scan level. Following these selection processes, we obtained each group’s final number of ECGs, as shown in Figure 3.

2.3. Signal Data Preprocessing

We employed several preprocessing techniques on our 10-s 12-lead ECG signal data to obtain accurate and reliable data (Figure 4). First, we decoded the data from Base 64 encryption and then passed it through an IIR Butterworth SOS and powerline noise filters with a moving average kernel for denoising and cleansing. Next, we segmented the denoised lead signals into individual heartbeats using a QRS peak detection algorithm, and this resulted in approximately 130 individual heartbeats for a 10-s 12-lead ECG, representing the PQRST complex per single ECG signal data. Any unrecognizable heartbeats were omitted from the data to ensure accuracy and consistency. The denoising, cleansing, and PQRST complex segmentation using peak detection algorithm were handled using the NeuroKit2 library [20], allowing for efficient and standardized data processing. After the individual data preprocessing, we inherited the 12-lead EKG’s annotation to the individual heartbeats to train with the individual heartbeats.

2.4. Overview of the Model Development

Table 1 and Table 2 shows ECGs and discrete heartbeat statistics used for training, validating, and testing the model. For the analysis of the one-dimensional discrete heartbeats and the whole 12-lead ECG signals, we employed popular deep learning architectures, which are ResNet-18, Conv1D with long short-term memory (LSTM), and Conv1D with transformer [21,22,23,24,25].

2.4.1. Model Input Length

Given that every architecture integrates a convolutional layer as its initial layer, we standardized the length of individual heartbeats to match the mean length across all observed discrete heartbeats, set at 700. Heartbeats exceeding this length were truncated accordingly, whereas shorter ones were zero-padded. For the 12-lead ECG signal, we established a consistent signal length of 5000. We consciously abstained from utilizing interpolation techniques to resize the signals, as this could potentially introduce undue signal distortion.

2.4.2. Model Architectures

ResNet-18 extracts essential features of the input using convolution operations like various convolutional neural networks. To solve the vanishing gradient problem of CNN architectures [26], ResNet-18 utilizes residual learning with skip connection, as shown in Figure 5a.
Combining Conv1D and LSTM layers (Figure 5b) in a neural network architecture can capture local and long-range temporal patterns in sequential data. Conv1D layers are adept at detecting local patterns, while LSTM layers excel at modeling longer-term dependencies [27]. Alternatively, a combination of Conv1D and transformer layers (Figure 5c) can capture both local and global dependencies in the input data, with transformer layers being well-suited for modeling global dependencies [28] and Conv1D layers being effective for detecting local patterns.

2.4.3. Model Parameters and Thresholds

During the training phase (Figure 6a), we used binary cross-entropy with logits loss and AdamW optimizer with an initial learning rate of 0.0001 to optimize the model’s parameters. The output of the fully connected layer was passed through a sigmoid function to obtain a probability value for each class, ranging from 0 to 1. For the discrete heartbeat input in the validation phase, we gathered the probability scores for discrete heartbeats that were separated from the same ECG; then we averaged all the probability scores of discrete heartbeats to represent the final probability score for the ECG. Using the final probability scores of the ECGs, we searched for an optimal threshold [29] of each class. The optimal thresholds were obtained by applying thresholds between 0 and 1 in increments of 0.01 to achieve the best F1 scores in the validation dataset for each T-NSR and AF-NSR class or T-NSR and CIA-NSR class. The optimal thresholds of each class were saved along with the model weights in the validation phases. The procedures for training and validating with entire 12-lead ECG signals were conducted consistently, using discrete heartbeats as an input, excluding the step of gathering discrete heartbeats.

2.4.4. Ensemble Model for Generalizability

For each architecture, we trained five different models using five different fixed seeds that control random variables for weight initialization, data shuffling, and dropout. Experimenting with five different seeds and ensembling them can be helpful in several ways. First, it can reduce the variance in the model’s performance caused by randomness in the training process. By training the model with different random seeds, we can obtain several different versions of the model, each with its own biases and strengths. Ensembling these models can help to reduce the impact of individual biases and improve the overall performance of the model. Secondly, ensembling models trained with different seeds can provide a more robust estimate of the model’s performance. By combining the outputs of several different models, we can reduce the impact of outliers and obtain a more accurate estimate of the model’s true performance [30].
In the testing phase (Figure 6b), we ensembled the probability value of all five models by averaging the probability values for each class of an ECG. Then, we evaluated those probability values with the averaged thresholds of five models.

2.4.5. Metrics for Model Performance Evaluation

The F1 score, AUC of the ROC (AUROC), precision (positive predictive value), recall, and negative predictive value (NPV) for each T-NSR, AF-NSR pair, and T-NSR, CIA-NSR pair were used to evaluate the performance of our model. The F1 score in Equation (3) is the harmonic mean of the precision (Equation (1)) and recall (Equation (2)). The F1 score is often used as an evaluation metric in various medical AI fields, along with the AUC of the ROC. The AUC of the ROC is a performance metric ranging from 0.5 to 1 that shows the discriminatory ability of the model. The AUC of the ROC alone is not suitable to validate a model’s performance since the AUC of the ROC is sensitive to class-imbalanced datasets, such as our datasets (6.6446 NSR to 1 AF-NSR). In other words, the AUC of the ROC will be biased towards evaluating the majority class: T-NSR ECGs. The NPV in Equation (4) measures the proportion of true negative predictions among all the negative predictions. To carefully evaluate our model to the class-imbalanced dataset, we propose F1, AUROC, precision, recall, and NPV for model evaluation.
Precision = (True Positive)/(True Positive + False Positive)
Recall = (True Positive)/(True Positive + False Negative)
F1 Score = 2/((1/Precision) + (1/Recall))
NPV = (True Negative)/(True Negative + False Negative)

3. Results

3.1. Results for Different Architectures

The Conv1D+LSTM model exhibited the best performance for T-NSR/AF-NSR, achieving an average AUC of 0.9419, as illustrated in Figure 7a. Meanwhile, the ResNet-18 model stood out for T-NSR/CIA-NSR, with an average AUC of 0.9272, depicted in Figure 7b.
The findings from our study indicate that utilizing discrete heartbeats from normal sinus rhythm ECG signals as the input in deep learning models demonstrated higher efficacy in predicting future occurrences of arrhythmia and atrial fibrillation, as evident from the outcomes presented in Table 3, Table 4, Table 5 and Table 6. Specifically, for the analysis of T-NSR and CIA-NSR in Table 4, the LSTM model trained with discrete heartbeats achieved an AUC score of 0.9222, outperforming the LSTM model trained with entire 12-lead ECG signals, which achieved an AUC score of 0.8909. Similarly, for the analysis of T-NSR and AF-NSR, the LSTM model utilizing discrete heartbeats achieved an average AUC score of 0.9419, surpassing the AUC score of 0.9124 obtained by the LSTM model trained with entire 12-lead ECG signals.

3.2. Paired t-Test for Discrete Heartbeat and 12-Lead Input

We sought to statistically compare the performance of two modeling methods using a paired t-test. A paired t-test is suitable in this context because it evaluates if there’s a significant difference between two paired groups. The “pairing” in our case came from evaluating the two input methods on the same dataset across five different seed models. The null hypothesis (H0) for our test was set as: “There is no significant difference between the performance metrics of the two input methods”. Conversely, the alternative hypothesis (H1) was set as: “There is a significant difference between the performance metrics of the two input methods.” The metrics of interest in our study were the F1 score and AUROC. The results consistently indicate p-values less than the significance level of 0.05 for both the F1 and AUROC across all models, as shown in Table 7, Table 8, Table 9 and Table 10.
A p-value below the 0.05 threshold is typically interpreted as strong evidence against the null hypothesis in many scientific disciplines. It suggests that the observed data (in our case, the differences in the performance metrics between the two methods) would be unlikely if the null hypothesis were true. Therefore, we reject the null hypothesis in favor of the alternative hypothesis, suggesting that there was a significant difference between the performance metrics of the two input methods.

4. Discussion

This study presents evidence of the effectiveness of using discrete heartbeats extracted from normal sinus rhythm ECGs in predicting future arrhythmia and atrial fibrillation incidences with deep learning methods. The results of the study also suggest that a specific biomarker for future incidences of arrhythmia and atrial fibrillation may be present in the normal sinus rhythm ECG signal.
It is worth noting that the study only utilized signals from ECG recordings and did not incorporate additional patient information, such as electronic medical records, which may raise concerns about privacy and the potential compromise of patient confidentiality. Despite relying on limited data, the study still demonstrates high performance in predicting future arrhythmia and atrial fibrillation incidences from normal sinus rhythm ECGs, suggesting that ECG signals alone may be sufficient for accurate prediction. This finding is promising, as it indicates that analyzing discrete heartbeats extracted from normal sinus rhythm ECGs may facilitate efficient and precise diagnosis and treatment without requiring extensive patient information.
Our dataset observed a pronounced proportion of CIA-NSR to T-NSR, with a ratio of 11,929 to 35,455, equating to approximately 33.6%. This stands in contrast to general population statistics, where the prevalence of arrhythmias is around 5% [31]. People typically seek hospital care for distinct health concerns, especially those related to cardiac issues. Consequently, the dataset may naturally represent a heightened occurrence of CIA, mirroring a patient group more prone to cardiac irregularities. While this offers insight into real-world situations, it may not accurately reflect the distribution in the wider community.
In artificial intelligence research using ECGs, there have been studies that predict the clinical data of patients. Several studies have successfully indicated patients’ clinical data, such as gender classification, age prediction, and heart failure prognosis [32,33,34]. These endeavors have been recognized for their accuracy, highlighting that the clinical information is already present in the ECG signals. Even without the help of artificial intelligence, anatomical and electrophysiological remodeling of the heart is reflected in the ECGs of patients with arrhythmias, including atrial fibrillation [35].
There are several limitations to the study that should be taken into consideration. Firstly, the research relied on data from only two hospitals, an aspect that inherently needs broader external validation. Broadening our data sources to encompass more hospitals or diverse patient groups would enhance the robustness and generalizability of our conclusions. Secondly, it is essential to acknowledge that the T-NSR ECGs examined in this study might inadvertently encompass instances of AF-NSR or CIA-NSR due to the absence of continuous data for labeling. This potential overlap exists despite our rigorous data collection from patients who had three or more T-NSR ECGs within a year and exhibited no clinical symptoms of AF or CIA during medical evaluations by physicians. Such challenges persist in intermittent electrocardiogram research unless continuous monitoring is employed, like long-term implantable loop recorders [36]. To overcome this limitation, we are exploring incorporating data from wearable devices for 24 and 74 h immediately following the 12-lead electrocardiogram recording as a follow-up study. Lastly, our research was retrospective, and it is recognized that a prospective study would offer a more rigorous evaluation of our findings. Recognizing this need, we initiated the “PROVISION-AF trial” in February 2023, a prospective, multicenter study registered with ClinicalTrials.gov under NCT05725187. This forward-looking approach aims to validate and potentially refine our model in real-time scenarios, enhancing its reliability and adaptability across a broader range of healthcare contexts.
For future research, it would be beneficial to investigate the specific heartbeats within individual electrocardiogram signals that predict the future incidence of atrial fibrillation and arrhythmia. By identifying these heartbeats, we can determine which components or features of discrete heartbeats act as potential biomarkers. Additionally, we could group study participants based on relevant demographic and medical factors and assign distinct threshold values to each subgroup to gain more nuanced insights into the predictive value of specific discrete heartbeat features for arrhythmia and atrial fibrillation. These approaches could provide greater insight into the underlying mechanisms and physiological factors contributing to the development of arrhythmia and atrial fibrillation and potentially develop more personalized diagnostics.
Based on the study presented in this paper, we obtained approval (Approval No. 2023000086) from the Ministry of Food and Drug Safety in South Korea for our exploratory clinical trials. Leveraging the deep learning-based cardiac arrhythmia prediction, we have developed SYN-MAC, a software-as-a-service (SaaS) product in Figure 8 offered by Synergy A.I. Co., Ltd., Seoul, Republic of Korea. This software is designed to predict future incidences of clinically significant arrhythmias and categorize them as “high risk” or “low risk” based on the threshold value of combined discrete heartbeats. With this software, we will conduct additional confirmatory clinical trials in live environments, focusing on enhancing the prediction accuracy of clinically important arrhythmias and advancing AI-based medical technologies for the early detection of diverse heart diseases.

5. Conclusions

This study’s results suggest that using discrete heartbeats extracted from normal sinus rhythm ECG signals to predict future clinically important arrhythmia and atrial fibrillation incidences rather than using entire 12-lead ECG signals with deep learning models is a promising approach. The LSTM models for both atrial fibrillation and clinically important arrhythmia prediction using discrete heartbeat showed strong performance compared to using entire 12-lead ECG signals. The study demonstrated that ECG signals alone were sufficient for accurate prediction, and a potential biomarker may be present in the normal sinus rhythm ECG signal. This suggests that using discrete heartbeats with deep learning models may enable the detection of subtle patterns in ECG signals, which could lead to a more accurate and earlier diagnosis of clinically important arrhythmia and atrial fibrillation.

Author Contributions

Conceptualization, T.S., J.P. and Y.K. (Yeji Kim); data curation, Y.K. (Yehyun Kim), M.L. and J.Y.; formal analysis, Y.K. (Yehyun Kim), M.L. and J.Y.; funding acquisition, T.S. and J.P.; investigation, H.M. and H.C.; methodology, Y.K. (Yehyun Kim), M.L., H.M. and H.C.; project administration, T.S. and J.P.; resources, Y.K. (Yeji Kim); software, Y.K. (Yehyun Kim) and M.L.; supervision, T.S. and J.P.; validation, Y.K. (Yehyun Kim), M.L., Y.K. (Yeji Kim), H.M. and H.C.; visualization, Y.K. (Yehyun Kim), M.L. and J.Y.; writing—original draft, Y.K. (Yehyun Kim) and M.L.; writing—review and editing, T.S., J.P., J.Y., Y.K. (Yeji Kim), H.M. and H.C. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by an Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2022-00155966, Artificial Intelligence Convergence Innovation Human Resources Development—Ewha Womans University).

Institutional Review Board Statement

This study was conducted in accordance with the Declaration of Helsinki, and approved by the Institutional Review Board of Ewha Womans University Mokdong Hospital, Seoul, Republic of Korea. (Approval No. 2019-11-016-017).

Informed Consent Statement

Patient consent was waived for this study, considering this study was a retrospective study.

Data Availability Statement

The clinical data were used under IRB approval for use only in the current study. Hence, the dataset used in this study is not publicly available.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Page, R.L.; Wilkinson, W.E.; Clair, W.K.; McCarthy, E.A.; Pritchett, E.L. Asymptomatic Arrhythmias in Patients with Symptomatic Paroxysmal Atrial Fibrillation and Paroxysmal Supraventricular Tachycardia. Circulation 1994, 89, 224–227. [Google Scholar] [CrossRef] [PubMed]
  2. Reiffel, J.A.; Verma, A.; Kowey, P.R.; Halperin, J.L.; Gersh, B.J.; Wachter, R.; Pouliot, E.; Ziegler, P.D. Incidence of Previously Undiagnosed Atrial Fibrillation Using Insertable Cardiac Monitors in a High-Risk Population. JAMA Cardiol. 2017, 2, 1120–1127. [Google Scholar] [CrossRef] [PubMed]
  3. Healey, J.S.; Connolly, S.J.; Gold, M.R.; Israel, C.W.; Van Gelder, I.C.; Capucci, A.; Lau, C.P.; Fain, E.; Yang, S.; Bailleul, C.; et al. Subclinical Atrial Fibrillation and the Risk of Stroke. N. Engl. J. Med. 2012, 366, 120–129. [Google Scholar] [CrossRef] [PubMed]
  4. Gopinathannair, R.; Etheridge, S.P.; Marchlinski, F.E.; Spinale, F.G.; Lakkireddy, D.; Olshansky, B. Arrhythmia-Induced Cardiomyopathies: Mechanisms, Recognition, and Management. J. Am. Coll. Cardiol. 2015, 66, 1714–1728. [Google Scholar] [CrossRef] [PubMed]
  5. Britton, M.; Gustafsson, C. Non-Rheumatic Atrial Fibrillation as a Risk Factor for Stroke. Stroke 1985, 16, 182–188. [Google Scholar] [CrossRef]
  6. Wolf, P.A.; Dawber, T.R.; Thomas, H.E.; Kannel, W.B. Epidemiologic Assessment of Chronic Atrial Fibrillation and Risk of Stroke: The Fiamingham Study. Neurology 1978, 28, 973. [Google Scholar] [CrossRef]
  7. Stewart, S.; Hart, C.L.; Hole, D.J.; McMurray, J.J.V. A Population-Based Study of the Long-Term Risks Associated with Atrial Fibrillation: 20-Year Follow-up of the Renfrew/Paisley Study. Am. J. Med. 2002, 113, 359–364. [Google Scholar] [CrossRef]
  8. Chugh, S.S.; Havmoeller, R.; Narayanan, K.; Singh, D.; Rienstra, M.; Benjamin, E.J.; Gillum, R.F.; Kim, Y.-H.; McAnulty, J.H.; Zheng, Z.-J.; et al. Worldwide Epidemiology of Atrial Fibrillation: A Global Burden of Disease 2010 Study. Circulation 2014, 129, 837–847. [Google Scholar] [CrossRef]
  9. Mozaffarian, D.; Benjamin, E.J.; Go, A.S.; Arnett, D.K.; Blaha, M.J.; Cushman, M.; Das, S.R.; De Ferranti, S.; Després, J.P.; Fullerton, H.J.; et al. Heart Disease and Stroke Statistics-2016 Update: A Report From the American Heart Association. Circulation 2016, 133, e38–e360. [Google Scholar]
  10. Mobile Photoplethysmographic Technology to Detect Atrial Fibrillation. J. Am. Coll. Cardiol. 2019, 74, 2365–2375. [CrossRef]
  11. Steinhubl, S.R.; Waalen, J.; Edwards, A.M.; Ariniello, L.M.; Mehta, R.R.; Ebner, G.S.; Carter, C.; Baca-Motes, K.; Felicione, E.; Sarich, T.; et al. Effect of a Home-Based Wearable Continuous ECG Monitoring Patch on Detection of Undiagnosed Atrial Fibrillation. JAMA 2018, 320, 146. [Google Scholar] [CrossRef] [PubMed]
  12. Raghunath, S.; Pfeifer, J.M.; Ulloa-Cerna, A.E.; Nemani, A.; Carbonati, T.; Jing, L.; van Maanen, D.P.; Hartzel, D.N.; Ruhl, J.A.; Lagerman, B.F.; et al. Deep Neural Networks Can Predict New-Onset Atrial Fibrillation from the 12-Lead ECG and Help Identify Those at Risk of Atrial Fibrillation–Related Stroke. Circulation 2021, 143, 1287–1298. [Google Scholar] [CrossRef]
  13. Baek, Y.-S.; Lee, S.-C.; Choi, W.; Kim, D.-H. A New Deep Learning Algorithm of 12-Lead Electrocardiogram for Identifying Atrial Fibrillation during Sinus Rhythm. Sci. Rep. 2021, 11, 12818. [Google Scholar] [CrossRef] [PubMed]
  14. Hendriks, J.M.L.; Fabritz, L. AI Can Now Identify Atrial Fibrillation through Sinus Rhythm. Lancet 2019, 394, 812–813. [Google Scholar] [CrossRef] [PubMed]
  15. Sanz-García, A.; Cecconi, A.; Vera, A.; Camarasaltas, J.M.; Alfonso, F.; Ortega, G.J.; Jimenez-Borreguero, J. Electrocardiographic Biomarkers to Predict Atrial Fibrillation in Sinus Rhythm Electrocardiograms. Heart (Br. Card. Soc.) 2021, 107, 1813–1819. [Google Scholar] [CrossRef] [PubMed]
  16. Christopoulos, G.; Graff-Radford, J.; Lopez, C.L.; Yao, X.; Attia, Z.I.; Rabinstein, A.A.; Petersen, R.C.; Knopman, D.S.; Mielke, M.M.; Kremers, W.; et al. Artificial Intelligence–Electrocardiography to Predict Incident Atrial Fibrillation. Circ. Arrhythmia Electrophysiol. 2020, 13, e009355. [Google Scholar] [CrossRef]
  17. Oh, S.L.; Ng, E.Y.K.; Tan, R.S.; Acharya, U.R. Automated Beat-Wise Arrhythmia Diagnosis Using Modified U-Net on Extended Electrocardiographic Recordings with Heterogeneous Arrhythmia Types. Comput. Biol. Med. 2019, 105, 92–101. [Google Scholar] [CrossRef]
  18. Degirmenci, M.; Ozdemir, M.A.; Izci, E.; Akan, A. Arrhythmic Heartbeat Classification Using 2D Convolutional Neural Networks. IRBM 2021, 45, 422–433. [Google Scholar] [CrossRef]
  19. Goldberger, E. The AVl, AVr, and AVf Leads. Am. Heart J. 1942, 24, 378–396. [Google Scholar] [CrossRef]
  20. Makowski, D.; Pham, T.; Lau, Z.J.; Brammer, J.C.; Lespinasse, F.; Pham, H.; Schölzel, C.; Chen, S.H.A. NeuroKit2: A Python Toolbox for Neurophysiological Signal Processing. Behav. Res. Methods 2021, 53, 1689–1696. [Google Scholar] [CrossRef]
  21. Jing, E.; Zhang, H.; Li, Z.; Liu, Y.; Ji, Z.; Ganchev, I. ECG Heartbeat Classification Based on an Improved ResNet-18 Model. Comput. Math. Methods Med. 2021, 2021, 6649970. [Google Scholar] [CrossRef]
  22. Cao, M.; Zhao, T.; Li, Y.; Zhang, W.; Benharash, P.; Ramezani, R. ECG Heartbeat Classification Using Deep Transfer Learning with Convolutional Neural Network and STFT Technique. J. Phys. Conf. Ser. 2023, 2547, 012031. [Google Scholar] [CrossRef]
  23. Zhao, D.; Jiang, R.; Feng, M.; Yang, J.; Wang, Y.; Hou, X.; Wang, X. A Deep Learning Algorithm Based on 1D CNN-LSTM for Automatic Sleep Staging. Technol. Health Care 2021, 30, 323–336. [Google Scholar] [CrossRef]
  24. Han, S.; Lee, W.; Eom, H.; Kim, J.; Park, C. Detection of Arrhythmia Using 1D Convolution Neural Network with LSTM Model. IEIE Trans. Smart Process. Comput. 2020, 9, 261–265. [Google Scholar] [CrossRef]
  25. Che, C.; Zhang, P.; Zhu, M.; Qu, Y.; Jin, B. Constrained Transformer Network for ECG Signal Processing and Arrhythmia Classification. BMC Med. Inform. Decis. Mak. 2021, 21, 184. [Google Scholar] [CrossRef] [PubMed]
  26. Basodi, S.; Ji, C.; Zhang, H.; Pan, Y. Gradient Amplification: An Efficient Way to Train Deep Neural Networks. Big Data Min. Anal. 2020, 3, 196–207. [Google Scholar] [CrossRef]
  27. Salehinejad, H.; Sankar, S.; Barfett, J.; Colak, E.; Valaee, S. Recent Advances in Recurrent Neural Networks. arXiv 2018, arXiv:1801.01078. [Google Scholar]
  28. Xu, Y.; Zhang, Q.; Zhang, J.; Tao, D. ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias. Semantic Scholar. Available online: https://www.semanticscholar.org/paper/ViTAE%3A-Vision-Transformer-Advanced-by-Exploring-Xu-Zhang/576c462dbc1f3d732b919ef1daac37a817123e52 (accessed on 29 August 2023).
  29. Lipton, Z.C.; Elkan, C.; Naryanaswamy, B. Optimal Thresholding of Classifiers to Maximize F1 Measure. Mach. Learn. Knowl. Discov. Databases 2014, 8725, 225–239. [Google Scholar] [CrossRef]
  30. Renda, A.; Barsacchi, M.; Bechini, A.; Marcelloni, F. Comparing Ensemble Strategies for Deep Learning: An Application to Facial Expression Recognition. Expert Syst. Appl. 2019, 136, 1–11. [Google Scholar] [CrossRef]
  31. Lindberg, T.; Wimo, A.; Elmståhl, S.; Qiu, C.; Bohman, D.M.; Sanmartin Berglund, J. Prevalence and Incidence of Atrial Fibrillation and Other Arrhythmias in the General Older Population: Findings from the Swedish National Study on Aging and Care. Gerontol. Geriatr. Med. 2019, 5, 233372141985968. [Google Scholar] [CrossRef]
  32. Chang, C.-H.; Lin, C.-S.; Luo, Y.-S.; Lee, Y.-T.; Lin, C. Electrocardiogram-Based Heart Age Estimation by a Deep Learning Model Provides More Information on the Incidence of Cardiovascular Disorders. Front. Cardiovasc. Med. 2022, 9, 754909. [Google Scholar] [CrossRef] [PubMed]
  33. Hirota, N.; Suzuki, S.; Arita, T.; Yagi, N.; Otsuka, T.; Yamashita, T. Prediction of Biological Age and All-Cause Mortality by 12-Lead Electrocardiogram in Patients without Structural Heart Disease. BMC Geriatr. 2021, 21, 460. [Google Scholar] [CrossRef] [PubMed]
  34. Attia, Z.I.; Friedman, P.A.; Noseworthy, P.A.; Lopez-Jimenez, F.; Ladewig, D.J.; Satam, G.; Pellikka, P.A.; Munger, T.M.; Asirvatham, S.J.; Scott, C.G.; et al. Age and Sex Estimation Using Artificial Intelligence from Standard 12-Lead ECGs. Circ. Arrhythmia Electrophysiol. 2019, 12, e007284. [Google Scholar] [CrossRef] [PubMed]
  35. Park, J.; Kim, T.H.; Lee, J.S.; Park, J.K.; Uhm, J.S.; Joung, B.; Lee, M.H.; Pak, H.N. Prolonged PR Interval Predicts Clinical Recurrence of Atrial Fibrillation after Catheter Ablation. J. Am. Heart Assoc. 2014, 3, e001277. [Google Scholar] [CrossRef] [PubMed]
  36. Lee, R.; Mittal, S. Utility and Limitations of Long-Term Monitoring of Atrial Fibrillation Using an Implantable Loop Recorder. Heart Rhythm 2018, 15, 287–295. [Google Scholar] [CrossRef]
Figure 1. Conventional and proposed input approach for ECG analysis.
Figure 1. Conventional and proposed input approach for ECG analysis.
Diagnostics 13 02849 g001
Figure 2. Normal sinus rhythm ECGs labeled by automatic symptom analysis reports were selected if AF or CIA occurred within 14 days after the respective normal sinus rhythm ECGs. Trained practitioners validated and relabeled the selected normal sinus rhythm as AF-NSR or CIA-NSR.
Figure 2. Normal sinus rhythm ECGs labeled by automatic symptom analysis reports were selected if AF or CIA occurred within 14 days after the respective normal sinus rhythm ECGs. Trained practitioners validated and relabeled the selected normal sinus rhythm as AF-NSR or CIA-NSR.
Diagnostics 13 02849 g002
Figure 3. Study group selection procedure for T-NSR/AF-NSR and T-NSR/CIA-NSR.
Figure 3. Study group selection procedure for T-NSR/AF-NSR and T-NSR/CIA-NSR.
Diagnostics 13 02849 g003
Figure 4. The 10-s 12-lead ECG signals were decoded from Base 64 and denoised using IIR Butterworth SOS and powerline noise filters. The clean signals were then segmented into individual heartbeats using a QRS peak detection algorithm via the NeuroKit2 library.
Figure 4. The 10-s 12-lead ECG signals were decoded from Base 64 and denoised using IIR Butterworth SOS and powerline noise filters. The clean signals were then segmented into individual heartbeats using a QRS peak detection algorithm via the NeuroKit2 library.
Diagnostics 13 02849 g004
Figure 5. Model architectures. (a) ResNet-18 architecture; (b) LSTM architecture with Conv-1D layer; (c) transformer architecture with Conv-1D layer.
Figure 5. Model architectures. (a) ResNet-18 architecture; (b) LSTM architecture with Conv-1D layer; (c) transformer architecture with Conv-1D layer.
Diagnostics 13 02849 g005
Figure 6. (a) Both inputs of entire 12-lead ECG and discrete heartbeats separated from 12-lead ECG were used to train AI Models with different architectures. All logit values from discrete heartbeats were averaged to represent the final logit value for the 12-lead ECG. Optimal thresholds were determined in the validation phase and then were saved along with the model weights; (b) test phase loaded model weights and threshold value that were saved in the training phase, then evaluated the ECG as T-NSR or CIA-NSR by comparing the threshold value and the final logit value.
Figure 6. (a) Both inputs of entire 12-lead ECG and discrete heartbeats separated from 12-lead ECG were used to train AI Models with different architectures. All logit values from discrete heartbeats were averaged to represent the final logit value for the 12-lead ECG. Optimal thresholds were determined in the validation phase and then were saved along with the model weights; (b) test phase loaded model weights and threshold value that were saved in the training phase, then evaluated the ECG as T-NSR or CIA-NSR by comparing the threshold value and the final logit value.
Diagnostics 13 02849 g006
Figure 7. (a) Conv1D+LSTM model AUC-ROC plot for T-NSR/AF-NSR prediction; (b) ResNet-18 model AUC-ROC plot for T-NSR/CIA-NSR prediction.
Figure 7. (a) Conv1D+LSTM model AUC-ROC plot for T-NSR/AF-NSR prediction; (b) ResNet-18 model AUC-ROC plot for T-NSR/CIA-NSR prediction.
Diagnostics 13 02849 g007
Figure 8. SaaS product SYN-MAC. SaaS product by Synergy A.I. Co., Ltd., located in Seoul, Republic of Korea. for predicting future occurrences of arrhythmias from normal sinus rhythm ECGs.
Figure 8. SaaS product SYN-MAC. SaaS product by Synergy A.I. Co., Ltd., located in Seoul, Republic of Korea. for predicting future occurrences of arrhythmias from normal sinus rhythm ECGs.
Diagnostics 13 02849 g008
Table 1. Data overview. Training, validation, and test data for T-NSR and AF-NSR.
Table 1. Data overview. Training, validation, and test data for T-NSR and AF-NSR.
AF-NSR/T-NSRNumber of HeartbeatsNumber of ECGs
T-NSRTraining3,177,26321,028
Validation1,058,1076972
Testing977,8756514
AF-NSRTraining604,7003225
Validation198,6721073
Testing228,7081385
Table 2. Data overview. Training, validation, and testing data for T-NSR and CIA-NSR.
Table 2. Data overview. Training, validation, and testing data for T-NSR and CIA-NSR.
CIA-NSR/T-NSRNumber of HeartbeatsNumber of ECGs
T-NSRTraining3,177,26321,028
Validation1,058,1076972
Testing977,8756514
CIA-NSRTraining1,113,0896855
Validation375,7002329
Testing423,5812543
Table 3. Test dataset performance evaluation for AF-NSR and T-NSR.
Table 3. Test dataset performance evaluation for AF-NSR and T-NSR.
AF-NSR/T-NSRInputResNet-18Conv1D+
LSTM
Conv1D+
Transformer
Average F1Heartbeat0.84680.84990.8371
12-Lead0.83020.80780.7837
Average AUCHeartbeat0.93920.94190.9318
12-Lead0.92780.91240.8982
T-NSR F1Heartbeat0.95800.95960.9570
12-Lead0.95640.94990.9440
AF-NSR F1Heartbeat0.73570.74020.7171
12-Lead0.70390.66560.6234
T-NSR PrecisionHeartbeat0.92760.93670.9408
12-Lead0.93020.91310.9070
AF-NSR PrecisionHeartbeat0.79810.81080.8232
12-Lead0.79470.73430.6618
T-NSR RecallHeartbeat0.99040.98370.9738
12-Lead0.98410.98980.9841
AF-NSR RecallHeartbeat0.68230.68090.6352
12-Lead0.63180.60870.5892
T-NSR NPVHeartbeat0.91050.86930.8124
12-Lead0.86390.88750.8230
AF-NSR NPVHeartbeat0.94580.94570.9382
12-Lead0.93780.93360.9295
Table 4. Test dataset performance evaluation for CIA-NSR and T-NSR.
Table 4. Test dataset performance evaluation for CIA-NSR and T-NSR.
CIA-NSR/T-NSRInputResNet-18Conv1D+
LSTM
Conv1D+
Transformer
Average F1Heartbeat0.83610.83650.8392
12-Lead0.83170.80490.7903
Average AUCHeartbeat0.92720.92220.9248
12-Lead0.91840.89090.8789
T-NSR F1Heartbeat0.91490.91310.9161
12-Lead0.91300.89750.8904
CIA-NSR F1Heartbeat0.75700.76010.7623
12-Lead0.75050.71220.6902
T-NSR PrecisionHeartbeat0.86750.87530.8719
12-Lead0.86040.84250.8407
CIA-NSR PrecisionHeartbeat0.80560.77310.8070
12-Lead0.79410.69900.6682
T-NSR RecallHeartbeat0.96780.95390.9669
12-Lead0.97250.96020.9464
CIA-NSR RecallHeartbeat0.71370.74750.7324
12-Lead0.71140.72590.7137
T-NSR NPVHeartbeat0.88270.84680.8026
12-Lead0.89430.84140.7975
CIA-NSR NPVHeartbeat0.89300.90270.8927
12-Lead0.89170.89140.8852
Table 5. Validation dataset performance evaluation for AF-NSR and T-NSR.
Table 5. Validation dataset performance evaluation for AF-NSR and T-NSR.
AF-NSR/T-NSRInputResNet-18Conv1D+
LSTM
Conv1D+
Transformer
Average F1Heartbeat0.84800.86500.8373
12-Lead0.85020.81180.7984
Average AUCHeartbeat0.94510.95230.9134
12-Lead0.93140.91360.8842
T-NSR F1Heartbeat0.96610.96920.9242
12-Lead0.96740.95960.9068
AF-NSR F1Heartbeat0.73000.76070.7503
12-Lead0.73300.66410.6900
T-NSR PrecisionHeartbeat0.94690.95470.8826
12-Lead0.94620.93090.8621
AF-NSR PrecisionHeartbeat0.75200.80440.7524
12-Lead0.80830.69780.6771
T-NSR RecallHeartbeat0.98610.98420.9699
12-Lead0.98950.99010.9571
AF-NSR RecallHeartbeat0.70920.72160.7482
12-Lead0.67050.63340.7033
T-NSR NPVHeartbeat0.85850.85500.7928
12-Lead0.88850.87010.8078
AF-NSR NPVHeartbeat0.95970.96160.9084
12-Lead0.95500.94940.8996
Table 6. Validation dataset performance evaluation for CIA-NSR and T-NSR.
Table 6. Validation dataset performance evaluation for CIA-NSR and T-NSR.
CIA-NSR/T-NSRInputResNet-18Conv1D+
LSTM
Conv1D+
Transformer
Average F1Heartbeat0.83200.82680.817
12-Lead0.82590.80640.7984
Average AUCHeartbeat0.91960.91050.9108
12-Lead0.91440.88470.8842
T-NSR F1Heartbeat0.92230.92020.9136
12-Lead0.91880.91170.9068
CIA-NSR F1Heartbeat0.74170.73340.7204
12-Lead0.73300.70110.6900
T-NSR PrecisionHeartbeat0.88270.88270.8845
12-Lead0.87420.86310.8616
CIA-NSR PrecisionHeartbeat0.76760.75720.7634
12-Lead0.75740.69380.6771
T-NSR RecallHeartbeat0.96560.96100.9447
12-Lead0.96820.96620.9571
CIA-NSR RecallHeartbeat0.71750.71100.6820
12-Lead0.71010.70850.7033
T-NSR NPVHeartbeat0.85660.84100.8024
12-Lead0.86740.84240.8078
CIA-NSR NPVHeartbeat0.90760.90540.9014
12-Lead0.90280.90190.8996
Table 7. p-values for test dataset between heartbeat and 12-lead inputs for AF-NSR/T-NSR.
Table 7. p-values for test dataset between heartbeat and 12-lead inputs for AF-NSR/T-NSR.
AF-NSR/T-NSRResNet-18Conv1D+
LSTM
Conv1D+
Transformer
p-value Avg. F10.01190.00810.0230
p-value Avg. AUC0.03930.00420.0104
Table 8. p-values for test dataset between heartbeat and 12-lead inputs for CIA-NSR / T-NSR.
Table 8. p-values for test dataset between heartbeat and 12-lead inputs for CIA-NSR / T-NSR.
CIA-NSR/T-NSRResNet-18Conv1D+
LSTM
Conv1D+
Transformer
p-value Avg. F10.04340.00920.009
p-value Avg. AUC0.02530.01320.0126
Table 9. p-values for valid. dataset between heartbeat and 12-lead inputs for AF-NSR / T-NSR.
Table 9. p-values for valid. dataset between heartbeat and 12-lead inputs for AF-NSR / T-NSR.
AF-NSR/T-NSRResNet-18Conv1D+
LSTM
Conv1D+
Transformer
p-value Avg. F10.00890.00830.015
p-value Avg. AUC0.01650.00480.0091
Table 10. p-values for valid. dataset between heartbeat and 12-lead inputs for CIA-NSR / T-NSR.
Table 10. p-values for valid. dataset between heartbeat and 12-lead inputs for CIA-NSR / T-NSR.
CIA-NSR/T-NSRResNet-18Conv1D+
LSTM
Conv1D+
Transformer
p-value Avg. F10.02330.00750.0106
p-value Avg. AUC0.02120.00940.0107
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Kim, Y.; Lee, M.; Yoon, J.; Kim, Y.; Min, H.; Cho, H.; Park, J.; Shin, T. Predicting Future Incidences of Cardiac Arrhythmias Using Discrete Heartbeats from Normal Sinus Rhythm ECG Signals via Deep Learning Methods. Diagnostics 2023, 13, 2849. https://doi.org/10.3390/diagnostics13172849

AMA Style

Kim Y, Lee M, Yoon J, Kim Y, Min H, Cho H, Park J, Shin T. Predicting Future Incidences of Cardiac Arrhythmias Using Discrete Heartbeats from Normal Sinus Rhythm ECG Signals via Deep Learning Methods. Diagnostics. 2023; 13(17):2849. https://doi.org/10.3390/diagnostics13172849

Chicago/Turabian Style

Kim, Yehyun, Myeonggyu Lee, Jaeung Yoon, Yeji Kim, Hyunseok Min, Hyungjoo Cho, Junbeom Park, and Taeyoung Shin. 2023. "Predicting Future Incidences of Cardiac Arrhythmias Using Discrete Heartbeats from Normal Sinus Rhythm ECG Signals via Deep Learning Methods" Diagnostics 13, no. 17: 2849. https://doi.org/10.3390/diagnostics13172849

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop