Next Article in Journal
Total Pancreatectomy for Multicentric Cystic Neuroendocrine Tumor of the Pancreas: A Case Report
Next Article in Special Issue
Frequency of Missed Findings on Chest Radiographs (CXRs) in an International, Multicenter Study: Application of AI to Reduce Missed Findings
Previous Article in Journal
Parkinson’s Disease Symptoms Associated with Developing On-State Axial Symptoms Early after Subthalamic Deep Brain Stimulation
Previous Article in Special Issue
Development of an Interactive AI System for the Optimal Timing Prediction of Successful Weaning from Mechanical Ventilation for Patients in Respiratory Care Centers
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Radiomics for the Prediction of Response to Antifibrotic Treatment in Patients with Idiopathic Pulmonary Fibrosis: A Pilot Study

1
Department of Medical Imaging, Chi Mei Hospital, Tainan 710, Taiwan
2
Department of Radiology, Faculty of Medicine, College of Medicine, Kaohsiung Medical University, Kaohsiung 807, Taiwan
3
Department of Health and Nutrition, Chia Nan University of Pharmacy and Science, Tainan 717, Taiwan
4
Institute of Biomedical Sciences, National Sun Yat-sen University, Kaohsiung 804, Taiwan
5
Division of Pulmonary and Critical Care Medicine, Mackay Memorial Hospital, Taipei 104, Taiwan
6
Department of Biomedical Imaging and Radiological Sciences, National Yang Ming Chiao Tung University, Taipei 112, Taiwan
7
Department of Radiology, School of Medicine, College of Medicine, Taipei Medical University, Taipei 110, Taiwan
8
Department of Radiology, Wan Fang Hospital, Taipei Medical University, Taipei 116, Taiwan
9
Department of Radiology, Mackay Memorial Hospital, Taipei 104, Taiwan
10
Department of Medicine, Mackay Medical College, New Taipei City 252, Taiwan
11
Mackay Junior College of Medicine, Nursing, and Management, New Taipei City 252, Taiwan
*
Authors to whom correspondence should be addressed.
These authors contributed equally to this work.
Diagnostics 2022, 12(4), 1002; https://doi.org/10.3390/diagnostics12041002
Submission received: 19 March 2022 / Revised: 12 April 2022 / Accepted: 14 April 2022 / Published: 15 April 2022
(This article belongs to the Special Issue Artificial Intelligence in Lung Diseases 2.0)

Abstract

:
Antifibrotic therapy has changed the treatment paradigm for idiopathic pulmonary fibrosis (IPF); however, a subset of patients still experienced rapid disease progression despite treatment. This study aimed to determine whether CT-based radiomic features can predict therapeutic response to antifibrotic agents. In this retrospective study, 35 patients with IPF on antifibrotic treatment enrolled from two centers were divided into training (n = 26) and external validation (n = 9) sets. Clinical and pulmonary function data were collected. The patients were categorized into stable disease (SD) and progressive disease (PD) groups based on functional or radiologic criteria. From pretreatment non-enhanced high-resolution CT (HRCT) images, twenty-six radiomic features were extracted through whole-lung texture analysis, and six parenchymal patterns were quantified using dedicated imaging platforms. The predictive factors for PD were determined via univariate and multivariate logistic regression analyses. In the training set (SD/PD: 12/14), univariate analysis identified eight radiomic features and ground-glass opacity percentage (GGO%) as potential predicators of PD. However, multivariate analysis found that the single independent predictor was the sum entropy (accuracy, 80.77%; AUC, 0.75). The combined sum entropy-GGO% model improved the predictive performance in the training set (accuracy, 88.46%; AUC, 0.77). The overall accuracy of the combined model in the validation set (SD/PD: 7/2) was 66.67%. Our preliminary results demonstrated that radiomic features based on pretreatment HRCT could predict the response of patients with IPF to antifibrotic treatment.

1. Introduction

Idiopathic pulmonary fibrosis (IPF) is a chronic, progressive, fibrosing lung disease of unknown etiology that primarily affects the elderly population [1]. It is characterized by an excessive fibrotic deposition in lung parenchyma by myofibroblasts that impedes gas exchange and ultimately leads to respiratory failure [1,2]. The overall prognosis is poor, and mortality usually occurs within 3–5 years after its diagnosis [3]. Fortunately, the availability of two antifibrotic agents, namely nintedanib and pirfenidone, has changed the landscape of care for IPF patients with improved clinical outcomes. Previous phase III trials (INPULSIS-1 and -2, ASCEND, and CAPCITY trials) and two recent network meta-analyses have consistently shown the remarkable effectiveness of both agents in slowing disease progression and reducing acute exacerbation (AE) compared to placebo [4,5,6,7,8]. Survival benefits have also been corroborated by real-world evidence from long-term cohort studies [9,10]. Nonetheless, the clinical course of IPF is variable and largely unpredictable, even after treatment [11]. A subset of patients treated with nintedanib (29.5%) or pirfenidone (12.6–21.0%) experienced an annual ≥10% decline in forced vital capacity (FVC), which is a strong predictor of mortality, upon completion of phase III trials [4,5,6,12]. Meanwhile, subgroup analyses found that the treatment effect is independent of baseline severity, sex, age, and smoking status [13,14]. As such, there is currently a knowledge gap regarding the determinants of therapeutic response.
Over the past decade, a number of prediction models for mortality has been developed and validated in IPF, such as the GAP (gender, age, and physiology variables) index and staging system [15]. Yet, no prediction model for therapeutic response has been proposed to date. To address this unmet need, high-resolution computed tomography (HRCT) may be a helpful tool as it plays a crucial role in diagnosing and monitoring IPF [1,16]. Furthermore, the extent of lung abnormalities on HRCT has been shown to correlate with lung function parameters and survival in these patients [17,18]. Preliminary studies revealed that certain CT findings, such as fibrosis and ground-glass opacities (GGOs), tend to increase more significantly in patients with unsatisfactory response [19,20]; in turn, these findings highlight the prognostic value of CT in determining their treatment response.
However, it can be challenging to evaluate and characterize the complex parenchymal abnormalities on HRCT with visual approach in the context of diffuse lung diseases including IPF. With the recent advances in computer-aided quantitative imaging techniques, radiologists can now minimize inter-and intraobserver variations in serial assessment, and improve their ability to capture the full extent of disease evolution [21,22]. Among these novel techniques, radiomics has emerged as a promising adjunct to clinical decision-making, which refers to the process of extracting quantitative imaging biomarkers from medical images through texture analysis using advanced computational algorithms [23,24]. These imaging biomarkers are also useful in predicting the clinical behavior and prognosis of various diseases [23,25,26,27,28], making them particularly important in the era of precision medicine. This study aimed to predict the treatment response of patients with IPF to antifibrotic agents using radiomics-based three-dimensional CT texture analysis.

2. Materials and Methods

2.1. Study Population

We retrospectively enrolled two separate cohorts of study patients from two tertiary care centers. The institutional review boards of both centers approved this study with waiver of informed consent. Cohort 1 consisted of 474 consecutive patients undergoing HRCT in Mackay memorial hospital, Taipei (institution 1), with suspected interstitial lung disease (ILD) from March 2010 to April 2021. Cohort 2 comprised 12 consecutive patients registered in the institutional pharmacy of Chi Mei hospital, Tainan (institution 2), for receiving antifibrotic agents from August 2020 to January 2022. The diagnosis of IPF in these patients was established through multidisciplinary discussion (MDD) as per the ATS/ERS/JRS/ALAT guidelines published in 2018 [16]. However, prior to the initiation of MDD team, diagnosis was made through consensus reached by pulmonologists and rheumatologists when definite or probable UIP pattern was evidenced on HRCT; for verification, each case was re-evaluated using the same criteria during data collection process. Only patients treated with nintedanib (150 mg twice daily) or pirfenidone (801 mg three times daily; maintenance dose) for at least three months were eligible for inclusion.
Exclusion criteria include: (1) no follow-up HRCT study or pulmonary function tests (PFTs) after treatment; (2) alternative diagnosis, or subtle fibrosis that was insufficient to establish the diagnosis of IPF; (3) presence of factors that could confound diagnosis and image analysis, including concomitant diseases (i.e., pneumonia, large pleural effusion, mass greater than 5 cm), or prior lung surgery.

2.2. Definition of Disease Progression

While IPF is, by definition, a progressive disease, a ≥10% decline in FVC over six to twelve months is generally considered significant and was used as a primary efficacy endpoint in drug trials because it portends an increased risk of mortality [12,29]. However, there is no consensus on defining disease progression for short-term evaluation. A recent meta-analysis identified an optimal cut-off of 5.7% over three-months to predict mortality with comparable performance to using the threshold of 10% at twelve-months [30]. With this background, we categorized patients into the progressive disease (PD) group if one of the following radiologic or functional criteria was met; if not, patients were categorized into the stable disease (SD) group. Functional criteria include: (1) FVC decline by ≥5.7% and ≥10% in patients who had been treated for 3–6 months and 6–12 months, respectively, upon outcome assessment; (2) AE reported by the pulmonologists; (3) all-cause mortality. Radiologic criteria include: (1) increased extent of fibrosis or decreased normally attenuated lung by ≥10%; (2) AE.

2.3. Acquisition Protocol for HRCT

In both institutions, all HRCT studies were carried out from the apex to the base of lung at end inspiration with the patients in supine position. Axial CT images (1.5-mm-thick) obtained within three months prior to the initiation of antifibrotic therapy were used for image analysis in this study. Detailed information of CT parameters is given in Supplementary Materials Table S1.

2.4. Semiquantitative Fibrosis Quantification (Fibrotic Score)

Two experienced thoracic radiologists (C.-H.Y. and W.-M.H., with 19 and 9 years of expertise, respectively), who were blinded to clinical information, independently reviewed the non-enhanced CT images and rated the fibrotic scores. The fibrotic score is a 6-slices method proposed by Fraser et al. [31], which was based on cross-sectional CT images obtained at six anatomical levels: (1) aortic arch; (2) 1 cm below the carina; (3) right pulmonary venous confluence; (4) halfway between the third and fifth sections; (5) 1 cm above the right hemi-diaphragm; and (6) 2 cm below the right hemi-diaphragm (Figure 1). The total fibrotic score was calculated as the average percentage of three fibrotic components (honeycombing, reticulation, and GGO with traction bronchiectasis) of the six slices.

2.5. Whole-Lung CT Texture Analysis

Three-dimensional volumetric texture analysis of the HRCT images in DICOM (Digital Imaging and Communications in Medicine) format was implemented on a dedicated artificial intelligence (AI) platform (QUIBIM Precision 2.8, QUIBIM SL, Valencia, Spain). The automatic activation of quantitative analysis was based on the AI platform engine rule configured to verify if image DICOM tags matched predefined configurations for CT scans. The task of automated whole-lung segmentation was powered by 2-class U-Net-based Convolutional Neural Network (CNN) algorithm with deep supervision layers. This post-processing technique objectively quantifies imaging features related to lung parenchymal heterogeneity and pathology. Major categories of features were then extracted, including: (1) first-order statistics, which are derived from histogram of the voxel intensities; and (2) gray-level co-occurrence matrix (GLCM) statistics, which are second-order statistical measure of the spatial relationship of pixels that take into account the direction, local neighborhood and magnitude of changes [24]. The radiomics workflow is presented in Figure 2.

2.6. Automated Quantification of Parenchymal Patterns

Quantitative analysis of CT parenchymal patterns was carried out using an AI software (AVIEW, Coreline Soft, Seoul, Korea) established on content-based image retrieval (CBIR). Lung segmentation and disease pattern classification was performed with the use of a two-dimensional (2D) U-Net architecture with deep CNN through process previously described [32]. Automated parenchymal pattern analysis was classified following DICOM images successfully received by the AI platform. Each slice of HRCT was classified into six parenchymal patterns (honeycombing, reticulation, GGO, emphysema, consolidation, and normal lung) by validated CBIR system in pixel-level (Figure 3). To quantify the spatial distribution of parenchymal patterns, whole lung volume was divided into 64 cuboid (4 × 4 × 4) and regional fraction of each disease pattern in each cuboid was calculated after pattern extraction.

2.7. Statistical Analysis

All statistical analyses were performed using SPSS (v22.0, IBM, Chicago, IL, USA). Continuous and categorial variables of the clinical data and baseline PTFs between the SD and PD groups were compared using Mann-Whitney U test and Fisher’s Exact test, respectively. Intraclass correlation coefficient (ICC) was used to examine inter-rater variability of fibrotic score measurements. Univariate and multivariate logistic regression analyses were used to determine the predictors of PD. Variables with p-value < 0.1 in the univariate analysis were included in the multivariate analysis [33], whereas only covariates with a two-tailed p-value < 0.05 in the multivariate analysis were regarded as the independent predictor. The predictive performance was validated by receiver operating characteristic (ROC) curve.

3. Results

3.1. Study Population

Training set—Of the initially screened 474 patients in cohort 1, 358 patients were excluded owing to the following reasons: alternative diagnosis (n = 190), presence of subtle fibrosis (n = 103), prespecified concomitant diseases (n = 60), and prior lung surgery (n = 5). The diagnosis of IPF was confirmed in 116 patients; among them, only 26 patients underwent antifibrotic treatment for at least three months and were finally included in the training set (SD/PD: 12/14).
Validation set—Among the 12 patients who underwent antifibrotic therapy in cohort 2, nine patients were included (SD/PD: 7/2) after the exclusion of three patients due to alternative diagnosis (n = 1; systemic sclerosis-related ILD), treatment duration shorter than three months (n = 1), and lung cancer greater than 5 cm (n = 1).
The selection process and clinical characteristics of the study patients are presented in Figure 2 and Table 1, respectively. There was no significant imbalance between the SD and PD groups in the two datasets based on baseline pulmonary function parameters, mean treatment duration, and smoking history. However, the GAP index and stage were significantly higher in the PD groups compared to the SD group in the training set.

3.2. Inter-Rater Reliability of the Fibrotic Score

The ICC was 0.91, and an ICC > 0.9 was considered excellent. In the training set, there was no significant difference in the mean fibrotic scores of the SD and PD groups (p < 0.1; Table 1).

3.3. Radiomic Feature Extraction and Selection

In total, 26 first-and second-order (GLCM) radiomic features were extracted (Table 2). In univariate logistic regression analysis, eight features showed significant differences (p < 0.1) between the SD and PD groups, which include entropy, difference entropy, sum entropy, kurtosis, skewness, inverse difference, and maximum probability. However, only the sum entropy remained statistically significant (OR = 0.01, p < 0.05) in multivariate logistic regression analysis (Table 3 and Figure 4).

3.4. Automated Quantification of Parenchymal Patterns

Of the six quantified radiologic patterns, only the GGO percentage (GGO%) was significantly different (p < 0.1) between the SD and PD groups based on univariate logistic regression analysis; however, this was not statistically significant (p < 0.05) in the multivariate logistic regression analysis (Table 3).

3.5. Development of Prediction Model with Performance Evaluation and Validation

In the training set, the AUCs of the models built with radiomic feature (sum entropy) and quantified parenchymal pattern (GGO%) were 0.75 (95% CI, 0.54–0.89; sensitivity, 100.00%; specificity, 58.33%; accuracy, 80.77) and 0.69 (95% CI, 0.48–0.85; sensitivity, 57.14%; specificity, 75.00%; accuracy, 65.38%), respectively. The combined sum entropy-GGO% model yielded an AUC of 0.77 (95% CI, 0.56–0.91; sensitivity, 100.00%; specificity, 75.00%; accuracy, 88.46%), which was higher than the values found when using any of the models alone (Table 4 and Figure 5). In contrast, the GAP model showed comparable values of AUC (0.73–0.77) but a lower accuracy (73.07%). In the validation set, although the accuracy of the combined model (66.67%) did not differ from that of the radiomic feature (sum entropy), it was higher than that of the GAP model (37.50%) (Table 5).

4. Discussion

Radiomics yields a plethora of objective quantitative features that entail prognostic implications. It is also widely recognized as a valuable tool to guide individualized patient care. In this study, univariate logistic regression analysis showed that the selected features—sum entropy, entropy, difference entropy, kurtosis, skewness, inverse difference, and maximum probability—enabled the prediction of PD after treatment, whereas multivariate logistic regression analysis showed that the single independent predictor of PD was the sum entropy, which demonstrated an accuracy of 80.77% (AUC, 0.75). On the other hand, while the quantified parenchymal pattern (GGO%) using AI software also predicts the PD based on univariate analysis, this was not observed in multivariate analysis. Regardless, this variable was not eliminated because our results were similar to those of Balestro et al., who suggested that GGO may be associated with unsatisfactory treatment response based on a visual semiquantitative analysis of temporal changes in CT after six months [19]. Interestingly, the combined sum entropy-GGO% model improved the overall predictive performance (accuracy, 88.46%; AUC, 0.77). Although originally developed for predicting mortality in patients with IPF, the GAP model was compared with our proposed models for the prediction of PD. The results showed that both the sum entropy and combined model outperformed the GAP model by achieving higher accuracy (80.77–88.46% vs. 73.07%), albeit with comparable AUCs. Similarly, both the sum entropy and combined model also showed a higher accuracy of 66.67% as compared to the GAP model (37.50%) on external validation. To the best of our knowledge, this study across two cohorts of IPF patients first demonstrated that CT-based radiomics can be used to predict their response to antifibrotic agents. Furthermore, automated whole-lung segmentation obviates subjective variations in manual ROI selection; hence, it offers better reproducibility.
Previous studies showed that radiomics enhances the diagnosis and severity staging of IPF [25,26], and predicts the functional decline and long-term survival of patients [27,28]. Notably, our previous work demonstrated that a CT-based radiomics model (specifically, energy and kurtosis) is capable of predicting lung cancer development in patients with IPF [34]. In the present study, certain radiomic features that distinguish between the PD and SD groups are in line with prior studies, including entropy-related features and kurtosis. Entropy is a measure of the inherent randomness in gray level intensities whereas sum entropy represents the sum of intensity differences of the neighborhood [24]; these two statistics have recently garnered much attention in oncology because they reflect tumor heterogeneity and are predictive of tumor stage and prognosis [35]. IPF is considered a cancer-like disease because it is marked by an uncontrolled fibroproliferation akin to neoplasia, and both diseases share several common genetic alterations and molecular pathways in pathogenesis [36]. We speculate that the higher entropy-related features observed in the PD group might reflect greater distortion of lung parenchyma caused by myofibroblasts in a manner similar to tumor cells. Kurtosis refers to a first-order statistic that measures the peakedness of intensity distribution. A low kurtosis, which represents an intensity distribution toward the two ends of high (fibrosis) and low (destructive lucent center of honeycombing) in the context of IPF [22], was shown to predict the disease progression and short-term mortality [27].
The past decade of research has focused largely on the impact of antifibrotic therapy on lung function; thus, little is known about the consequent changes in imaging or histopathology during follow-up after treatment. In a phase IIIb placebo-controlled trial, Lancaster et al., demonstrated that the increment in reticulation was smaller in the nintedanib group compared to that in the placebo group after six months using another quantitative texture analysis tool (computer-aided lung informatics for pathology evaluation and rating, CALIPER) [37]. Zhang et al., studied 28 patients with IPF undergoing lung transplantation and suggested that treated patients tend to have less histologic evidence of acute lung injury [38]. In a recent histologic analysis of 40 nintedanib-treated patients, Nemoto et al., found that edematous changes in the interlobular septum may be associated with poor response [2]. On the other hand, Jacob et al., reported that among CALIPER-derived parameters, the pulmonary vessel-related structure (VRS) score obtained from the upper-and mid-lung zones was the strongest predictor of 10% FVC decline or death at 12-months after the treatment [39].
An early prediction of therapeutic efficacy is beneficial in tailoring treatments for patients with IPF. Since the most common adverse effects of nintedanib (diarrhea, 62%) and pirfenidone (nausea, 36.8%) tend to occur in the first three months [40], this could result in an early withdrawal from therapy (4.4% of the cases in INPULSIS trials) [4]. Worst of all, treatment discontinuation is associated with a greater risk of death and potentially leads to an accelerated disease progression [40,41]. Our prediction model may also help strengthen the medication adherence particularly in patients who are likely to have favorable outcomes, and optimize the risk stratification of vulnerable subjects with unsatisfactory response who may require additional therapy.
This study has some limitations. First, given the rarity of IPF in Asian populations and the common, yet controversial, practice of withholding antifibrotic treatment from functionally stable subjects [1,9], the patient number is relatively small even though the training set includes a cohort of patients with IPF over a decade. The small sample size may render the prediction model prone to bias. In light of this, external validation was implemented to assess the reproducibility and generalizability of the model, and it showed an acceptable accuracy in predicting the PD. Second, selection bias could occur because the diagnosis of IPF was not established through MDD for all patients. Third, although a previous study suggested that VRS predicts PD [39], this study was unable to compare VRS with GGO since the software we used could not quantify this parameter. Finally, the length of antifibrotic therapy until outcome assessment was not uniform among the patients (ranging from 3–12 months); nonetheless, the definition of disease progression was modified based on the current evidence [12,29,30], and a significant imbalance between the SD and PD groups was not found in both the training and validation sets.

5. Conclusions

In this pilot study, a pretreatment HRCT-based radiomics model was developed to predict the response to antifibrotic treatment in patients with IPF. The preliminary results demonstrated that the sum entropy showed a high accuracy in predicting the PD and that the combined sum entropy-GGO% model further improved the overall predictive performance. These findings provide potential clinical benefits as they may allow the clinicians to tailor treatment strategy to individual patients. Future studies on larger-scale datasets are needed to corroborate our conclusions and to examine the robustness of the model.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/diagnostics12041002/s1, Table S1: Acquisition Protocols and Technical Parameters for HRCT Scans in the Two Institutions.

Author Contributions

Conceptualization, C.-C.Y. and W.-M.H.; methodology, C.-C.Y., C.-H.L. and W.-M.H.; software, C.-H.Y., C.-H.L. and W.-M.H.; validation, C.-H.Y., C.-Y.C. and Y.-T.K.; formal analysis, C.-C.Y. and W.-M.H.; investigation, C.-H.Y., C.-C.Y. and W.-M.H.; resources, C.-H.Y., C.-C.K. and W.-J.W.; data curation, W.-M.H.; writing—original draft preparation, C.-C.Y.; writing—review and editing, C.-C.Y., C.-Y.C., Y.-T.K. and W.-M.H.; visualization, C.-C.K., W.-J.W., C.-Y.C. and Y.-T.K.; supervision, C.-H.Y. and W.-M.H.; project administration, C.-H.Y. and W.-M.H.; funding acquisition, none. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

The study was conducted in accordance with the Declaration of Helsinki, and approved by the Institutional Review Boards of Mackay Memorial Hospital, Taipei branch, Taipei, Taiwan (protocol code: 21MMHIS180e; approved on 18 July 2021), and Chi-Mei Hospital, Tainan, Taiwan (protocol code: 11102-004; approved on 23 February 2022).

Informed Consent Statement

Patient informed consent was waived because the studied images and clinical information were obtained from de-identified data.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Lederer, D.J.; Martinez, F.J. Idiopathic Pulmonary Fibrosis. N. Engl. J. Med. 2018, 378, 1811–1823. [Google Scholar] [CrossRef] [PubMed]
  2. Nemoto, M.; Zaizen, Y.; Kataoka, K.; Kuroda, K.; Tabata, K.; Bychkov, A.; Sumikawa, H.; Johkoh, T.; Aoshima, M.; Kondoh, Y.; et al. Histologic factors associated with nintedanib efficacy in patients with idiopathic pulmonary fibrosis. PLoS ONE 2021, 16, e0245147. [Google Scholar] [CrossRef]
  3. Fernandez Fabrellas, E.; Peris Sanchez, R.; Sabater Abad, C.; Juan Samper, G. Prognosis and Follow-Up of Idiopathic Pulmonary Fibrosis. Med. Sci. 2018, 6, 51. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  4. Richeldi, L.; du Bois, R.M.; Raghu, G.; Azuma, A.; Brown, K.K.; Costabel, U.; Cottin, V.; Flaherty, K.R.; Hansell, D.M.; Inoue, Y.; et al. Efficacy and safety of nintedanib in idiopathic pulmonary fibrosis. N. Engl. J. Med. 2014, 370, 2071–2082. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  5. King, T.E., Jr.; Bradford, W.Z.; Castro-Bernardini, S.; Fagan, E.A.; Glaspole, I.; Glassberg, M.K.; Gorina, E.; Hopkins, P.M.; Kardatzke, D.; Lancaster, L.; et al. A phase 3 trial of pirfenidone in patients with idiopathic pulmonary fibrosis. N. Engl. J. Med. 2014, 370, 2083–2092. [Google Scholar] [CrossRef] [Green Version]
  6. Noble, P.W.; Albera, C.; Bradford, W.Z.; Costabel, U.; Glassberg, M.K.; Kardatzke, D.; King, T.E., Jr.; Lancaster, L.; Sahn, S.A.; Szwarcberg, J.; et al. Pirfenidone in patients with idiopathic pulmonary fibrosis (CAPACITY): Two randomised trials. Lancet 2011, 377, 1760–1769. [Google Scholar] [CrossRef]
  7. Loveman, E.; Copley, V.R.; Scott, D.A.; Colquitt, J.L.; Clegg, A.J.; O’Reilly, K.M. Comparing new treatments for idiopathic pulmonary fibrosis--a network meta-analysis. BMC Pulm. Med. 2015, 15, 37. [Google Scholar] [CrossRef] [Green Version]
  8. Skandamis, A.; Kani, C.; Markantonis, S.L.; Souliotis, K. Systematic review and network meta-analysis of approved medicines for the treatment of idiopathic pulmonary fibrosis. J. Drug Assess. 2019, 8, 55–61. [Google Scholar] [CrossRef] [Green Version]
  9. Behr, J.; Prasse, A.; Wirtz, H.; Koschel, D.; Pittrow, D.; Held, M.; Klotsche, J.; Andreas, S.; Claussen, M.; Grohe, C.; et al. Survival and course of lung function in the presence or absence of antifibrotic treatment in patients with idiopathic pulmonary fibrosis: Long-term results of the INSIGHTS-IPF registry. Eur. Respir. J. 2020, 56, 1902279. [Google Scholar] [CrossRef]
  10. Kang, J.; Han, M.; Song, J.W. Antifibrotic treatment improves clinical outcomes in patients with idiopathic pulmonary fibrosis: A propensity score matching analysis. Sci. Rep. 2020, 10, 15620. [Google Scholar] [CrossRef]
  11. Nathan, S.D.; Albera, C.; Bradford, W.Z.; Costabel, U.; du Bois, R.M.; Fagan, E.A.; Fishman, R.S.; Glaspole, I.; Glassberg, M.K.; Glasscock, K.F.; et al. Effect of continued treatment with pirfenidone following clinically meaningful declines in forced vital capacity: Analysis of data from three phase 3 trials in patients with idiopathic pulmonary fibrosis. Thorax 2016, 71, 429–435. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  12. Paterniti, M.O.; Bi, Y.; Rekic, D.; Wang, Y.; Karimi-Shah, B.A.; Chowdhury, B.A. Acute Exacerbation and Decline in Forced Vital Capacity Are Associated with Increased Mortality in Idiopathic Pulmonary Fibrosis. Ann. Am. Thorac. Soc. 2017, 14, 1395–1402. [Google Scholar] [CrossRef] [PubMed]
  13. Noble, P.W.; Albera, C.; Bradford, W.Z.; Costabel, U.; du Bois, R.M.; Fagan, E.A.; Fishman, R.S.; Glaspole, I.; Glassberg, M.K.; Lancaster, L.; et al. Pirfenidone for idiopathic pulmonary fibrosis: Analysis of pooled data from three multinational phase 3 trials. Eur. Respir. J. 2016, 47, 243–253. [Google Scholar] [CrossRef] [PubMed]
  14. Costabel, U.; Inoue, Y.; Richeldi, L.; Collard, H.R.; Tschoepe, I.; Stowasser, S.; Azuma, A. Efficacy of Nintedanib in Idiopathic Pulmonary Fibrosis across Prespecified Subgroups in INPULSIS. Am. J. Respir. Crit. Care Med. 2016, 193, 178–185. [Google Scholar] [CrossRef]
  15. Ley, B.; Ryerson, C.J.; Vittinghoff, E.; Ryu, J.H.; Tomassetti, S.; Lee, J.S.; Poletti, V.; Buccioli, M.; Elicker, B.M.; Jones, K.D.; et al. A multidimensional index and staging system for idiopathic pulmonary fibrosis. Ann. Intern. Med. 2012, 156, 684–691. [Google Scholar] [CrossRef]
  16. Raghu, G.; Remy-Jardin, M.; Myers, J.L.; Richeldi, L.; Ryerson, C.J.; Lederer, D.J.; Behr, J.; Cottin, V.; Danoff, S.K.; Morell, F.; et al. Diagnosis of Idiopathic Pulmonary Fibrosis. An Official ATS/ERS/JRS/ALAT Clinical Practice Guideline. Am. J. Respir. Crit. Care Med. 2018, 198, e44–e68. [Google Scholar] [CrossRef]
  17. Lang, D.; Akbari, K.; Walcherberger, S.; Hergan, B.; Horner, A.; Hepp, M.; Kaiser, B.; Pieringer, H.; Lamprecht, B. Computed tomography findings as determinants of pulmonary function tests in fibrotic interstitial lung diseases-Network-analyses and multivariate models. Chron. Respir. Dis. 2020, 17, 1479973120967025. [Google Scholar] [CrossRef]
  18. Jacob, J.; Bartholmai, B.J.; Rajagopalan, S.; Kokosi, M.; Nair, A.; Karwoski, R.; Raghunath, S.M.; Walsh, S.L.; Wells, A.U.; Hansell, D.M. Automated Quantitative Computed Tomography Versus Visual Computed Tomography Scoring in Idiopathic Pulmonary Fibrosis: Validation Against Pulmonary Function. J. Thorac. Imaging 2016, 31, 304–311. [Google Scholar] [CrossRef] [Green Version]
  19. Balestro, E.; Cocconcelli, E.; Giraudo, C.; Polverosi, R.; Biondini, D.; Lacedonia, D.; Bazzan, E.; Mazzai, L.; Rizzon, G.; Lococo, S.; et al. High-Resolution CT Change over Time in Patients with Idiopathic Pulmonary Fibrosis on Antifibrotic Treatment. J. Clin. Med. 2019, 8, 1469. [Google Scholar] [CrossRef] [Green Version]
  20. Sugino, K.; Shimizu, H.; Nakamura, Y.; Isshiki, T.; Matsumoto, K.; Homma, S. Clinico-radiological features and efficacy of anti-fibrotic agents in atypical idiopathic pulmonary fibrosis. J. Thorac. Dis. 2018, 10, 899–908. [Google Scholar] [CrossRef] [Green Version]
  21. Bartholmai, B.J.; Raghunath, S.; Karwoski, R.A.; Moua, T.; Rajagopalan, S.; Maldonado, F.; Decker, P.A.; Robb, R.A. Quantitative computed tomography imaging of interstitial lung diseases. J. Thorac. Imaging 2013, 28, 298–307. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  22. Chen, A.; Karwoski, R.A.; Gierada, D.S.; Bartholmai, B.J.; Koo, C.W. Quantitative CT Analysis of Diffuse Lung Disease. Radiographics 2020, 40, 28–43. [Google Scholar] [CrossRef] [PubMed]
  23. Frix, A.N.; Cousin, F.; Refaee, T.; Bottari, F.; Vaidyanathan, A.; Desir, C.; Vos, W.; Walsh, S.; Occhipinti, M.; Lovinfosse, P.; et al. Radiomics in Lung Diseases Imaging: State-of-the-Art for Clinicians. J. Pers. Med. 2021, 11, 602. [Google Scholar] [CrossRef] [PubMed]
  24. Parekh, V.; Jacobs, M.A. Radiomics: A new application from established techniques. Expert Rev. Precis. Med. Drug Dev. 2016, 1, 207–226. [Google Scholar] [CrossRef] [Green Version]
  25. Stefano, A.; Gioe, M.; Russo, G.; Palmucci, S.; Torrisi, S.E.; Bignardi, S.; Basile, A.; Comelli, A.; Benfante, V.; Sambataro, G.; et al. Performance of Radiomics Features in the Quantification of Idiopathic Pulmonary Fibrosis from HRCT. Diagnostics 2020, 10, 306. [Google Scholar] [CrossRef]
  26. Kim, H.J.; Brown, M.S.; Chong, D.; Gjertson, D.W.; Lu, P.; Kim, H.J.; Coy, H.; Goldin, J.G. Comparison of the quantitative CT imaging biomarkers of idiopathic pulmonary fibrosis at baseline and early change with an interval of 7 months. Acad. Radiol. 2015, 22, 70–80. [Google Scholar] [CrossRef]
  27. Best, A.C.; Meng, J.; Lynch, A.M.; Bozic, C.M.; Miller, D.; Grunwald, G.K.; Lynch, D.A. Idiopathic pulmonary fibrosis: Physiologic tests, quantitative CT indexes, and CT visual scores as predictors of mortality. Radiology 2008, 246, 935–940. [Google Scholar] [CrossRef]
  28. Budzikowski, J.D.; Foy, J.J.; Rashid, A.A.; Chung, J.H.; Noth, I.; Armato, S.G., 3rd. Radiomics-based assessment of idiopathic pulmonary fibrosis is associated with genetic mutations and patient survival. J. Med. Imaging 2021, 8, 031903. [Google Scholar] [CrossRef]
  29. Reichmann, W.M.; Yu, Y.F.; Macaulay, D.; Wu, E.Q.; Nathan, S.D. Change in forced vital capacity and associated subsequent outcomes in patients with newly diagnosed idiopathic pulmonary fibrosis. BMC Pulm. Med. 2015, 15, 167. [Google Scholar] [CrossRef] [Green Version]
  30. Khan, F.A.; Stewart, I.; Moss, S.; Fabbri, L.; Robinson, K.A.; Johnson, S.R.; Jenkins, R.G. Three-Month FVC Change: A Trial Endpoint for IPF Based on Individual Participant Data Meta-Analysis. Am. J. Respir. Crit. Care Med. 2022; ahead of print. [Google Scholar] [CrossRef]
  31. Fraser, E.; St Noble, V.; Hoyles, R.K.; Benamore, R.; Ho, L.P. Readily accessible CT scoring method to quantify fibrosis in IPF. BMJ Open Respir. Res. 2020, 7, e000584. [Google Scholar] [CrossRef] [PubMed]
  32. Choe, J.; Hwang, H.J.; Seo, J.B.; Lee, S.M.; Yun, J.; Kim, M.J.; Jeong, J.; Lee, Y.; Jin, K.; Park, R.; et al. Content-based Image Retrieval by Using Deep Learning for Interstitial Lung Disease Diagnosis with Chest CT. Radiology 2022, 302, 187–197. [Google Scholar] [CrossRef] [PubMed]
  33. Bursac, Z.; Gauss, C.H.; Williams, D.K.; Hosmer, D.W. Purposeful selection of variables in logistic regression. Source Code Biol. Med. 2008, 3, 17. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  34. Liang, C.H.; Liu, Y.C.; Wan, Y.L.; Yun, C.H.; Wu, W.J.; Lopez-Gonzalez, R.; Huang, W.M. Quantification of Cancer-Developing Idiopathic Pulmonary Fibrosis Using Whole-Lung Texture Analysis of HRCT Images. Cancers 2021, 13, 5600. [Google Scholar] [CrossRef]
  35. Dercle, L.; Ammari, S.; Bateson, M.; Durand, P.B.; Haspinger, E.; Massard, C.; Jaudet, C.; Varga, A.; Deutsch, E.; Soria, J.C.; et al. Limits of radiomic-based entropy as a surrogate of tumor heterogeneity: ROI-area, acquisition protocol and tissue site exert substantial influence. Sci. Rep. 2017, 7, 7952. [Google Scholar] [CrossRef]
  36. Vancheri, C. Common pathways in idiopathic pulmonary fibrosis and cancer. Eur. Respir. Rev. 2013, 22, 265–272. [Google Scholar] [CrossRef] [Green Version]
  37. Lancaster, L.; Goldin, J.; Trampisch, M.; Kim, G.H.; Ilowite, J.; Homik, L.; Hotchkin, D.L.; Kaye, M.; Ryerson, C.J.; Mogulkoc, N.; et al. Effects of Nintedanib on Quantitative Lung Fibrosis Score in Idiopathic Pulmonary Fibrosis. Open Respir. Med. J. 2020, 14, 22–31. [Google Scholar] [CrossRef]
  38. Zhang, Y.; Jones, K.D.; Achtar-Zadeh, N.; Green, G.; Kukreja, J.; Xu, B.; Wolters, P.J. Histopathological and molecular analysis of idiopathic pulmonary fibrosis lungs from patients treated with pirfenidone or nintedanib. Histopathology 2019, 74, 341–349. [Google Scholar] [CrossRef]
  39. Jacob, J.; Bartholmai, B.J.; Rajagopalan, S.; van Moorsel, C.H.M.; van Es, H.W.; van Beek, F.T.; Struik, M.H.L.; Kokosi, M.; Egashira, R.; Brun, A.L.; et al. Predicting Outcomes in Idiopathic Pulmonary Fibrosis Using Automated Computed Tomographic Analysis. Am. J. Respir. Crit. Care Med. 2018, 198, 767–776. [Google Scholar] [CrossRef]
  40. Wright, W.A.; Crowley, L.E.; Parekh, D.; Crawshaw, A.; Dosanjh, D.P.; Nightingale, P.; Thickett, D.R. Real-world retrospective observational study exploring the effectiveness and safety of antifibrotics in idiopathic pulmonary fibrosis. BMJ Open Respir. Res. 2021, 8, e000782. [Google Scholar] [CrossRef]
  41. Okamori, S.; Asakura, T.; Masuzawa, K.; Yasuda, H.; Kamata, H.; Ishii, M.; Betsuyaku, T. Suspected accelerated disease progression after discontinuation of nintedanib in patients with idiopathic pulmonary fibrosis: Two case reports. Medicine 2017, 96, e9081. [Google Scholar] [CrossRef] [PubMed]
Figure 1. Axial thin-section HRCT images of a 66-year-old male patient diagnosed with IPF through MDD with a (definite) UIP pattern. The total fibrotic score was calculated as the average percentage of the three fibrotic components (honeycombing, reticulation, and GGO with traction bronchiectasis) of the six slices.
Figure 1. Axial thin-section HRCT images of a 66-year-old male patient diagnosed with IPF through MDD with a (definite) UIP pattern. The total fibrotic score was calculated as the average percentage of the three fibrotic components (honeycombing, reticulation, and GGO with traction bronchiectasis) of the six slices.
Diagnostics 12 01002 g001
Figure 2. Flow diagrams of patient selection process and radiomics workflow.
Figure 2. Flow diagrams of patient selection process and radiomics workflow.
Diagnostics 12 01002 g002
Figure 3. Automated quantification of CT parenchymal patterns in a 74-year-old male patient with IPF. Axial thin-section HRCT images obtained at the lung base, without (A,D) and with (B,E) overlay of color maps depicting the distribution and extent of different parenchymal abnormalities. The glyph and chart (C) summarize the percentage of the six parenchymal patterns.
Figure 3. Automated quantification of CT parenchymal patterns in a 74-year-old male patient with IPF. Axial thin-section HRCT images obtained at the lung base, without (A,D) and with (B,E) overlay of color maps depicting the distribution and extent of different parenchymal abnormalities. The glyph and chart (C) summarize the percentage of the six parenchymal patterns.
Diagnostics 12 01002 g003
Figure 4. Comparison of the selected radiomic features and GGO% in the SD and PD groups.
Figure 4. Comparison of the selected radiomic features and GGO% in the SD and PD groups.
Diagnostics 12 01002 g004
Figure 5. ROC curves of the prediction models for progressive disease. (A) The radiomic feature (sum entropy) demonstrated an acceptable AUC of 0.75 which was higher than that of the quantified parenchymal pattern (GGO%; AUC, 0.69), but was not superior to that of (B) the GAP index and stage (AUC, 0.77 and 0.73, respectively). The combined sum entropy-GGO% model improved the predictive performance with an AUC of 0.77.
Figure 5. ROC curves of the prediction models for progressive disease. (A) The radiomic feature (sum entropy) demonstrated an acceptable AUC of 0.75 which was higher than that of the quantified parenchymal pattern (GGO%; AUC, 0.69), but was not superior to that of (B) the GAP index and stage (AUC, 0.77 and 0.73, respectively). The combined sum entropy-GGO% model improved the predictive performance with an AUC of 0.77.
Diagnostics 12 01002 g005
Table 1. Comparison of Patient Characteristics in the Training and Validation Sets.
Table 1. Comparison of Patient Characteristics in the Training and Validation Sets.
CharacteristicsTraining SetValidation Set
SD (n = 12)PD (n = 14)p-ValueSD (n = 7)PD (n = 2)p-Value
Age72.58 ± 10.2176.14 ± 7.880.32676.71 ± 7.9969.00 ± 0.000.018 *
Sex (M)7 (58%)12 (86%)0.1396 (86%)00.043 *
Smoking5 (42%)7 (50%)0.6864 (57%)00.030 *
PFTs
 FVC (%)82.17 ± 19.6982.09 ± 22.800.99391.57 ± 26.6165.000.386
 FEV1 (%)90.42 ± 20.3779.90 ± 30.580.32194.43 ± 20.3573.000.363
 DLCO (%)69.36 ± 19.7162.45 ± 17.270.39263.33 ± 33.6354.000.807
 TLC (%)75.50 ± 8.9472.07 ± 12.030.42589.29 ± 20.2360.000.224
GAP index3.00 ± 1.414.43 ± 1.600.025 *4.00 ± 1.264.50 ± 0.710.625
GAP stage1.42 ± 0.672.00 ± 0.680.038 *2.00 ± 0.632.00 ± 0.001.000
Treatment duration (weeks)48.95 ± 17.6035.96 ±18.640.08236.96 ± 14.4816.50 ± 2.120.099
Fibrotic score19.89 ± 10.5925.50 ± 11.080.20021.90 ± 9.0626.72 ± 23.650.639
Lung volume
(mL)
3242.25 ± 666.333057.29 ± 849.690.5463308.96 ± 1362.993553.67 ± 49.140.816
Values are given as mean ± standard deviation. * Indicates statistical significance. Abbreviation: DLCO, diffusing capacity of the lung for carbon monoxide; FEV1, forced expiratory volume in 1 s; FVC, forced vital capacity; GAP, gender-age-physiology index and staging system; PD, progressive disease group; PFTs, pulmonary function tests; SD, stable disease group; TLC, total lung capacity.
Table 2. Comparison of Radiomic Features of Stable Disease (SD) and Progressive Disease (PD) Groups in the Training Set.
Table 2. Comparison of Radiomic Features of Stable Disease (SD) and Progressive Disease (PD) Groups in the Training Set.
MetricsFeaturesSDPDp-Value
First orderEnergy2.21 × 10122.03 × 10120.385
Entropy8.238.800.030 *
Kurtosis21.2815.770.028 *
Skewness5.004.000.025 *
Mean−411.04−357.930.415
Standard deviation359.54382.530.169
Median−530.11−479.290.409
10th percentile−678.26−675.090.951
90th percentile13.85137.230.213
Second order
(GLCM)
Autocorrelation444.15556.280.075
Cluster prominence627,504.74590,318.290.651
Cluster shade10,774.0710,488.910.820
Contrast50.0158.240.101
Correlation1.451.440.788
Difference entropy6.196.550.038 *
Difference variance30.5533.300.247
Dissimilarity5.956.790.048 *
Inverse difference0.910.830.035 *
IMC1−0.28−0.280.837
IMC21.551.560.642
Maximum probability0.110.070.028 *
Sum average54.1361.100.059
Sum entropy9.8710.450.036 *
Sum of squares93.72107.590.160
Sum variance324.86372.130.190
* Indicates statistical significance. Abbreviation: GLCM, grey level co-occurrence matrix; IMC, information measure of correlation.
Table 3. Univariate and Multivariate Logistic Regression Analyses to Differentiate the Progressive Disease (PD) Group from the Stable Disease (SD) Group in the Training Set.
Table 3. Univariate and Multivariate Logistic Regression Analyses to Differentiate the Progressive Disease (PD) Group from the Stable Disease (SD) Group in the Training Set.
CharacteristicsUnivariate Regression AnalysisMultivariate Regression Analysis
OR95% CIp-ValueOR95% CIp-Value
Entropy 4.371.05–18.300.04 *3.42 × 10750.02–5.94 × 101530.06
Difference entropy8.150.99–66.940.05 *1.67 × 10160.01–4.14 × 10400.19
Sum entropy3.931.01–15.320.05 *0.010.01–0.220.04 *
Kurtosis0.850.73–1.010.04 *0.900.25–3.250.87
Skewness0.400.16–0.950.04 *0.010.01–63.140.29
Dissimilarity 2.300.97–5.480.06 *0.010.01–525.210.16
Inverse difference0.030.01–0.950.05 *1.40 × 10610.35–5.58 × 101220.05
Maximum probability0.020.01–0.470.04 *0.010.01–2.21 × 10420.58
GGO%1.040.97–1.090.09 *1.100.99–1.220.07
Honeycombing%0.750.21–2.730.67
Reticulation%1.060.84–1.340.62
Emphysema%1.040.89–1.130.92
Age 1.080.98–1.190.13
Sex 4.290.65–28.260.13
Smoking 1.400.30–6.620.67
* Indicates statistical significance.
Table 4. Prediction Performance of Different Models in the Training Set.
Table 4. Prediction Performance of Different Models in the Training Set.
CharacteristicsCut-OffAUCSensitivity (%)Specificity (%)Accuracy (%)
Entropy>7.95 0.76 [0.55–0.90]100.058.3380.77
Difference entropy>6.390.74 [0.54–0.89]78.5775.00 76.92
Sum entropy>9.600.75 [0.54–0.89]100.0058.3380.77
Kurtosis ≤19.450.73 [0.52–0.88]85.7166.6776.92
Skewness ≤4.980.76 [0.56–0.91]92.8666.6780.77
Dissimilarity >5.620.76 [0.55–0.90]92.8658.3376.92
Inverse difference≤0.900.77 [0.56–0.91]78.5775.0076.92
Maximum probability≤0.090.74 [0.54–0.89]85.7166.6776.92
GGO%>16.000.69 [0.48–0.85]57.1475.0065.38
Sum entropy + GGO% 0.77 [0.56–0.91]100.0075.0088.46
GAP index>30.77 [0.56–0.91]78.5766.6773.07
GAP stage>10.73 [0.52–0.88]78.5766.6773.07
Table 5. Prediction Accuracy of Different Models in the Validation Set Using the Cut-Off Values Derived from the Training Set.
Table 5. Prediction Accuracy of Different Models in the Validation Set Using the Cut-Off Values Derived from the Training Set.
CharacteristicsSensitivity (%)Specificity (%)Accuracy (%)
Entropy50.0071.4366.67
Difference entropy50.0085.7177.78
Sum entropy50.0071.4366.67
Kurtosis50.0042.8644.44
Skewness50.0028.5733.33
Dissimilarity50.0085.7177.78
Inverse difference50.0071.4366.67
Maximum probability50.0071.4366.67
GGO%50.0050.0050.00
Sum entropy + GGO%50.0074.4366.67
GAP index100.0016.6737.50
GAP stage100.0016.6737.50
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Yang, C.-C.; Chen, C.-Y.; Kuo, Y.-T.; Ko, C.-C.; Wu, W.-J.; Liang, C.-H.; Yun, C.-H.; Huang, W.-M. Radiomics for the Prediction of Response to Antifibrotic Treatment in Patients with Idiopathic Pulmonary Fibrosis: A Pilot Study. Diagnostics 2022, 12, 1002. https://doi.org/10.3390/diagnostics12041002

AMA Style

Yang C-C, Chen C-Y, Kuo Y-T, Ko C-C, Wu W-J, Liang C-H, Yun C-H, Huang W-M. Radiomics for the Prediction of Response to Antifibrotic Treatment in Patients with Idiopathic Pulmonary Fibrosis: A Pilot Study. Diagnostics. 2022; 12(4):1002. https://doi.org/10.3390/diagnostics12041002

Chicago/Turabian Style

Yang, Cheng-Chun, Chin-Yu Chen, Yu-Ting Kuo, Ching-Chung Ko, Wen-Jui Wu, Chia-Hao Liang, Chun-Ho Yun, and Wei-Ming Huang. 2022. "Radiomics for the Prediction of Response to Antifibrotic Treatment in Patients with Idiopathic Pulmonary Fibrosis: A Pilot Study" Diagnostics 12, no. 4: 1002. https://doi.org/10.3390/diagnostics12041002

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop