Next Article in Journal
Management of Skin Toxicities in Cancer Treatment: An Australian/New Zealand Perspective
Next Article in Special Issue
Impact of Preprocessing Parameters in Medical Imaging-Based Radiomic Studies: A Systematic Review
Previous Article in Journal
Vaginal Intraepithelial Neoplasia (VaIN) after Hysterectomy Is Strongly Associated with Persistent HR-HPV Infection
Previous Article in Special Issue
Radiogenomics: Contemporary Applications in the Management of Rectal Cancer
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Cancers
Volume 16
Issue 14
10.3390/cancers16142525
cancers-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

18F-Fluorodeoxyglucose Positron Emission Tomography-Based Risk Score Model for Prediction of Five-Year Survival Outcome after Curative Resection of Non-Small-Cell Lung Cancer

by
Chae Hong Lim
1,
Sang-Won Um
2,
Hong Kwan Kim
3,
Yong Soo Choi
3,
Hong Ryul Pyo
4,
Myung-Ju Ahn
5 and
Joon Young Choi
6,*
1
Department of Nuclear Medicine, Soonchunhyang University College of Medicine, Seoul 04401, Republic of Korea
2
Division of Pulmonary and Critical Care Medicine, Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul 03181, Republic of Korea
3
Department of Thoracic and Cardiovascular Surgery, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul 03181, Republic of Korea
4
Department of Radiation Oncology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul 03181, Republic of Korea
5
Division of Hematology-Oncology, Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul 03181, Republic of Korea
6
Department of Nuclear Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul 03181, Republic of Korea
*
Author to whom correspondence should be addressed.
Cancers 2024, 16(14), 2525; https://doi.org/10.3390/cancers16142525
Submission received: 30 May 2024 / Revised: 9 July 2024 / Accepted: 10 July 2024 / Published: 12 July 2024
(This article belongs to the Special Issue Radiomics and Imaging in Cancer Analysis)
Browse Figures
Versions Notes

Abstract

:

Simple Summary

The 18F-FDG PET parameters reflecting the intensity and distribution of glucose uptake by the tumor are associated with prognosis in non-small-cell lung cancer (NSCLC) patients. We developed and evaluated an imaging-based model utilizing these 18F-FDG PET-derived features for predicting the five-year survival in NSCLC patients after curative surgery. The PET-based risk score constructed using the LASSO logistic method outperformed the predictive performances of individual 18F-FDG PET parameters. The PET-based risk score was an independent prognostic factor for clinical variables. Additionally, it demonstrated better predictive performance when combined with clinical variables. The FDG PET-based imaging model could aid in risk stratification for personalized adjuvant treatment and follow-up management of NSCLC patients after surgery.

Abstract

The aim of our retrospective study is to develop and assess an imaging-based model utilizing 18F-FDG PET parameters for predicting the five-year survival in non-small-cell lung cancer (NSCLC) patients after curative surgery. A total of 361 NSCLC patients who underwent curative surgery were assigned to the training set (n = 253) and the test set (n = 108). The LASSO regression model was used to construct a PET-based risk score for predicting five-year survival. A hybrid model that combined the PET-based risk score and clinical variables was developed using multivariate logistic regression analysis. The predictive performance was determined by the area under the curve (AUC). The individual features with the best predictive performances were co-occurrence_contrast (AUC = 0.675) and SUL peak (AUC = 0.671). The PET-based risk score was identified as an independent predictor after adjusting for clinical variables (OR 5.231, 95% CI 1.987–6.932; p = 0.009). The hybrid model, which integrated clinical variables, significantly outperformed the PET-based risk score alone in predictive accuracy (AUC = 0.771 vs. 0.696, p = 0.022), a finding that was consistent in the test set. The PET-based risk score, especially when integrated with clinical variables, demonstrates good predictive ability for five-year survival in NSCLC patients following curative surgery.

1. Introduction

Lung cancer is a leading cause of death among cancer patients worldwide, with non-small-cell lung cancer (NSCLC) comprising 85% of these cases [1,2]. Despite recent advancements in treatment modalities, therapeutic strategies based on cancer staging have limitations in providing the most effective treatments for individual patients [3,4]. Due to its reliance on identifying visible lesions, conventional staging is limited in its ability to detect microscopic cancer cells or the complex biological activities that drive tumor aggression [5]. In the era of precision medicine, a more accurate risk stratification approach could increase the efficiency of the selection of treatment options for NSCLC patients and, potentially, improve survival rates [6].
Several laboratory and pathological variables have been proposed as predictive markers for more accurate prognostic forecasting in NSCLC patients [7]. However, biomarkers derived from tissue samples are invasive, and laboratory markers obtained from blood tests reflect the secondary effects of lesions. Recently, there has been a focus on identifying non-invasive and lesion-specific prognostic imaging biomarkers employing CT-derived radiological parameters, a method referred to as radiomics [8]. Nevertheless, CT-derived structural information is limited in capturing the complex biological and functional characteristics of tumors [9].
18F-Fluoro-2-deoxyglucose (18F-FDG) positron emission tomography/computed tomography (PET/CT) has improved cancer staging and offered insights into the tumor metabolism, correlating with aggressiveness and metastatic potential [10,11]. Numerous studies have demonstrated that quantitative features derived from 18F-FDG PET, calculated based on the intensity and distribution of glucose uptake by the tumor, serve as valuable prognostic indicators [12,13,14,15]. Furthermore, recent studies have focused on developing imaging-based prognostic models by combining a large number of 18F-FDG PET-derived parameters. Although several models have also been developed for NSCLC, they often include both operable and inoperable patients [16,17]. Research targeting NSCLC patients who have received curative resection may be more clinically useful for determining the intensity of adjuvant therapy or postoperative surveillance.
Constructing predictive models for cancer prognosis using high-dimensional data with numerous features requires careful consideration. The continuous-time Cox proportional hazards model, while widely employed in survival analysis, requires the fulfillment of the proportional hazards assumption. This assumption is often violated in practice due to time-varying covariate effects and unobserved heterogeneity, which might be more prevalent in high-dimensional data [18]. A discrete-time prediction model for determining the survival status at a specific time point could offer a more suitable alternative for survival analysis with a large number of 18F-FDG-derived parameters [19]. We hypothesize that the integration of 18F-FDG PET-derived parameters with clinical variables will yield a predictive model with superior accuracy for five-year survival outcomes in NSCLC patients who have undergone curative surgery compared to models based solely on clinical data.

2. Materials and Methods

2.1. Study Population

Our study was approved by the Institutional Review Board, which waived the need for informed consent due to its retrospective nature. Consecutive NSCLC patients who underwent curative resection between January 2016 and December 2017 were retrospectively reviewed. The inclusion criteria were as follows: (1) patients who underwent pretreatment 18F-FDG PET/CT scans on a GE Discovery STE scanner, (2) no neoadjuvant chemotherapy or radiotherapy, and (3) postoperative follow-up for at least five years (unless death occurred). The exclusion criteria were as follows: (1) patients with incomplete medical records, (2) tumors with insufficient metabolic activity to be delineated by an SUV cut-off of 2.5, (3) a coexisting primary cancer, and (4) cases of multiple primary lung cancers. The final study population included 361 patients. The study recruitment process is presented in Figure 1.

2.2. Data Collection

All clinical and pathological data were collected from electronic medical records. Clinical characteristics included age, sex, type of surgery, and adjuvant treatment history. Tumor characteristics of tumor size, histological subtype, and stage were obtained from pathologic reports. Tumor staging was conducted in accordance with the eighth edition of the American Joint Committee on Cancer (AJCC) TNM staging system. The primary clinical endpoint of this analysis was the five-year survival status, defined as either survival or death from any cause for follow-up five years after surgery.

2.3. FDG PET/CT Image Acquisition

All patients were instructed to fast for at least 6 h before undergoing PET/CT scans. Their blood glucose levels were maintained below 200 mg/dL at the time of the 18F-FDG injection. Whole-body PET and CT images were acquired 60 min post-injection of 5.0 MBq/kg 18F-FDG without intravenous or oral contrast, using a GE Healthcare (Milwaukee, WI, USA) Discovery STE scanner. Continuous spiral CT was performed with a 16-slice helical CT at 140 keV, with a current of 30–170 mA using AutomA mode and a section width of 3.75 mm. Subsequently, emission PET data were acquired from the head to the thigh for 2.5 min per frame in a three-dimensional mode. PET images were reconstructed using the ordered subsets expectation maximization (OSEM) algorithm with 20 subsets and 2 iterations for the Discovery STE (matrix size 128 × 128, voxel size 3.9 × 3.9 × 3.3 mm3), with CT data utilized for the attenuation correction.

2.4. FDG PET/CT Image Analysis and Feature Extraction

The volume of interest (VOI) for the lung lesion was delineated on PET images using MIM version 7.0 (MIM Software Inc., Cleveland, OH, USA). An experienced nuclear medicine physician, blinded to the clinical information except for the tumor site, identified the target tumor. Tumor segmentation was conducted using a gradient-based method (‘PET Edge’) [20]. Operators initiated contouring by dragging a cursor from the center of the lesion towards its periphery. The algorithm generated six axes, adjusting their length when a significant gradient was detected, to outline a 3D VOI around the tumor. These VOIs were exported as DICOM-RT structures and imported into the Chang-Gung Image Texture Analysis toolbox (CGITA, available at http://code.google.com/p/cgita, accessed on 1 March 2020), which is supported by MATLAB software (version 2014b; MathWorks, Inc., Natick, MA, USA), for feature extraction from the PET images [21]. The calculation of textural features involved resampling the gray level using a fixed bin width method at 0.4 SUV units, derived from 64 gray levels ranging from 0 to 25 [22]. A total of 70 18F-FDG PET-derived features were calculated and categorized into several groups, as listed in Supplementary Table S1.

2.5. Feature Selection and Imaging-Based Risk Score Model Construction

The study cohort was randomly divided into training (70%) and test (30%) sets using the “createDataPartition” function in the “caret” package in R to ensure representative and unbiased distribution. The training set was utilized to develop a risk score model based on the 18F-FDG PET features to predict the five-year survival status. For selecting relevant variables to construct a predictive model, we initially evaluated the discriminative power using Receiver Operating Characteristic (ROC) curve analysis for a set of 70 features. Features exhibiting an area under the curve (AUC) of less than 0.6 were considered irrelevant and excluded from the model construction [23]. Subsequently, LASSO regression was chosen for its ability to handle high-dimensional data and prevent overfitting [24]. The optimal lambda (λ) was determined through 10-fold cross-validation, minimizing the mean cross-validation error. The “glmnet” package in R was used to perform LASSO regression. The PET-based risk score was calculated using a LASSO-based formula, created by multiplying each selected variable by its respective non-zero coefficient at the optimal λ value.

2.6. Statistical Analysis

All statistical analyses were conducted using the open-source software R version 4.0.2 (The R Foundation for Statistical Computing, Vienna, Austria) or MedCalc version 15.5 (MedCalc Software Ltd., Ostend, Belgium). Categorical variables were compared using the Chi-square test or Fisher’s exact test as appropriate. Continuous variables were compared between two groups using the Mann–Whitney U test. A multivariable logistic regression analysis was used to identify risk factors associated with five-year survival and to develop a hybrid model by combining the PET-based risk score with the clinical risk factors. The model performances were evaluated by calculating the AUC of the ROC curve, along with the 95% confidence interval (CI). To compare the AUC values between the models, DeLong’s test was used [25]. The optimal cut-off value for the PET-based risk score to predict binary outcomes was determined using the Youden Index from the ROC curve analysis. Calibration curves were created to assess the alignment between predicted probabilities and observed outcomes. All tests were two-sided, with statistical significance set at p < 0.05. The goodness of fit was determined using the Hosmer–Lemeshow test [26], where a p value above 0.05 indicated good calibration.

3. Results

3.1. Baseline Characteristics of Patients

The study comprised 237 male and 124 female subjects, with an average age of 63.1 ± 10.2 years. Of these, 253 were allocated to the training set, while the remaining 108 formed the test set. Table 1 summarizes the demographic details and tumor characteristics for the two sets. Clinical characteristics did not differ significantly between sets, ensuring a balanced distribution. Patients were categorized into survivor (n = 253) and non-survivor (n = 108) groups based on five-year survival, with baseline characteristics detailed in Table 2. Significant differences were observed in age, sex, tumor size, histology, and pathologic stage between groups, while adjuvant therapy history showed no significant difference (p = 0.648).

3.2. Establishment of the Imaging-Based Model

In the training set, the 18F-FDG PET-derived features were ordered by their AUC values, indicating discriminative power for the binary outcomes of the five-year survival status (Supplementary Table S2). The features with the best predictive performances were CO_contrast (0.675) and SUL peak (0.671), followed by TLG (0.669) and ISZ_size-zone variability (0.668) (Figure 2). Among the 70 features, 24 were excluded as irrelevant with poor performance (AUC < 0.6). Applying the remaining 46 features to the LASSO method, 5 were finally selected for an imaging-based risk score model (Figure 3). The predictive probability of death within five years for each patient was calculated using a simple linear combination of five selected indicators multiplied by their respective coefficients, as follows:
PET-based risk score = 0.113427981 + (0.0000002977 × CO_contrast) + (0.0000492625 × VA_intensity variability) + (0.010082504 × NID_contrast) + (0.013133648 × SUL peak) + (1.414154452 × TFCC_code similarity).
In the training set, the PET-based risk score had a median of 0.116 and an interquartile range (IQR) of 0.079–0.203. The optimal cut-off value of the risk score for predicting the five-year survival was identified as 0.111. Representative cases with high and low scores are illustrated in Figure 4.

3.3. Construction of Clinical and Hybrid Model

In the training set, we developed a clinical model that includes the five clinical risk factors that showed significant differences between survivor and non-survivor groups. A hybrid model was also constructed by incorporating an 18F-FDG PET-based risk score into the same clinical variables. The results of the two multivariable logistic regression models are shown in Table 3. In the clinical model, age (OR 1.080, 95% CI 1.041–1.120; p < 0.001) and tumor size (OR 1.214, 95% CI 1.018–1.448; p = 0.031) were identified as independent predictors of five-year survival. In the hybrid model, age (OR 1.083, 95% CI 1.043–1.125; p < 0.001) and the PET-based risk score (OR 5.231, 95% CI 1.987–6.932; p = 0.009) were identified as independent predictors of five-year survival. Tumor size (OR 1.046, 95% CI 0.848–1.291; p = 0.673) did not show independent significance when adjusted for the PET-based risk score.

3.4. Model Validtion

We applied the LASSO-based formula, developed using the training set, to the test set to calculate the PET-based risk scores for each case. The median risk score from the test set was 0.112 (IQR, 0.068–0.225), with no significant difference compared to the training set (p = 0.953). In the test set, the multivariable logistic regression results were consistent with the training set findings: in the clinical model, age and tumor size were independent variables, while, in the hybrid model, age and the PET-based risk score were independent predictors (Table 4).

3.5. Model Performance and Calibration

In the training set, ROC curves were generated for the PET-based risk score and clinical and hybrid models (Figure 5a). The associated AUC values for the five-year survival status were 0.696 (95% CI: 0.635–0.752), 0.756 (95% CI: 0.698–0.807), and 0.771 (95% CI: 0.715–0.822), respectively. The hybrid model demonstrated the best discriminative performance for predicting the 5-year survival status. When comparing AUCs (Supplementary Table S3), the hybrid model significantly outperformed the PET-based risk score (p = 0.022). The performance of the hybrid model demonstrated a modest improvement over the clinical model; however, the difference was not significant (p = 0.233). In the test set, the hybrid model also maintained the highest AUC (0.759, 95% CI: 0.667–0.836) (Figure 5b). However, the difference was not significant compared to those of the PET-based risk score (AUC = 0.724, 95% CI: 0.630–0.806; p = 0.334) or the clinical model (AUC = 0.731, 95% CI: 0.637–0.811; p = 0.213). The calibration curve of the hybrid model with the best predictive performance revealed a good agreement between the observed outcome and prediction in the training and test sets (Figure 6). Additionally, the Hosmer–Lemeshow test yielded a non-significant statistic in two sets (p  =  0.360 and p = 0.630, respectively), indicating that the model fit well.

4. Discussion

In the era of precision medicine, identifying cancer patients with a poor prognosis is crucial for individualized management strategies. Biomarkers signaling a poor prognosis can improve the risk stratification beyond traditional tumor staging [27]. Recent technological advances have enabled the extraction of diverse invisible information from imaging studies in the oncological area. 18F-FDG PET/CT offers significant prognostic insights by reflecting the metabolic characteristics of cancer. Employing these 18F-FDG PET features, the current study developed an imaging-based score formula for predicting five-year survival after curative surgery in NSCLC patients. This PET-based risk score was an independent prognostic factor distinct from clinically related prognostic variables. The PET-based risk score, as demonstrated in the representative case in Figure 4, may provide prognostic information that tumor staging alone cannot capture, potentially aiding in patient management. The integrated hybrid model incorporating clinical variables demonstrated good discriminative performances for predicting survival status at five years in the training (AUC = 0.771) and testing sets (AUC = 0.759). Although the difference was not significant, the predictive performance of the hybrid model was superior to that of the clinical model alone in both the training and test sets.
Metabolic heterogeneity has recently been recognized as a crucial factor in cancer progression [28]. 18F-FDG PET-derived textural features can provide information more closely associated with these tumor characteristics compared to those extracted from CT [29]. In our study, among 70 PET-derived features, the contrast from the co-occurrence matrix showed the highest predictive accuracy for survival (AUC = 0.675), with other features from the same matrix also demonstrating strong performance (AUCs ranging from 0.653 to 0.663). The co-occurrence matrix represents local tumor heterogeneity by analyzing the spatial relationship between pixel intensities within an image [30]. Prior studies on PET-derived textural features in NSCLC have consistently identified those from the co-occurrence matrix as offering superior prognostic predictive power [31]. These results suggest the significance of local tumor heterogeneity over global heterogeneity in affecting patient outcomes. Additional research is required to validate these observations and investigate their implications.
Recent prognostic studies are focused on developing models that combine various associated features to improve clinical usefulness. In high-dimensional data, identifying relevant features is crucial for developing effective models, as it may include numerous unrelated variables [24]. We initially excluded irrelevant variables by analyzing the individual AUCs of the features. To apply the developed model to new clinical data, it is also necessary to reduce the risk of overfitting in the training data. Recently, the LASSO method has been widely employed in the modeling of high-dimensional data to address these issues [32]. After the LASSO method was applied, an imaging-based model that incorporates five relevant features was constructed. These features included CO_contrast and SUL_peak, each demonstrating the best individual predictive performance. However, the other three variables showed relatively lower individual predictive performance, with AUC values ranging between 0.618 and 0.645. This suggests that the LASSO model prioritizes not only high predictive accuracy but also adaptability across various datasets. In testing, the model consistently exhibited comparable predictive performance, affirming its robustness.
The Cox proportional hazards model, widely used in survival analysis, presumes that variables’ effects remain constant over time. This assumption is not always true in practice, limiting its effectiveness when hazard ratios fluctuate. Its impact could be particularly significant in the analyses of high-dimensional data containing many potential variables [18]. A recent study highlighted that a discrete-time prediction model, which estimates the survival status at a specific time point, could serve as a superior option for survival analysis with many features [19]. This alternative avoids the need for the assumption of proportional hazards. Furthermore, this binary outcome prediction method enhances the interpretability, simplifying the comprehension for both medical professionals and patients. We focused specifically on the status of survival at the five-year mark, considered a crucial time point in cancer prognosis [33]. Our model indicates that patients unlikely to survive past five years might benefit from continued therapy or close monitoring post-treatment.
Previous research has developed prognostic models using 18F-FDG PET-derived textural features for NSCLC. Ahn et al. introduced a random forest model based on these features to predict three-year recurrence in 93 stage I–III NSCLC patients post-curative surgery, achieving an AUC of 0.956 [34]. Another study utilized a naive Bayes model to forecast the two-year recurrence in 77 stage I–III NSCLC patients treated with curative intent, reaching an AUC of 0.816 [35]. However, for the accurate analysis of high-dimensional data, machine learning methods typically require larger sample sizes. The LASSO technique is effective for high-dimensional, low-sample-size data analysis [32]. Li et al. developed a model using pretreatment 18F-FDG PET/CT textural features and the LASSO method in 368 NSCLC patients, demonstrating its utility in overall survival prediction. This model, combining clinical and imaging-based scores, outperformed a clinical-only model in OS prediction, with an AUC of 0.891 versus 0.846 [16]. Yang et al. found that adding PET/CT-based risk scores to clinical parameters significantly improved the OS prediction in 315 NSCLC patients, showing high concordance index values in both the training (0.776) and validation (0.789) cohorts [17]. Our study also showed a modest improvement in the predictive performance for survival outcomes using a hybrid model that integrates clinical variables and PET-based risk scores relative to using a clinical-only model. However, distinct from prior studies that comprised a broad range of patient populations (stages I–IV) undergoing both curative and palliative treatments, our research focused exclusively on a larger cohort receiving curative therapy. Our model can provide more valuable insights for post-surgery follow-up strategies or decisions on adjuvant therapy.
PET-based feature analysis has several remaining challenges. A major issue is textural feature variation due to differing tumor delineation methods [36]. Advanced tumor segmentation techniques that apply deep learning for improved tumor delineation have been introduced but not validated [37]. Our study used PET Edge, a validated, widely-used, gradient-based delineation method [20]. This semi-automated method improves feature value reproducibility by reducing inter-observer variability, surpassing manual segmentation. Moreover, it can better capture necrotic portions associated with prognosis compared to models with fixed thresholds [38]. Variability due to the use of different software is also a major issue. Recently, the International Biomarker Standardization Initiative (IBSI) recommended using LifeX or pyradiomics in radiomics feature analysis to address this. These features are advantageous for comparing radiomics features across multi-imaging modalities because they are commonly used in PET, CT, and MRI [39]. However, the CGITA software we used is focused on PET texture analysis, and its clinical utility has been validated [21]. Future studies are warranted to compare the prognostic significance of 18F-FDG PET-derived textural features extracted using various software tools.
Our study has several limitations. First, its retrospective design and the use of single-center data introduce inherent biases and limit the generalizability of our findings. Additionally, the limited sample size affects the power of our conclusions. The predictive model was not externally validated with an independent cohort from another institution, which may further restrict its generalizability and robustness. Potential confounding variables that were not considered also could have influenced the results. Last, certain cases had a short follow-up period, which might impact the assessment of long-term survival outcomes.

5. Conclusions

Among the PET-derived features, CO_contrast and SUL_peak demonstrated the best individual predictive performance for five-year survival status and were included in the LASSO-based formula. The 18F-FDG PET-based risk score, derived using the LASSO method, served as an independent prognostic indicator, distinct from clinical variables, for predicting the five-year survival status in patients with NSCLC who underwent curative surgery. Our PET-based risk score, particularly when combined with clinical variables, offers a valuable tool for risk stratification in NSCLC following curative surgery. This model can aid clinicians in identifying high-risk patients who may benefit from more aggressive follow-up and adjuvant therapies, ultimately improving personalized treatment strategies and patient outcomes. Future research should focus on validating our model with independent, multi-center cohorts to enhance the generalizability. Additionally, by comparing the prognostic value of the PET-derived features using different image analysis software and exploring advanced tumor segmentation techniques.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/cancers16142525/s1, Supplementary Table S1: List of 70 quantitative PET-based radiomic features. Supplementary Table S2: AUC values of 46 relevant 18F-FDG PET-based features. Supplementary Table S3: Comparison of predictive performances in three models.

Author Contributions

All authors contributed equally to this work. Conceptualization, C.H.L. and J.Y.C.; data curation, C.H.L. and J.Y.C.; formal analysis, C.H.L. and J.Y.C.; funding acquisition, J.Y.C.; investigation, C.H.L. and J.Y.C.; methodology, C.H.L. and J.Y.C.; project administration, C.H.L. and J.Y.C.; resources, S.-W.U., H.K.K., H.R.P., Y.S.C., M.-J.A. and J.Y.C.; software, C.H.L. and J.Y.C.; supervision, J.Y.C.; validation, C.H.L. and J.Y.C.; visualization, C.H.L. and J.Y.C.; writing—original draft preparation, C.H.L.; writing—review and editing, C.H.L., S.-W.U., H.K.K., H.R.P., Y.S.C., M.-J.A. and J.Y.C. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT) (RS-2024-00334782), Future Medicine 20*30 Project of the Samsung Medical Center (#SMO1240791), and by the Soonchunhyang University Research Fund.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki and was approved by the Institutional Review Board of the Samsung Medical Center (project file no. 2022-01-086).

Informed Consent Statement

Patient consent was waived due to its retrospective nature.

Data Availability Statement

Restrictions apply to the availability of these data. Data were obtained from the Samsung Medical Center and are available from the corresponding author with the permission of the Samsung Medical Center.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Sung, H.; Ferlay, J.; Siegel, R.L.; Laversanne, M.; Soerjomataram, I.; Jemal, A.; Bray, F. Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries. CA Cancer J. Clin. 2021, 71, 209–249. [Google Scholar] [CrossRef]
  2. Zappa, C.; Mousa, S.A. Non-small cell lung cancer: Current treatment and future advances. Transl. Lung Cancer Res. 2016, 5, 288–300. [Google Scholar] [CrossRef] [PubMed]
  3. Araghi, M.; Mannani, R.; Heidarnejad maleki, A.; Hamidi, A.; Rostami, S.; Safa, S.H.; Faramarzi, F.; Khorasani, S.; Alimohammadi, M.; Tahmasebi, S. Recent advances in non-small cell lung cancer targeted therapy; an update review. Cancer Cell Int. 2023, 23, 162. [Google Scholar] [CrossRef] [PubMed]
  4. Flury, D.V.; Minervini, F.; Kocher, G.J. Heterogeneity of stage IIIA non-small cell lung cancer-different tumours, different nodal status, different treatment, different prognosis: A narrative review. Curr. Chall. Thorac. Surg. 2022, 4, 13. [Google Scholar] [CrossRef]
  5. Fares, J.; Fares, M.Y.; Khachfe, H.H.; Salhab, H.A.; Fares, Y. Molecular principles of metastasis: A hallmark of cancer revisited. Signal Transduct. Target. Ther. 2020, 5, 28. [Google Scholar] [CrossRef] [PubMed]
  6. Middleton, G.; Robbins, H.; Andre, F.; Swanton, C. A state-of-the-art review of stratified medicine in cancer: Towards a future precision medicine strategy in cancer. Ann. Oncol. 2022, 33, 143–157. [Google Scholar] [CrossRef] [PubMed]
  7. Lin, J.; Carter, C.A.; McGlynn, K.A.; Zahm, S.H.; Nations, J.A.; Anderson, W.F.; Shriver, C.D.; Zhu, K. A prognostic model to predict mortality among non–small-cell lung cancer patients in the US military health system. J. Thorac. Oncol. 2015, 10, 1694–1702. [Google Scholar] [CrossRef] [PubMed]
  8. Zhang, Y.; Oikonomou, A.; Wong, A.; Haider, M.A.; Khalvati, F. Radiomics-based prognosis analysis for non-small cell lung cancer. Sci. Rep. 2017, 7, srep46349. [Google Scholar] [CrossRef] [PubMed]
  9. García-Figueiras, R.; Baleato-González, S.; Padhani, A.R.; Luna-Alcalá, A.; Vallejo-Casas, J.A.; Sala, E.; Vilanova, J.C.; Koh, D.-M.; Herranz-Carnero, M.; Vargas, H.A. How clinical imaging can assess cancer biology. Insights Into Imaging 2019, 10, 28. [Google Scholar] [CrossRef] [PubMed]
  10. Lim, C.H.; Park, Y.-J.; Shin, M.; Cho, Y.S.; Choi, J.Y.; Lee, K.-H.; Hyun, S.H. Tumor SUVs on 18F-FDG PET/CT and aggressive pathological features in esophageal squamous cell carcinoma. Clin. Nucl. Med. 2020, 45, e128–e133. [Google Scholar] [CrossRef] [PubMed]
  11. Xu, C.; Li, H.; Seng, D.; Liu, F. Significance of SUV Max for Predicting Occult Lymph Node Metastasis and Prognosis in Early-Stage Tongue Squamous Cell Carcinoma. J. Oncol. 2020, 2020, 6241637. [Google Scholar] [CrossRef]
  12. Im, H.-J.; Pak, K.; Cheon, G.J.; Kang, K.W.; Kim, S.-J.; Kim, I.-J.; Chung, J.-K.; Kim, E.E.; Lee, D.S. Prognostic value of volumetric parameters of 18F-FDG PET in non-small-cell lung cancer: A meta-analysis. Eur. J. Nucl. Med. 2015, 42, 241–251. [Google Scholar] [CrossRef] [PubMed]
  13. Lee, H.; Choi, Y.-L.; Kim, H.K.; Choi, Y.S.; Kim, H.; Ahn, M.-J.; Pyo, H.R.; Choi, J.Y. Prognostic Significance of Volumetric Parameters Based on FDG PET/CT in Patients with Lung Adenocarcinoma Undergoing Curative Surgery. Cancers 2023, 15, 4380. [Google Scholar] [CrossRef] [PubMed]
  14. Hatt, M.; Tixier, F.; Pierce, L.; Kinahan, P.E.; Le Rest, C.C.; Visvikis, D. Characterization of PET/CT images using texture analysis: The past, the present… any future? Eur. J. Nucl. Med. Mol. Imaging 2017, 44, 151–165. [Google Scholar] [CrossRef] [PubMed]
  15. Erol, M.; Önner, H.; Küçükosmanoğlu, İ. Association of Fluorodeoxyglucose Positron Emission Tomography Radiomics Features with Clinicopathological Factors and Prognosis in Lung Squamous Cell Cancer. Nucl. Med. Mol. Imaging 2022, 56, 306–312. [Google Scholar] [CrossRef] [PubMed]
  16. Li, J.; Zhang, B.; Ge, S.; Deng, S.; Hu, C.; Sang, S. Prognostic value of 18F-FDG PET/CT radiomic model based on primary tumor in patients with non-small cell lung cancer: A large single-center cohort study. Front. Oncol. 2022, 12, 1047905. [Google Scholar] [CrossRef]
  17. Yang, B.; Zhong, J.; Zhong, J.; Ma, L.; Li, A.; Ji, H.; Zhou, C.; Duan, S.; Wang, Q.; Zhu, C. Development and validation of a radiomics nomogram based on 18F-fluorodeoxyglucose positron emission tomography/computed tomography and clinicopathological factors to predict the survival outcomes of patients with non-small cell lung cancer. Front. Oncol. 2020, 10, 1042. [Google Scholar] [CrossRef] [PubMed]
  18. Zeng, Z.; Gao, Y.; Li, J.; Zhang, G.; Sun, S.; Wu, Q.; Gong, Y.; Xie, C. Violations of proportional hazard assumption in Cox regression model of transcriptomic data in TCGA pan-cancer cohorts. Comput. Struct. Biotechnol. J. 2022, 20, 496–507. [Google Scholar] [CrossRef] [PubMed]
  19. Suresh, K.; Severn, C.; Ghosh, D. Survival prediction models: An introduction to discrete-time modeling. BMC Med Res. Methodol. 2022, 22, 207. [Google Scholar] [CrossRef] [PubMed]
  20. Werner-Wasik, M.; Nelson, A.D.; Choi, W.; Arai, Y.; Faulhaber, P.F.; Kang, P.; Almeida, F.D.; Xiao, Y.; Ohri, N.; Brockway, K.D.; et al. What Is the Best Way to Contour Lung Tumors on PET Scans? Multiobserver Validation of a Gradient-Based Method Using a NSCLC Digital PET Phantom. Int. J. Radiat. Oncol. 2012, 82, 1164–1171. [Google Scholar] [CrossRef]
  21. Fang, Y.-H.D.; Lin, C.-Y.; Shih, M.-J.; Wang, H.-M.; Ho, T.-Y.; Liao, C.-T.; Yen, T.-C. Development and Evaluation of an Open-Source Software Package “CGITA” for Quantifying Tumor Heterogeneity with Molecular Images. BioMed Res. Int. 2014, 2014, 248505. [Google Scholar] [CrossRef] [PubMed]
  22. Ha, S.; Park, S.; Bang, J.-I.; Kim, E.-K.; Lee, H.-Y. Metabolic Radiomics for Pretreatment 18F-FDG PET/CT to Characterize Locally Advanced Breast Cancer: Histopathologic Characteristics, Response to Neoadjuvant Chemotherapy, and Prognosis. Sci. Rep. 2017, 7, 1556. [Google Scholar] [CrossRef] [PubMed]
  23. Yang, S.; Berdine, G. The receiver operating characteristic (ROC) curve. Southwest. Respir. Crit. Care Chron. 2017, 5, 34–36. [Google Scholar] [CrossRef]
  24. Emmert-Streib, F.; Dehmer, M. High-dimensional LASSO-based computational regression models: Regularization, shrinkage, and selection. Mach. Learn. Knowl. Extr. 2019, 1, 359–383. [Google Scholar] [CrossRef]
  25. DeLong, E.R.; DeLong, D.M.; Clarke-Pearson, D.L. Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics 1988, 837–845. [Google Scholar] [CrossRef]
  26. Hosmer, D.W.; Hosmer, T.; Le Cessie, S.; Lemeshow, S. A comparison of goodness-of-fit tests for the logistic regression model. Stat. Med. 1997, 16, 965–980. [Google Scholar] [CrossRef]
  27. Zou, J.; Wang, E. Cancer biomarker discovery for precision medicine: New progress. Curr. Med. Chem. 2019, 26, 7655–7671. [Google Scholar] [CrossRef] [PubMed]
  28. Kim, J.; DeBerardinis, R.J. Mechanisms and implications of metabolic heterogeneity in cancer. Cell Metab. 2019, 30, 434–446. [Google Scholar] [CrossRef] [PubMed]
  29. Lee, J.W.; Kim, S.Y.; Han, S.W.; Lee, J.E.; Hong, S.H.; Lee, S.M.; Jo, I.Y. Clinical significance of peritumoral adipose tissue PET/CT imaging features for predicting axillary lymph node metastasis in patients with breast cancer. J. Pers. Med. 2021, 11, 1029. [Google Scholar] [CrossRef] [PubMed]
  30. Haralick, R.M.; Shanmugam, K.; Dinstein, I.H. Textural features for image classification. IEEE Trans. Syst. Man Cybern. 1973, 6, 610–621. [Google Scholar] [CrossRef]
  31. Han, S.; Woo, S.; Suh, C.H.; Kim, Y.J.; Oh, J.S.; Lee, J.J. A systematic review of the prognostic value of texture analysis in 18 F-FDG PET in lung cancer. Ann. Nucl. Med. 2018, 32, 602–610. [Google Scholar] [CrossRef] [PubMed]
  32. Park, J.E.; Park, S.Y.; Kim, H.J.; Kim, H.S. Reproducibility and Generalizability in Radiomics Modeling: Possible Strategies in Radiologic and Statistical Perspectives. Korean J. Radiol. 2019, 20, 1124–1137. [Google Scholar] [CrossRef] [PubMed]
  33. Jeon, D.S.; Kim, H.C.; Kim, S.H.; Kim, T.-J.; Kim, H.K.; Moon, M.H.; Beck, K.S.; Suh, Y.-G.; Song, C.; Ahn, J.S.; et al. Five-Year Overall Survival and Prognostic Factors in Patients with Lung Cancer: Results from the Korean Association of Lung Cancer Registry (KALC-R) 2015. Cancer Res. Treat. 2023, 55, 103–111. [Google Scholar] [CrossRef] [PubMed]
  34. Ahn, H.; Lee, H.; Kim, S.; Hyun, S. Pre-treatment 18F-FDG PET-based radiomics predict survival in resected non-small cell lung cancer. Clin. Radiol. 2019, 74, 467–473. [Google Scholar] [CrossRef] [PubMed]
  35. Park, S.B.; Kim, K.-U.; Park, Y.W.; Hwang, J.H.; Lim, C.H. Application of 18F-fluorodeoxyglucose PET/CT radiomic features and machine learning to predict early recurrence of non-small cell lung cancer after curative-intent therapy. Nucl. Med. Commun. 2023, 44, 161–168. [Google Scholar] [CrossRef] [PubMed]
  36. Im, H.-J.; Bradshaw, T.; Solaiyappan, M.; Cho, S.Y. Current methods to define metabolic tumor volume in positron emission tomography: Which one is better? Nucl. Med. Mol. Imaging 2018, 52, 5–15. [Google Scholar] [CrossRef] [PubMed]
  37. Park, J.; Kang, S.K.; Hwang, D.; Choi, H.; Ha, S.; Seo, J.M.; Eo, J.S.; Lee, J.S. Automatic Lung Cancer Segmentation in [18F]FDG PET/CT Using a Two-Stage Deep Learning Approach. Nucl. Med. Mol. Imaging 2023, 57, 86–93. [Google Scholar] [CrossRef] [PubMed]
  38. Guezennec, C.; Bourhis, D.; Orlhac, F.; Robin, P.; Corre, J.-B.; Delcroix, O.; Gobel, Y.; Schick, U.; Salaün, P.-Y.; Abgral, R. Inter-observer and segmentation method variability of textural analysis in pre-therapeutic FDG PET/CT in head and neck cancer. PLoS ONE 2019, 14, e0214299. [Google Scholar] [CrossRef] [PubMed]
  39. Garrido, P.; Conde, E.; de Castro, J.; Gómez-Román, J.J.; Felip, E.; Pijuan, L.; Isla, D.; Sanz, J.; Paz-Ares, L.; López-Ríos, F. Updated guidelines for predictive biomarker testing in advanced non-small-cell lung cancer: A National Consensus of the Spanish Society of Pathology and the Spanish Society of Medical Oncology. Clin. Transl. Oncol. 2020, 22, 989–1003. [Google Scholar] [CrossRef] [PubMed]
Cancers 16 02525 g001
Figure 1. Chart of study identification and inclusion and exclusion criteria.
Figure 1. Chart of study identification and inclusion and exclusion criteria.
Cancers 16 02525 g001
Cancers 16 02525 g002
Figure 2. Area under the curve (AUC) values for the top 35 18F-FDG PET-derived features predicting five-year survival after surgery.
Figure 2. Area under the curve (AUC) values for the top 35 18F-FDG PET-derived features predicting five-year survival after surgery.
Cancers 16 02525 g002
Cancers 16 02525 g003
Figure 3. 18F-FDG PET-derived feature selection using the least absolute shrinkage and selection operator (LASSO) logistic regression model. (a) Identification of the optimal penalization coefficient lambda (λ) in the LASSO model used 10-fold cross-validation and the minimum criterion. As a result, a λ value of 0.0305 was selected. (b) LASSO coefficient profiles of 5 features selected among 46 features. Each colored line represents the coefficient of each feature.
Figure 3. 18F-FDG PET-derived feature selection using the least absolute shrinkage and selection operator (LASSO) logistic regression model. (a) Identification of the optimal penalization coefficient lambda (λ) in the LASSO model used 10-fold cross-validation and the minimum criterion. As a result, a λ value of 0.0305 was selected. (b) LASSO coefficient profiles of 5 features selected among 46 features. Each colored line represents the coefficient of each feature.
Cancers 16 02525 g003
Cancers 16 02525 g004
Figure 4. Representative PET/CT images of NSCLC patients following right upper lobectomy for adenocarcinoma. (a) Maximum intensity projection (left), transaxial fusion PET/CT (right middle), and CT (right bottom) images of a 74-year-old female with a high FDG-based risk score of 0.171. The patient’s pathologic tumor stage was I [T2 (tumor size 3.5 cm) N0 (no metastasis after mediastinal lymph node dissection)], but she died 20 months after surgery due to recurrence. (b) Maximum intensity projection (left), transaxial fusion PET/CT (right middle), and CT (right bottom) images of 61-year-old female with a low FDG-based risk score (0.076). The patient’s pathologic tumor stage was III [T2 (tumor size 2.2 cm, but invaded the visceral pleural beyond the elastic layer) N2 (identified metastasis in right pulmonary hilar and lower paratracheal lymph nodes after mediastinal lymph node dissection)], but she was alive at the five-year follow-up.
Figure 4. Representative PET/CT images of NSCLC patients following right upper lobectomy for adenocarcinoma. (a) Maximum intensity projection (left), transaxial fusion PET/CT (right middle), and CT (right bottom) images of a 74-year-old female with a high FDG-based risk score of 0.171. The patient’s pathologic tumor stage was I [T2 (tumor size 3.5 cm) N0 (no metastasis after mediastinal lymph node dissection)], but she died 20 months after surgery due to recurrence. (b) Maximum intensity projection (left), transaxial fusion PET/CT (right middle), and CT (right bottom) images of 61-year-old female with a low FDG-based risk score (0.076). The patient’s pathologic tumor stage was III [T2 (tumor size 2.2 cm, but invaded the visceral pleural beyond the elastic layer) N2 (identified metastasis in right pulmonary hilar and lower paratracheal lymph nodes after mediastinal lymph node dissection)], but she was alive at the five-year follow-up.
Cancers 16 02525 g004
Cancers 16 02525 g005
Figure 5. Receiver operating characteristic curves for the PET-based risk score and clinical and hybrid models in the (a) training set and (b) test set.
Figure 5. Receiver operating characteristic curves for the PET-based risk score and clinical and hybrid models in the (a) training set and (b) test set.
Cancers 16 02525 g005
Cancers 16 02525 g006
Figure 6. Calibration curves of hybrid models in the (a) training set and (b) test set.
Figure 6. Calibration curves of hybrid models in the (a) training set and (b) test set.
Cancers 16 02525 g006
Table 1. Clinical characteristics of the training set and the test set.
Table 1. Clinical characteristics of the training set and the test set.
VariableTotal
(n = 361)		Training Set
(n = 253)		Test Set
(n = 108)		p Value
Age, years (mean ± SD)63.1 ± 10.262.8 ± 10.263.8 ± 10.20.382
Sex, male (%)237 (65.7)161 (63.6)76 (70.3)0.218
Tumor size, cm (mean ± SD)3.6 ± 1.93.7 ± 2.03.5 ± 1.60.327
Histology 0.762
Adenocarcinoma (%)232 (64.3)165 (65.2)67 (62.0)
Squamous cell carcinoma (%)104 (28.8)70 (27.7)34 (31.5)
Others (%)25 (6.9)18 (7.1)7 (6.5)
Tumor stage 0.526
I (%)140 (38.8)95 (37.6)45 (41.7)
II (%)105 (29.1)78 (30.8)27 (25.0)
III (%)116 (32.1)80 (31.6)36 (33.3)
Type of surgery 0.438
Pneumonectomy (%)12 (3.3)6 (2.4)6 (5.6)
Bilobectomy (%)17 (4.7)11 (4.4)6 (5.6)
Lobectomy (%)307 (85.1)218 (86.1)89 (82.4)
Segmentectomy (%)25 (6.9)18 (7.1)7 (6.4)
Adjuvant therapy 0.466
None (%)204 (56.5)138 (54.5)66 (61.1)
Chemotherapy (%)128 (35.5)94 (37.2)34 (31.5)
Radiotherapy (%)3 (0.8)3 (1.2)0 (0.0)
Chemoradiotherapy (%)26 (7.2)18 (7.1)8 (7.4)
Death within five years (%)108 (29.9)69 (27.3)39 (36.1)0.094
Data are presented as numbers (%) or mean ± standard deviation.
Table 2. Comparison of clinical characteristics between survivor and non-survivor groups.
Table 2. Comparison of clinical characteristics between survivor and non-survivor groups.
VariableNon-Survivor
(n = 108)		Survivor
(n = 253)		p Value
Age, years (mean ± SD)67.6 ± 9.361.2 ± 10.0<0.001
Sex, male (%)83 (76.9)154 (60.9)0.004
Tumor size, cm (mean ± SD)4.5 ± 2.43.3 ± 1.5<0.001
Histology, ADC (%)56 (51.9)176 (69.6)0.001
Tumor stage, II/III (%)80 (74.1)141 (55.7)0.001
Adjuvant therapy, presence (%)45 (41.7)112 (44.3)0.648
Data are presented as numbers (%) or mean ± standard deviation. SD, standard deviation; ADC, adenocarcinoma.
Table 3. Multivariable logistic regression analysis of the training set.
Table 3. Multivariable logistic regression analysis of the training set.
VariablesClinical ModelHybrid Model
Adjusted OR95% CIp ValueAdjusted OR95% CIp Value
Age, years1.0801.041–1.120<0.0011.0831.043–1.125<0.001
Sex (M vs. F)1.3370.644–2.7720.4361.0380.486–2.2170.924
Tumor stage (III/II vs. I)1.5670.733–3.3510.2471.3010.595–2.8450.510
Histology (ADC vs. others)0.6670.334–1.3350.2530.8810.421–1.8450.737
Tumor size, cm1.2141.018–1.4480.0311.0460.848–1.2910.673
PET-based risk score 5.2311.987–6.9320.009
OR, odds ratio; CI, confidence interval; ADC, adenocarcinoma.
Table 4. Multivariable logistic regression analysis of the test set.
Table 4. Multivariable logistic regression analysis of the test set.
VariablesClinical ModelHybrid Model
Adjusted OR95% CIp ValueAdjusted OR95% CIp Value
Age, years1.0511.003–1.1000.0361.0601.010–1.1130.019
Sex (M vs. F)1.5380.548–4.3160.4141.4500.500–4.2010.494
Tumor stage (III/II vs. I)1.3920.480–4.0410.5431.3860.462–4.1560.560
Histology (ADC vs. others)1.3230.506–3.4560.5681.6950.614–4.6800.309
Tumor size, cm1.4751.046–2.0800.0271.0080.619–1.6420.976
PET based-risk score 1311.5211.901–904,972.1320.031
HR, hazard ratio; CI, confidence interval; ADC, adenocarcinoma.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Lim, C.H.; Um, S.-W.; Kim, H.K.; Choi, Y.S.; Pyo, H.R.; Ahn, M.-J.; Choi, J.Y. 18F-Fluorodeoxyglucose Positron Emission Tomography-Based Risk Score Model for Prediction of Five-Year Survival Outcome after Curative Resection of Non-Small-Cell Lung Cancer. Cancers 2024, 16, 2525. https://doi.org/10.3390/cancers16142525

AMA Style

Lim CH, Um S-W, Kim HK, Choi YS, Pyo HR, Ahn M-J, Choi JY. 18F-Fluorodeoxyglucose Positron Emission Tomography-Based Risk Score Model for Prediction of Five-Year Survival Outcome after Curative Resection of Non-Small-Cell Lung Cancer. Cancers. 2024; 16(14):2525. https://doi.org/10.3390/cancers16142525

Chicago/Turabian Style

Lim, Chae Hong, Sang-Won Um, Hong Kwan Kim, Yong Soo Choi, Hong Ryul Pyo, Myung-Ju Ahn, and Joon Young Choi. 2024. "18F-Fluorodeoxyglucose Positron Emission Tomography-Based Risk Score Model for Prediction of Five-Year Survival Outcome after Curative Resection of Non-Small-Cell Lung Cancer" Cancers 16, no. 14: 2525. https://doi.org/10.3390/cancers16142525

APA Style

Lim, C. H., Um, S. -W., Kim, H. K., Choi, Y. S., Pyo, H. R., Ahn, M. -J., & Choi, J. Y. (2024). 18F-Fluorodeoxyglucose Positron Emission Tomography-Based Risk Score Model for Prediction of Five-Year Survival Outcome after Curative Resection of Non-Small-Cell Lung Cancer. Cancers, 16(14), 2525. https://doi.org/10.3390/cancers16142525

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop