Radiomics Profiling Identifies the Value of CT Features for the Preoperative Evaluation of Lymph Node Metastasis in Papillary Thyroid Carcinoma

Background: The aim of this study was to identify the increased value of integrating computed tomography (CT) radiomics analysis with the radiologists’ diagnosis and clinical factors to preoperatively diagnose cervical lymph node metastasis (LNM) in papillary thyroid carcinoma (PTC) patients. Methods: A total of 178 PTC patients were randomly divided into a training (n = 125) and a test cohort (n = 53) with a 7:3 ratio. A total of 2553 radiomic features were extracted from noncontrast, arterial contrast-enhanced and venous contrast-enhanced CT images of each patient. Principal component analysis (PCA) and Pearson’s correlation coefficient (PCC) were used for feature selection. Logistic regression was employed to build clinical–radiological, radiomics and combined models. A nomogram was developed by combining the radiomics features, CT-reported lymph node status and clinical factors. Results: The radiomics model showed a predictive performance similar to that of the clinical–radiological model, with similar areas under the curve (AUC) and accuracy (ACC). The combined model showed an optimal predictive performance in both the training (AUC, 0.868; ACC, 86.83%) and test cohorts (AUC, 0.878; ACC, 83.02%). Decision curve analysis demonstrated that the combined model has good clinical application value. Conclusions: Embedding CT radiomics into the clinical diagnostic process improved the diagnostic accuracy. The developed nomogram provides a potential noninvasive tool for LNM evaluation in PTC patients.


Introduction
Papillary thyroid carcinoma (PTC) is the most common primary thyroid malignancy, occurring in 90% of patients with thyroid carcinoma [1,2]. The incidence of PTC has increased dramatically in recent decades [3]. Although PTC is an inert cancer with relatively low recurrence and mortality rates, metastasis remains a concerning clinical problem. The occurrence of cervical lymph node metastasis (LNM) in patients with PTC is highly associated with local recurrence and poor overall survival [4,5]. Cervical LNM can be divided into central LNM (level VI) and lateral LNM (level II-V). The revised American Thyroid Association (ATA) guidelines recommend ipsilateral lobectomy for small unifocal tumors without extrathyroidal extension or LNM because of the slow progression of PTC [6]. For patients with evident lateral LNM, therapeutic lateral neck lymph node dissection is recommended [7]. Therefore, an accurate preoperative identification of cervical LNM is important for optimal staging, individualized treatment planning and prognosis evaluation for patients with PTC.

Patients
This retrospective study was approved by the institutional review board of the First Hospital of Shanxi Medical University (2021-K-K140), and carried out in accordance with the Declaration of Helsinki. The requirement for informed consent was signed for all patients. A total of 319 patients with suspected PTC in our institution from December 2017 to April 2021 were collected and screened. All patients identified in this study met the following inclusion criteria: (1) pathologically confirmed to have PTC; (2) a pathological lymph node diagnosis; (3) preoperative noncontrast CT scan and dual-phase dynamic contrast-enhanced CT images of sufficiently high quality for analysis; and (4) sufficient available clinical information. The exclusion criteria were as follows: (1) preoperative therapy (such as radiotherapy, chemotherapy or interventional therapy); (2) other malignancies; (3) postoperative pathological confirmation of multifocal PTC; and (4) unclear CT images of the primary tumors or tumors too small for segmentation and analysis (maximum diameter ≤ 3 mm). Finally, 178 patients (43 males and 135 females) with a mean age of 45.57 ± 13.00 years (range 9 to 73 years) who met the inclusion and exclusion criteria were enrolled in the study. The details of the included and excluded patients are shown in Figure 1. A total of 178 patients with PTC were divided into two groups: LNM (+) (n = 100) and LNM (−) (n = 78). Chi-square tests compared the clinical and radiological parameters between the LNM (+) and LNM (−) groups. The clinical and radiological factors with statistical difference were identified as the most diagnostic clinical and radiological factors. Clinical information, including age, sex, body mass index (BMI), thyroid hormone level and presence of nodular goiters or Hashimoto's thyroiditis, was obtained through the medical records system. According to clinical experience, the normal standards for BMI, thyroglobulin (TG), thyroid-stimulating hormone (TSH), free triiodothyronine (FT3), free thyroxine (FT4), thyroglobulin antibody (TGAb) and thyroid peroxidase antibody (TPOAb) were as follows: BMI, 18.5-23.9; TG, 3.5-77 ng/mL; TSH, 0.27-4.2 uIU/mL; FT3, 3.1-6.8 pmol/L; FT4, 12-22 pmol/L; TGAb, 0-115 IU/mL; TPOAb, 0-34 IU/mL. Any values outside these ranges were considered abnormal. Then, according to the age and sex listed in the clinical data, the patients were randomly divided into a training cohort (n = 125) and test cohort (n = 53) with a ratio of 7:3.
the Declaration of Helsinki. The requirement for informed consent was signed for all patients. A total of 319 patients with suspected PTC in our institution from December 2017 to April 2021 were collected and screened. All patients identified in this study met the following inclusion criteria: (1) pathologically confirmed to have PTC; (2) a pathological lymph node diagnosis; (3) preoperative noncontrast CT scan and dual-phase dynamic contrast-enhanced CT images of sufficiently high quality for analysis; and (4) sufficient available clinical information. The exclusion criteria were as follows: (1) preoperative therapy (such as radiotherapy, chemotherapy or interventional therapy); (2) other malignancies; (3) postoperative pathological confirmation of multifocal PTC; and (4) unclear CT images of the primary tumors or tumors too small for segmentation and analysis (maximum diameter ≤ 3 mm). Finally, 178 patients (43 males and 135 females) with a mean age of 45.57 ± 13.00 years (range 9 to 73 years) who met the inclusion and exclusion criteria were enrolled in the study. The details of the included and excluded patients are shown in Figure 1. A total of 178 patients with PTC were divided into two groups: LNM (+) (n = 100) and LNM (−) (n = 78). Chi-square tests compared the clinical and radiological parameters between the LNM (+) and LNM (−) groups. The clinical and radiological factors with statistical difference were identified as the most diagnostic clinical and radiological factors. Clinical information, including age, sex, body mass index (BMI), thyroid hormone level and presence of nodular goiters or Hashimoto's thyroiditis, was obtained through the medical records system. According to clinical experience, the normal standards for BMI, thyroglobulin (TG), thyroid-stimulating hormone (TSH), free triiodothyronine (FT3), free thyroxine (FT4), thyroglobulin antibody (TGAb) and thyroid peroxidase antibody (TPOAb) were as follows: BMI, 18.5-23.9; TG, 3.5-77 ng/mL; TSH, 0.27-4.2 uIU/mL; FT3, 3.1-6.8 pmol/L; FT4, 12-22 pmol/L; TGAb, 0-115 IU/mL; TPOAb, 0-34 IU/mL. Any values outside these ranges were considered abnormal. Then, according to the age and sex listed in the clinical data, the patients were randomly divided into a training cohort (n = 125) and test cohort (n = 53) with a ratio of 7:3.

CT Acquisition
Preoperative noncontrast and dual-phase contrast-enhanced CT scanning was performed with a third-generation Siemens dual-source CT device (Somatom Definition, Siemens, Germany) or a Philips spectral CT device (IQon spectral CT, Philips, The Netherlands). The patients were placed in a supine head-first position. The scanning range was from the skull base to the upper edge of the aortic arch. The patients were instructed not to swallow during scanning and to drop their shoulders as much as possible to avoid swallowing and clavicle artifacts. The CT parameters were as follows: (1) A and B X-ray tube voltages of 90 kV and Sn150 kV, respectively; automatic tube current regulation technology; pitch factor, 1.5; rotation time, 0.25 s; detector collimation, 192 × 0.6 mm; slice thickness, 2 mm; and reconstruction slice thickness, 0.75 mm; (2) tube voltage, 120 kV; tube current, 50-350 mAs; pitch factor, 0.969; rotation time, 0.5 s; detector collimation, 64 × 0.625 mm; slice thickness, 2 mm; and reconstruction slice thickness, 0.75 mm. After routine noncontrast CT scanning, 60 mL of iodinated nonionic contrast medium (iodixanol injection, Hengrui Medicine, China) was injected through the cubital vein at a flow rate of 3.5 mL/s with a concentration of 320 mg I/mL. Then, the patients underwent contrast-enhanced scanning 25 s (arterial phase) and 50 s (venous phase) after injection.

Radiologist Assessment of the Primary Tumors and LNMs
For the morphological evaluation of the primary tumors, two radiologists (5 and 7 years of diagnostic experience) observed and recorded the location of the primary PTC tumor (right lobe, left lobe or isthmus) and measured the anteroposterior diameter (AD), transverse diameter (TD), anteroposterior to transverse diameter ratio (A/T) and degree of calcification and capsule invasion. In cases of disagreement on qualitative indicators, a consensus was reached through discussion. The mean value of the quantitative indicators was taken as the final value. The anteroposterior and transverse diameters were recorded as greater or less than 6 mm. The A/T ratio was recorded as greater or less than 1 [39].
Three radiologists with 15, 7 and 5 years of diagnostic experience in head and neck cancers independently and blindly assessed lymph node status, and their diagnosis results are referred to as CT-reported LN status 1, CT-reported LN status 2 and CT-reported LN status 3, respectively. According to the National Comprehensive Cancer Network (NCCN) guidelines and previous studies [40,41], LNM was considered for patients who met at least one of the following conditions: (1) lymph node size larger than 10 mm; (2) round or irregular shape; (3) obscure boundary or encroachment of adjacent tissues; (4) calcified, cystic or necrotic tissue; (5) obvious enhancement; and (6) inhomogeneous enhancement. Figure 2 shows a typical case of LNM and one without LNM on CT.

CT Acquisition
Preoperative noncontrast and dual-phase contrast-enhanced CT scanning was performed with a third-generation Siemens dual-source CT device (Somatom Definition, Siemens, Germany) or a Philips spectral CT device (IQon spectral CT, Philips, The Netherlands). The patients were placed in a supine head-first position. The scanning range was from the skull base to the upper edge of the aortic arch. The patients were instructed not to swallow during scanning and to drop their shoulders as much as possible to avoid swallowing and clavicle artifacts. The CT parameters were as follows: (1) A and B X-ray tube voltages of 90 kV and Sn150 kV, respectively; automatic tube current regulation technology; pitch factor, 1.5; rotation time, 0.25 s; detector collimation, 192 × 0.6 mm; slice thickness, 2 mm; and reconstruction slice thickness, 0.75 mm; (2) tube voltage, 120 kV; tube current, 50-350 mAs; pitch factor, 0.969; rotation time, 0.5 s; detector collimation, 64 × 0.625 mm; slice thickness, 2 mm; and reconstruction slice thickness, 0.75 mm. After routine noncontrast CT scanning, 60 mL of iodinated nonionic contrast medium (iodixanol injection, Hengrui Medicine, China) was injected through the cubital vein at a flow rate of 3.5 mL/s with a concentration of 320 mg I/mL. Then, the patients underwent contrast-enhanced scanning 25 s (arterial phase) and 50 s (venous phase) after injection.

Radiologist Assessment of the Primary Tumors and LNMs
For the morphological evaluation of the primary tumors, two radiologists (5 and 7 years of diagnostic experience) observed and recorded the location of the primary PTC tumor (right lobe, left lobe or isthmus) and measured the anteroposterior diameter (AD), transverse diameter (TD), anteroposterior to transverse diameter ratio (A/T) and degree of calcification and capsule invasion. In cases of disagreement on qualitative indicators, a consensus was reached through discussion. The mean value of the quantitative indicators was taken as the final value. The anteroposterior and transverse diameters were recorded as greater or less than 6 mm. The A/T ratio was recorded as greater or less than 1 [39].
Three radiologists with 15, 7 and 5 years of diagnostic experience in head and neck cancers independently and blindly assessed lymph node status, and their diagnosis results are referred to as CT-reported LN status 1, CT-reported LN status 2 and CT-reported LN status 3, respectively. According to the National Comprehensive Cancer Network (NCCN) guidelines and previous studies [40,41], LNM was considered for patients who met at least one of the following conditions: (1) lymph node size larger than 10 mm; (2) round or irregular shape; (3) obscure boundary or encroachment of adjacent tissues; (4) calcified, cystic or necrotic tissue; (5) obvious enhancement; and (6) inhomogeneous enhancement. Figure 2 shows a typical case of LNM and one without LNM on CT.

Tumor Segmentation and Feature Extraction
Manual segmentation was performed slice by slice on the unenhanced CT images and arterial and venous contrast-enhanced CT images by a radiologist with 15 years of diagnostic experience using ITK-SNAP (www.itksnap.org accessed on 25 April 2022) to obtain the tumor regions of interest (ROIs). The three-phase CT images were delineated with a constant window width (350 HU) and window level (60 HU) without avoiding cystic areas, areas of necrosis, hemorrhage and calcification. The following guideline was followed when performing ROI segmentation: the three-phase CT images were compared when the boundary of the lesion was not clear, and areas with the same enhancement pattern were included in the ROI when the lesion invaded the surrounding tissue. Figure 3 shows the 3D ROI segmentation of primary PTC tumors on three-phase CT images.

Tumor Segmentation and Feature Extraction
Manual segmentation was performed slice by slice on the unenhanced CT images and arterial and venous contrast-enhanced CT images by a radiologist with 15 years of diagnostic experience using ITK-SNAP (www.itksnap.org accessed on 25 April 2022) to obtain the tumor regions of interest (ROIs). The three-phase CT images were delineated with a constant window width (350 HU) and window level (60 HU) without avoiding cystic areas, areas of necrosis, hemorrhage and calcification. The following guideline was followed when performing ROI segmentation: the three-phase CT images were compared when the boundary of the lesion was not clear, and areas with the same enhancement pattern were included in the ROI when the lesion invaded the surrounding tissue. Figure 3 shows the 3D ROI segmentation of primary PTC tumors on three-phase CT images. After tumor segmentation, radiomic feature extraction was automatically performed within the ROI on each of the three-phase CT images using FeAture Explorer (FAE) software [42], based on the Pyradiomics open source module [43]. A total of 851 features were extracted from the ROI in each phase, including 18 first-order, 14 shape, 24 gray-level cooccurrence matrix (GLCM), 16 gray-level run-length matrix (GLRLM), 16 gray-level size zone matrix (GLSZM), 5 neighborhood gray-tone difference matrix (NGTDM), 14 gray-level dependent matrix (GLDM) and 744 wavelet features. Finally, a total of 2553 features were extracted from the three-phase CT images of each patient. To evaluate the stability and reproducibility of the radiomics features, 20 patients were randomly selected for a double-blinded comparison of the manual segmentations by two After tumor segmentation, radiomic feature extraction was automatically performed within the ROI on each of the three-phase CT images using FeAture Explorer (FAE) software [42], based on the Pyradiomics open source module [43]. A total of 851 features were extracted from the ROI in each phase, including 18 first-order, 14 shape, 24 gray-level cooccurrence matrix (GLCM), 16 gray-level run-length matrix (GLRLM), 16 gray-level size zone matrix (GLSZM), 5 neighborhood gray-tone difference matrix (NGTDM), 14 graylevel dependent matrix (GLDM) and 744 wavelet features. Finally, a total of 2553 features were extracted from the three-phase CT images of each patient. To evaluate the stability and reproducibility of the radiomics features, 20 patients were randomly selected for a double-blinded comparison of the manual segmentations by two radiologists, and features with intraclass and interclass correlation coefficients (ICCs) lower than 0.75 were removed for further analysis.

Feature Selection and Model Construction
After the stability and reproducibility evaluations, the remaining features were normalized by transforming the data into new scores using mean or z-score normalization (with a mean of 0 and a standard deviation of 1). We chose principal component analysis (PCA) or Pearson's correlation coefficient (PCC) feature dimension reduction methods to remove redundant features, and then ANOVA or Kruskal-Wallis test methods were used to select the most useful predictive LNM-related features.
We constructed three clinical-radiological models based on the clinical factors and CT-reported lymph node status assessed by three radiologists with 5, 7 and 15 years of diagnostic experience. Univariable analyses were used to identify the clinical-radiological factors associated with LNMs. A noncontrast radiomics model, arterial contrast radiomics model and venous contrast radiomics model were constructed based on the corresponding selected feature subsets, and then a three-phase radiomics model was constructed based on the selected features of all three feature subsets. Radiomics scores (Rad scores) were calculated using a linear combination of the final selected features with their corresponding weights to build a radiomics signature. Finally, a combined model integrating the most diagnostic clinical factors, radiological characteristics and the radiomics signature was constructed. All models were developed using multiple logistic regression algorithms. We used internal 5-fold cross-validation on the training cohort for hyperparameter optimization and to train the optimal model. The predictive performances of the different models were evaluated using receiver operating characteristic (ROC) curve analysis, quantified by area under the curve (AUC) and accuracy (ACC), and compared through the DeLong test. The feature selection and model construction were performed using FAE software, which provides a radiomics pipeline to develop radiomic models with different combinations of feature normalization methods, feature selection algorithms and classifiers [41]. The best model configuration was determined based on the highest AUC value in the test cohort.
A nomogram for the combined model was generated to provide clinicians with an individualized and visual tool for diagnosing LNM in PTC. The clinical and radiological parameters were identified in the nomogram model by using binary logistic regression analysis, and the forward selection method was used to screen the variables to construct the nomogram as potential predictors (enter value: 0.05, remove value: 0.10). To evaluate the agreement between the nomogram-predicted and actual LNM probabilities, calibration curves were plotted, and the Hosmer-Lemeshow test was applied in both the training cohort and test cohort. Decision curve analysis was performed to compare the combined model, the three-phase radiomics model and the clinical-radiological model in terms of clinical usefulness by quantifying the net benefits at different threshold probabilities in the test cohort.

Statistical Analysis
Feature extraction, feature selection and model construction were performed with FAE software (https://github.com/salan668/FAE accessed on 25 April 2022). We performed the statistical analysis with SPSS software (version 23.0) and R software (version 3.6.1, https://www.r-project.org accessed on 25 April 2022). For continuous variables, Student's t-test or the Kruskal-Wallis test was used to assess the statistical significance of differences between groups. For categorical variables, Pearson's chi-square test was used. A two-sided p-value less than 0.05 was considered statistically significant.

Patient Characteristics and the Clinical-Radiological Models
The characteristics of the PTC patients in the LNM (+) and LNM (−) groups are shown in Table 1. Age, capsule, AD, TD, A/T, CT-reported LN status 1 and CT-reported LN status 3 were significantly different between the two groups (p < 0.05). Table 2 shows the characteristics of the patients in the training cohort and test cohort. There were no significant differences between the two cohorts in any of the clinical, radiological and pathological characteristics (all p > 0.05). Table 3 shows the association between the actual LNM state and patient characteristics in the training cohort and test cohort. Clinicalradiological model 1 was established with two features and achieved an AUC of 0.781 in the training cohort and 0.758 in the test cohort. Clinical-radiological model 2 was established with seven features and achieved an AUC of 0.796 in the training cohort and 0.729 in the test cohort, and clinical-radiological model 3 was established with six features and achieved an AUC of 0.800 in the training cohort and 0.743 in the test cohort. The coefficients of the features in the clinical-radiological models are shown in Supplementary Table S1.

Feature Selection and Radiomic Models
Most of the radiomics features that showed favorable stability and reproducibility (ICC > 0.75) in the intraobserver and interobserver agreement assessment were selected for further analysis, including 794 (93.9%) features from the noncontrast CT images, 770 (90.5%) features from the arterial contrast-enhanced CT images and 778 (91.2%) features from the venous contrast-enhanced CT images. After feature reduction and selection, 16 features were selected to construct the noncontrast model, 15 were selected to construct the arterial contrast model and 11 were selected to construct the venous contrast model. Of the combined 42 features of the three models, 14 were further selected by PCC and ANOVA to construct the three-phase radiomics model using multiple logistic regression. After integrating the most diagnostic clinical factors, radiological characteristics and radiomics features, 10 features were selected by PCA and ANOVA methods to construct the combined model using multiple logistic regression. The coefficients of the selected radiomics features in the different models are shown in Supplementary Table S2.
The predictive performances of the different models are shown in  In order to evaluate the diagnostic performance of the combined model in different LNM location subgroups, all of the 100 patients with LNM were divided into three groups: central LNM (33 cases), lateral LNM (27 cases) and central and lateral LNM (40 cases). The diagnostic performance of the combined model was evaluated though a stratified analysis in the central LNM, lateral LNM and central and lateral LNM subgroups. To ensure the balance of the dataset, we randomly selected 33 patients, 27 patients and 40 patients in the non-metastatic group (78 cases), respectively, as the control groups. Table 5 shows the diagnostic performance of the combined model in different LN location subgroups. The combined model has a certain diagnostic value in predicting central LNM, lateral LNM and central and lateral LNM for PTC patients. The AUC of the combined model in the central LNM prediction (AUC = 0.833) was similar to that of the lateral LNM prediction (AUC = 0.823). However, the sensitivity of the combined model in the central LNM prediction (78.79%) is higher than that of the lateral LNM prediction (66.67%), and the specificity of the combined model in the central LNM prediction (72.73%) is relatively lower than that of the lateral LNM prediction (81.48%). Each parameter of the combined model in the central and lateral LNM prediction is more than 85%, and the AUC is as high as 0.960. The combined model had the most sensitive and specific for diagnosing patients with central and lateral LNM.

Radiomics Nomogram and Clinical Utility
The radiomics signature, age, AD, A/T and CT-reported LN status 1 were identified as independent predictive factors for LNM by multivariable logistic regression analysis to construct the combined model. A radiomics nomogram was developed by integrating the radiomics signature and clinical-radiological factors (Figure 4a). The calibration curves of the radiomics nomogram showed good agreement between the predicted LNM probabilities and the actual pathological findings in both the training cohort ( Figure 4b) and test cohort (Figure 4c), and the Hosmer-Lemeshow test yielded p-values of 0.454 and 0.248, respectively. In the training cohort, the combined nomogram integrating clinical-radiological factors and the radiomics signature had a significantly better predictive performance than the best clinical-radiological model (AUC, 0.868 vs. 0.781, DeLong test p = 0.003; ACC, 86.83% vs. 75.20%) (Figure 5a). In the test cohort, the combined nomogram also showed better performance than the clinical-radiological model (AUC, 0.878 vs. 0.758, DeLong test p = 0.017; ACC, 83.02% vs. 73.58%) (Figure 5b). The decision curve analysis for the combined nomogram, the three-phase radiomics model and the clinical-radiological model are illustrated in Figure 6. Among the three methods, the combined model showed a higher overall net benefit than the other two models. When the threshold probability was between 0.1 and 0.18, using the nomogram based only on the three-phase radiomics model offered a higher net benefit than treating all patients or treating no patients. When the threshold probability was between 0.18 and 0.25, a higher net benefit was obtained by using only the nomogram based on the clinical-radiological model. The combined nomogram exhibited a greater net benefit in predicting LNM than the three-phase radiomics model and the clinical-radiological model for a threshold probability higher than 24%.
Diagnostics 2022, 12, x FOR PEER REVIEW 13 three-phase radiomics model offered a higher net benefit than treating all patien treating no patients. When the threshold probability was between 0.18 and 0.25, a hi net benefit was obtained by using only the nomogram based on the clinical-radiolo model. The combined nomogram exhibited a greater net benefit in predicting LNM the three-phase radiomics model and the clinical-radiological model for a thres probability higher than 24%.  Calibration curve of the nomogram of the combined model in the test cohort (p = 0.248). Calibration curves describe the model's calibration in terms of agreement between the predicted probability of LNM and the actual lymph node status. The dotted line represents perfect performance, the gray solid line represents the actual performance and the black solid line represents the corrected prediction performance of the nomogram of the combined model. Abbreviations: LNM, lymph node metastasis; AD, anteroposterior diameter; A/T, anteroposterior to transverse diameter ratio; CT, computed tomography.

Discussion
In our study, a comprehensive analysis integrating CT radiomic features, clinical factors and radiological characteristics was performed to preoperatively predict the LNM status of PTC patients. The results showed that CT radiomics features provided a similar discriminative value to clinical-radiological factors in both the training and test cohorts. However, the combined model integrating the clinical-radiological factors and radiomics features had a significantly improved predictive performance. Finally, a combined nomogram was established for individualized predicting of LNM probability in PTC, showing good agreement in the calibration and the best net benefit in the decision curve analysis. Our findings support the incorporation of radiomics analysis into the clinical workflow to improve the preoperative diagnosis of LNM in patients with PTC.
At present, although there is controversy in clinical practice about the possible impact of the radiation dose caused by the use of enhanced CT, according to the American Thyroid Association guidelines, CT is an effective adjunct imaging modality to US for patients with clinically suspected disease progression, such as aggressive primary tumor or enlarged LNM [7,44]. Several previous studies suggest that it is not necessary to delay radioactive iodine therapy when concerned about the excessive iodine content of contrast-enhanced CT, because the iodine clears up in 4-8 weeks [45,46]. Therefore, recent

Discussion
In our study, a comprehensive analysis integrating CT radiomic features, clinical factors and radiological characteristics was performed to preoperatively predict the LNM status of PTC patients. The results showed that CT radiomics features provided a similar discriminative value to clinical-radiological factors in both the training and test cohorts. However, the combined model integrating the clinical-radiological factors and radiomics features had a significantly improved predictive performance. Finally, a combined nomogram was established for individualized predicting of LNM probability in PTC, showing good agreement in the calibration and the best net benefit in the decision curve analysis. Our findings support the incorporation of radiomics analysis into the clinical workflow to improve the preoperative diagnosis of LNM in patients with PTC.
At present, although there is controversy in clinical practice about the possible impact of the radiation dose caused by the use of enhanced CT, according to the American Thyroid Association guidelines, CT is an effective adjunct imaging modality to US for patients with clinically suspected disease progression, such as aggressive primary tumor or enlarged LNM [7,44]. Several previous studies suggest that it is not necessary to delay radioactive iodine therapy when concerned about the excessive iodine content of contrast-enhanced CT, because the iodine clears up in 4-8 weeks [45,46]. Therefore, recent guidelines and studies suggest CT for the detection of metastatic lymph nodes in patients with PTC [47,48]. However, routine clinical US and CT diagnoses are greatly affected by the clinical experience and subjectivity of the radiologist. We hope to create a more objective and comprehensive method to predict LNM in PTC patients.
To study the efficacy of routine CT diagnosis combined with clinical risk factors in predicting LNM in PTC patients, we retrospectively collected 18 clinical factors and combined them with three CT-reported LN statuses diagnosed by three radiologists with 15, 7 and 5 years of diagnostic experience in head and neck cancers to build three clinicalradiological models. Chi-square tests showed that there were significant differences in age, capsule, location, AD, TD, A/T, CT-reported LN status 1 and CT-reported LN status 3 between the metastatic group and the non-metastatic group. Hît , u et al. found that the total tumor diameter and unilateral multifocality were independent predicting factors of metastatic papillary thyroid microcarcinoma [49]. Our study shows roughly the same results, that AD and TD were the independent radiological risk factors for diagnosing LNM of PTC. Since patients with multifocality were excluded in our study, whether unilateral multifocality is an independent predictor factor needs further verification. To explore the efficacy of radiologists with different working experience in diagnosing cervical LNM in patients with PTC, CT-reported LN status 2 was also included in the clinical-radiological model 2. In the test cohort, the clinical-radiological model 1 showed the best performance with AUC of 0.758, ACC of 73.58%, sensitivity of 80.00% and relatively low specificity of 65.22%. The sensitivities of the three clinical-radiological models were relatively higher (73.33-80.00%), the specificities were lower (65.22-69.57%) and the accuracy was from 69.81% to 73.58%. Despite combining the clinical risk factors with the diagnostic result of 15 years of diagnostic experience, the clinical-radiological model had a relatively lower diagnostic efficacy in predicting LNM in PTC patients.
For the radiomics model, the high-throughput feature extraction and selection is the key to radiomics model construction. These high-dimensional imaging features cannot be directly recognized by the naked eye. High-throughput feature extraction and selection captures the heterogeneity in the lesions in a non-invasive way, quantifies the deep-seated features and forms a potential database. Then, we can select the features with the most tumor information and analyze the relationship between these features and the LNM results by using the machine learning method of multiple logistic regression analysis to build the predictive radiomics model. In this study, we built four radiomics models: noncontrast model, arterial contrast model, venous contrast model and three-phase radiomics model. PCC or PCA dimensionality reduction method and multiple logistic regression analysis were used to establish these models. Nine of the fourteen radiomics features that constructed the three-phase radiomics model belong to shape features, such as the major axis length, minor axis length, sphericity, surface volume ratio and maximum 2D diameter row. This may indicate that the shape of the tumor is closely related to LNM of PTC patients. There were four radiomics features belonging to gray-level size zone matrix (GLSZM) texture features and one belonging to gray-level cooccurrence matrix (GLCM) texture features after the wavelet transform of the original CT images. These texture features represented noise removal and edge enhancement, and the radiomics signature combining these features from three-phase images describing different aspects of tumor appearance might capture hidden characteristics, offer insight into the heterogeneity of the tumor microenvironment (calcification, bleeding, cystic change, etc.) and thus create a more accurate model to predict the LNM of PTC patients. The diagnostic performances of the four radiomics models were close. Although the radiomic features of the three phase images were combined, no significant improvement in the diagnostic performance of the three-phase radiomics model was found. In the test cohort, the sensitivities of the noncontrast model, venous contrast model and three-phase radiomics model were relatively higher (80.00-86.67%), the specificities were relatively lower (60.87-78.26%), the accuracies were from 73.58% to 79.25% and the AUCs were from 0.781 to 0.813. The arterial contrast model had higher specificity (86.96%) but lower sensitivity (66.67%), although the Delong test found that the diagnostic performance of the radiomics model was similar to that of the clinical-radiological model. The radiomics method based on the computer image processing and machine learning modeling has some special advantages for the prediction of LNM in PTC patients, such as being more objective, more automated and less time-consuming.
Radiomics nomograms provide physicians with a visual and quantitative tool to identify LNM in patients with PTC and present recommendations/guidance for clinical decision making [29,50]. Lu et al. proposed a radiomics nomogram that incorporated radiomics features, CT-reported LN status, sex and age for the preoperative prediction of cervical LN metastasis in patients with PTC [37]. Zhou et al. proposed a radiomics nomogram that incorporated conventional CT images with radiomics features dual-energy CT-derived iodine maps in diagnosing cervical LNM in patients with PTC [38]. These two successful studies indicated the feasibility of applying radiomics to predict LN status in PTC. In our study, when the most diagnostic clinical factors, radiological characteristics and radiomics features were integrated, the combined nomogram yielded the best predictive performances with an AUC, ACC, sensitivity and specificity of 0.868, 86.83%, 88.57% and 70.91%, respectively, in the training cohort and 0.878, 83.02%, 90.00% and 73.91%, respectively, in the test cohort. Compared with the previous CT radiomics (AUC = 0.822 in the test cohort) [37] and ultrasound radiomics (AUC = 0.727 in the test cohort) [36] research for the prediction of LNM in PTC patients, our model achieved good diagnostic efficacy. Decision curve analysis was used to demonstrate the potential application value of our nomogram; this analysis confirmed the increased value of our research method combining radiomics features and clinical risk predictors for the prediction of LNM in PTC through the net benefit according to the threshold probability. Although the combined model has good diagnostic efficacy in preoperative prediction of LNM of PTC patients, this study is not intended to replace the radiologist's diagnosis. We focus more on exploring the possibility of embedding radiomics technologies based on computer image processing into clinical workflows, and identifying the increased value of integrating CT radiomics signature with radiologist diagnosis and clinical factors (age, AD and A/T) to construct a combined model for the accurate prediction of LNM of PTC patients. Our proposed combined nomogram can give radiologists more confidence to make accurate diagnoses for LNM states and give surgeons more confidence to develop optimal treatment strategies and assess the prognosis of PTC patients.
Although the proposed comprehensive radiomics analysis has certain advantages over conventional clinical and radiological factors, the limitations of our study merit discussion. First, our model was trained and tested based on retrospectively collected datasets from a single center, and the reproducibility and robustness of the model must be externally validated through multicenter and prospective studies. Second, the heterogeneity of the imaging parameters between two different CT machines may have a certain impact on the results. Previous studies have shown that CT radiomics features could be affected by slice thickness, bin width, voxel size, number of gray levels, etc., in images from different CT scanners [51]. Appropriate image preprocessing, such as voxel size normalization and gray level normalization, can reduce the impact of image parameter changes on the variability of radiomic features [52]. In our study, although all CT images were normalized to 0-256 gray levels before feature extraction, the voxel size normalization was not applied to the CT images. We think this may affect the stability of the texture features, and thus the prediction efficiency of the radiomics model. As the sample size of our dual-source CT cases increases, the extent of this impact will be investigated in future research. Third, radiomics analysis was only performed on conventional and enhanced CT images. In future studies, we will incorporate iodine maps and other spectral images from dual-energy CT in radiomics analysis to mine valuable information and improve the predictive performance for LNM in PTC.

Conclusions
In conclusion, we developed a combined model based on CT imaging features and clinical risk factors to predict cervical LNM in patients with PTC. Radiomics analysis plays an important role in diagnosing cervical LNM in PTC, and our combined model performed better than the other models. The nomogram of the combined model based on preoperative CT can improve the accuracy of the prediction of cervical LNM in PTC patients and help clinicians make more reasonable clinical diagnosis and treatment decisions.
Supplementary Materials: The following supporting information can be downloaded at: https: //www.mdpi.com/article/10.3390/diagnostics12051119/s1, Table S1: The coefficients of features in the clinical-radiological model; Table S2: The coefficients of radiomics features in the different models.  Informed Consent Statement: Informed consent was obtained from all subjects involved in the study.

Data Availability Statement:
The datasets used and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Conflicts of Interest:
The authors declare no conflict of interest.