Next Article in Journal
Evaluation of SARS-CoV-2 Detection Systems Using Clinical Samples and Standard Material: A Comparative Study
Next Article in Special Issue
Computed Tomography-Based Radiomics Using Tumor and Vessel Features to Assess Resectability in Cancer of the Pancreatic Head
Previous Article in Journal
Usefulness of Endoscopic Retrograde Cholangiopancreatography in the Diagnosis and Treatment of Traumatic Pancreatic Injury in Children
Previous Article in Special Issue
Radiomics in Oncology III
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

Image Guided Radiotherapy (IGRT) and Delta (Δ) Radiomics—An Urgent Alliance for the Front Line of the War against Head and Neck Cancers

by
Camil Ciprian Mireștean
1,2,
Roxana Irina Iancu
3,4,* and
Dragoș Petru Teodor Iancu
5,6
1
Department of Oncology and Radiotherapy, University of Medicine and Pharmacy Craiova, 200349 Craiova, Romania
2
Department of Surgery, Railways Clinical Hospital Iasi, 700506 Iași, Romania
3
Oral Pathology Department, “Gr. T. Popa” Faculty of Dental Medicine, University of Medicine and Pharmacy, 700115 Iași, Romania
4
Department of Clinical Laboratory, “St. Spiridon” Emergency Universitary Hospital, 700111 Iași, Romania
5
Oncology and Radiotherapy Department, Faculty of Medicine, “Gr. T. Popa” University of Medicine and Pharmacy, 700115 Iași, Romania
6
Department of Radiation Oncology, Regional Institute of Oncology, 700483 Iași, Romania
*
Author to whom correspondence should be addressed.
Diagnostics 2023, 13(12), 2045; https://doi.org/10.3390/diagnostics13122045
Submission received: 15 March 2023 / Revised: 24 May 2023 / Accepted: 9 June 2023 / Published: 13 June 2023
(This article belongs to the Special Issue Radiomics in Oncology 3rd Edition)

Abstract

:
The identification of a biomarker that is response predictive could offer a solution for the stratification of the treatment of head and neck cancers (HNC) in the context of high recurrence rates, especially those associated with loco-regional failure. Delta (Δ) radiomics, a concept based on the variation of parameters extracted from medical imaging using artificial intelligence (AI) algorithms, demonstrates its potential as a predictive biomarker of treatment response in HNC. The concept of image-guided radiotherapy (IGRT), including computer tomography simulation (CT) and position control imaging with cone-beam-computed tomography (CBCT), now offers new perspectives for radiomics applied in radiotherapy. The use of Δ features of texture, shape, and size, both from the primary tumor and from the tumor-involved lymph nodes, demonstrates the best predictive accuracy. If, in the case of treatment response, promising Δ radiomics results could be obtained, even after 24 h from the start of treatment, for radiation-induced xerostomia, the evaluation of Δ radiomics in the middle of treatment could be recommended. The fused models (clinical and Δ radiomics) seem to offer benefits, both in comparison to the clinical model and to the radiomic model. The selection of patients who benefit from induction chemotherapy is underestimated in Δ radiomic studies and may be an unexplored territory with major potential. The advantage offered by “in house” simulation CT and CBCT favors the rapid implementation of Δ radiomics studies in radiotherapy departments. Positron emission tomography (PET)-CT Δ radiomics could guide the new concepts of dose escalation on radio-resistant sub-volumes based on radiobiological criteria, but also guide the “next level” of HNC adaptive radiotherapy (ART).

1. Introduction

Concurrent chemo-radiotherapy (CCRT) as a single treatment, or induction chemotherapy (IC) followed by CCRT, are part of the approach strategies for locally advanced head and neck squamous cell carcinoma (LA-HNSCC). The identification of a biomarker predictive of response could provide a solution for treatment stratification, in the context of high recurrence rates, especially for those associated with loco-regional recurrence. The proposed concept of radiomics involves extracting a large volume of data from medical imaging to improve diagnostic accuracy, but also to create prognostic and predictive models. We propose to identify arguments for the use of a new biomarker (variation Δ of radiomics features during treatment) predictive of the response of head and neck cancers to multimodal treatment by chemo-radiotherapy. The guiding of treatment based on a biomarker would have as a consequence the implementation of de-escalation strategies in order to limit toxicities or to escalate the treatment dose to increase the treatment response rate. Likewise, the prediction, by radiomics models, of the risk of toxicity, such as xerostomia, would have the consequence of replanning in order to limit doses to certain radiosensitive structures [1,2,3].

2. Aims and Scope

We proposed to synthesize, in a narrative review, the data that could be considered relevant for the implementation of the Δ radiomics concept, based on the variation of radiomics features between different moments of the acquisition of medical images for the purpose of treatment stratification. We performed a search in the MedLine database using the PubMed search interface with keywords delta, Δ, radiomics, head and neck cancers, HNC, HNSCC, biomarkers, and radiotherapy. The studies that were considered relevant were selected and analyzed and the data referring to radiomics (not featuring Δ variation), HNC diagnosis, or deep learning were excluded.

3. Radiomics—An Emerging Role in HNC

In the last decade, the development of image analysis processes and the rapid increase in the volume of high-quality medical data have facilitated the extraction of quantitative features from high-resolution medical imaging. The transformation of these images into data, invisible to the eye of the examiner, was called radiomics by Gillies and colleagues. The authors particularly note the potential of the method to facilitate clinical decisions and to improve the management of cancer patients. Even if radiomics is frequently used in clinical research and especially in oncology, the value of an analysis of feature variation (Δ-radiomics) during treatment should not be neglected, generally having pre-treatment and post-treatment images as references. If, in the early stages, the therapeutic success rate is relatively high, for LA-HNSCC cases treated by combining radiotherapy with Cisplatin or Anti-EGFR (epidermal growth factor receptor) Cetuximab, 40% of cases will not respond to treatment. After 2–3 years, 50–60% will present loco-regional recurrence, but even the rates of distant failure through metastasis of 20–30%, or 2–4% per year, of the second primary cannot be ignored [1,2,3,4].
Radiomics has two major advantages: it could be more specific, but it is also non-invasive. As a marker of tumor heterogeneity, radiomics has an advantage over biopsy due to the possibility of evaluating the tumor in toto. Whether in computer tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET) or even ultrasonography (US), mammography, or digital radiographs (XRD), medical images are analyzed and extracted radiomic features could be used to build diagnostic, predictive or prognostic models [1,5].
Radiomics is therefore the ideal answer for a partnership between the concept of precision medicine and, in particular, of precision oncology and non-invasive biomarkers that could provide a digital tumor phenotype. Using mathematical algorithms, radiomics is based on certain steps, including the initial selection of images, the delimitation of the region of interest (ROI) for two-dimensional images or of the volumes of interest (VOI) for three-dimensional images selected manually, semi-automatically or automatically. Image processing and feature extraction, image analysis and the construction of a model are also essential steps of radiomics. Jha et al. consider that the digital tumor phenotype offered by radiomics will play a decisive role in precision oncology in the future. Radiomics could provide essential data not only about tumor heterogeneity, but also about the tumor microenvironment (TME) [6,7].
The same applications as in the case of radiomics, including differential diagnosis, prognosis biomarker, prediction of treatment response, but also those anticipating the risk of toxicities associated with the treatment, can be translated to the Δ radiomics concept. Based on recordings of tumor heterogeneity before and after treatment, Δ radiomics analysis could offer a new perspective for chemotherapy and radiotherapy response prediction in HNC. A systematic review, that aimed to identify Δ radiomics studies from Embase, PubMed, and ScienceDirect databases, identified forty-eight studies that corresponded to the selection criteria, of which six studies, representing 12.5%, are related to HNC. It should be noted that the large number of Δ radiomics studies is equal to the number of studies that analyze the radiomic feature dynamics in gastrointestinal cancers (other than rectal cancer) and in rectal cancer. Only in the case of lung cancer were a doubled number of Δ radiomics analysis studies identified compared to HNC [2,6,8,9].

4. The Multidisciplinary Approach in Locally Advanced HNC: Challenges in ERA Precision Oncology

More than 20 years ago, the benefit of the multimodal approach in locally advanced HNC was demonstrated. The prospective randomized multicenter trial led by Wendt and colleagues demonstrates a net benefit and higher loco-regional control rates at 3 years (36% vs. 17%) in favor of a concomitant chemotherapy regimen, based on Cisplatin, 5-Fluorouracil, and Leucovorin, versus radiotherapy alone. Furthermore, treatment-related toxicity rates were considered relatively high, especially in regard to grade 3 or 4 mucositis, which were associated with 38% of cases with concurrent chemotherapy treatment. In the study arm treated with radiotherapy alone, the rate of moderate and severe mucositis was reported as 17%. Treatment breaks or postponements were also significantly more associated with concurrent treatment due to hematological toxicity [10].
Thus, considering that treatment gaps could lead to a severe compromise of loco-regional control, especially if gaps occurred before day 19 after the start of therapy, identifying a balanced regimen that would obtain the best therapeutic ratio is still a challenge. A gap in treatment of 1 day is considered detrimental, but the effect is more intense for >3 days of interruption of radiotherapy. An equivalent dose in 2Gy fractions (EQD2), based on the linear quadratic model (LQ), is widely used to compensate for breaks during the radiotherapy treatment, but the differences in cell repopulation from one case to another make this strategy imprecise. Another method of compensating radiotherapy treatment gaps is the administration of two daily doses at an interval of at least 6 h. Both strategies could create an increase in the risk of toxicities by escalating doses received by organs at risk (OARs) [11,12,13,14].
One of the controversies is associated with the administration of the regimen of 40 mg/m2 doses of Cisplatin weekly versus Cisplatin at 100 mg/m2 given every 3 weeks, both delivered concurrently with radiotherapy. Without a doubt, a dose of 30 mg/m2 weekly has been proven to be inferior to a dose of 100 mg/m2, but weekly administration could bring a radio-sensitization benefit with improvements in the toxicity profile. By evaluating, a randomized controlled phase III multicenter study comparing the two regimens of chemotherapy plus definitive radiotherapy in a total dose of 70 Gy in 35 fractions for 7 weeks, delivered only by intensity modulated irradiation techniques in cases of HNC stage III and IV non-nasopharyngeal, the authors considered an irradiation dose of at least 60 Gy, and a cumulative Cisplatin dose of at least 200 mg/m2, as a complete treatment. A 5% (94% vs. 89%) higher treatment completion rate in favor of the arm that received weekly Cisplatin treatment, and a 3% higher treatment completion rate (80% vs. 77%) in favor of the same regimen, justifies the consideration of weekly Cisplatin as a standard regimen. Totals of 61.8% vs. 53.4% in loco-regional control at two years, again in favor of the weekly Cisplatin regimen, is also an argument supporting it. It is still necessary to mention that, considering that, especially in developed countries, there is a rapid increase in the incidence of HNC associated with human papillomavirus (HPV) infection, a different disease from HNC related to a long history of smoking, in India, where the study was carried out, the percentage of HNC cases associated with HPV infection is still relatively low (10–15%) [15,16,17,18].
Cetuximab remains a treatment option for head and neck squamous cell carcinoma (HNSCC) ineligible for platinum-based therapy. Even if the phase 3 randomized trial conducted by Bonner et al. demonstrates a net advantage in OS at 5 years versus radiotherapy alone, Cetuximab remains only a feasible alternative to Cisplatin, with a different toxicity profile from chemotherapy that induces a skin rash. A De-ESCALaTE and RTOG 1016 trial proposed treatment de-escalation for oropharyngeal cancer cases, but the use of Cetxumab instead of Cispaltin in concurrent radiotherapy was associated with a detriment in OS. Currently, a regimen based on Gemcitabine and Nedaplatin is evaluated as a radio-sensitizer in concurrent settings with radiotherapy, but the high rates of adverse effects make it necessary to adjust the doses [19,20,21,22,23].
Immune checkpoint inhibitors (ICI) entered the therapeutic spectrum of locally advanced disease in possible synergistic association with radiotherapy, the success of the PACIFIC trial having opened this horizon. Durvalumab has become the standard of care as a consolidation treatment for locally advanced non-small-cell lung cancers that have responded favorably to curative chemo-radiotherapy. Even if, in HNC, Pembrolizumab concurrent with radiotherapy was not associated with either a superior tumor control or a benefit in OS, the PembroRad regimen associates a more favorable toxicity profile for locally advanced HNSCC cases not eligible for Cisplatin [24,25].
Even if it is not considered standard treatment, induction chemotherapy that associates Cisplatin and 5-fluorouracil demonstrated a 93% overall response rate (ORR), similar results (86%) being reported by the Radiation Therapy Oncology Group (RTOG) trial, and both rates of response having been reported more than 30 years ago. Currently, a combination between platinum, taxanes, and fluorouracil (TPF) is considered to be the optimal treatment, with superior results in induction both compared to the platinum–fluorouracil regimen and compared to the platinum–taxane combination. Haddad et al. mention the benefit of the TPF regime in survival and in the preservation of the organ, being recommended both for cases with a poor prognosis and for reducing the risk of distant metastases or loco-regional failure in operable cases or those with a good prognosis [26,27,28].

5. Image-Guided Radiotherapy (IGRT)—The Cornerstone for Radiomics

Modern radiotherapy has evolved along with advances in the diagnosis and management of HNC. The proximity of radiosensitive structures and the need to limit treatment related toxicities that affect the quality of life (QoL), but also the treatment discontinuation risk with possible tumor control failure, are arguments that led to the implementation of more conformal irradiation techniques based on the ballistic localization of the tumor and the radiosensitive structures named organs at risk (OARs). If, for most of HNC anatomical sites, the implementation of intensity modulated radiotherapy (IMRT), based on the concept of image-guided radiotherapy (IGRT), brings a significant benefit in reducing xerostomia and dysphagia, in the case of nasopharyngeal cancer the implementation of the IMRT technique brings a net benefit in local tumor control over the older conformal technique. Not only the parotid irradiation but also the minor salivary glands and submandibular gland could be spared by IMRT irradiation, the limitation of radiation doses to these structures also being a factor associated with the reduction in moderate to severe xerostomia rates [29,30,31,32].
An improved dose conformity is also associated with steeper dose gradients, thus increasing the risk of the geographic missing of target volumes. Both the use of the CT simulator and the integrated on-board imaging (OBI) system, an imaging-based system that improves the patient’s radiation therapy setup accuracy, are included in the concept of IGRT. The displacement of internal organs between radiotherapy fractions, but also the position variation for target volumes and OARs during a treatment fraction, are the causes of possible uncertainties mentioned by Fernández-Rodríguez et al. [33].
On-board cone-beam-computed tomography (CBCT), for the alignment of head and neck cancer patients before radiotherapy, is a time-consuming method that also involves an additional radiation dose for the patient. A study that included 56 cases demonstrated that, without imaging guidance, 40.8% of the fractions would have been delivered at >5 mm distance from the target. However, by correcting the daily shift by using the first five CBCT displacement measurements, this percentage of fractions delivered outside the target could be reduced to 19%. Zumsteg et al. recommends, in this context, the use of daily CBCT to reduce the uncertainties related to the positioning of HNC patients for radiotherapy treatment. The concept of using CBCT, not only to ensure the accuracy of treatment delivery but also for early treatment response prediction, was implemented by Sumner et al. The authors propose the correlation between the 3 month follow-up imaging responses and overall survival (OS) of patients with HNSCC and oropharyngeal location, with volume changes evaluated by CBCT during radiotherapy. The proposed method was the measurement of the largest nodal conglomerate (LNC) at the time of CT simulation and also by using CBCT at the end of radiotherapy. A 30% reduction in the cranio-caudal dimensions of the tumor of the tumor-involved lymph node volume was correlated with improved OS, but the variation in the cranio-caudal dimensions was also correlated with the imaging response 3 months after treatment. The authors note the potential of CBCT imaging to be an early predictor of response for oropharyngeal cancer [34,35,36].

6. Δ Radiomics in HNC—From Image-Guided Radiotherapy (IGRT) and Adaptive Radiotherapy (ART) to Treatment Response/Toxicity Prediction

Tran et al. propose a non-invasive method, based on Δ radiomics, of quantitative ultrasound (QUS) extracted from metastatic lymph nodes in order to predict the therapeutic response in the first 24 h after the start of treatment with radical radiotherapy. Spectral and texture features extracted from QUS, at the beginning of radiotherapy and at 24 h, 1 week, and 4 weeks after the beginning of treatment, were evaluated in the study. By dividing the patients according to the response to treatment at 3 months into partial and complete responders, the authors build a predictive Δ radiomic model. A single-feature naïve-Bayes classification provides the best correlation with the therapeutic response. The predictive accuracies were 80, 86, and 85% at 24 h, 1 week, and 4 weeks, respectively. The authors note that, even if QUS Δ radiomics, evaluated at 1 week and 4 weeks, offers better results, the method also offers reasonable accuracy for early radiotherapy treatment response prediction [37].
Dividing a cohort of 93 HNC patients into a training (60 cases) and validation set (33 cases), Sellami and colleagues evaluated, in a Δ radiomic study, 88 features extracted from the gross tumor volume (GTV) delineated on CBCT images. After the selection by receiver operating characteristic (ROC) curves of those radiomic features significant for the weekly response to CCRT, those radiomic features that had a significant variation in at least 5% of the cases were evaluated. After these repeated exclusions of redundant radiomic features, only Coarseness was considered as a Δ radiomic feature predictor of response to chemo-radiotherapy. The association of clinical variables with the model based on Δ-radiomics variation could increase the accuracy of treatment response prediction. Thus, a combined model that included the hemoglobin value demonstrated predictive superiority over the Δ radiomic model [38].
Adaptive radiotherapy (ART) originated at the end of the 1990s, but Yan et al. highlight the concept for the first time in an article published in 2010. The National Radiotherapy Advisory Group in the UK has been recommending four-dimensional adaptive radiotherapy (4D-ART) as the future standard in radiotherapy since 2007, it being considered an “extremely encouraging and greatly promoting adaptive radiotherapy”. ART could contribute to improving radiotherapy patient outcomes and methods such as auto-segmentation, automated planning, and deformable registration algorithms are proposed as a response to anatomical and functional variations of the structures involved in treatment planning. HNC, lung, and cervix cancers are mentioned as potential beneficiaries of ART. Currently, MRI-guided online adaptive radiotherapy is considered as a method with possible major benefits for liver, pancreas, abdominal lymph nodes, and prostate cancers’ treatment planning. Weight loss and tumor response are considered essential factors associated with the need to correct the treatment plans. Avgousti et al. mentions, in an analysis that includes 85 articles, four major criteria advocating for ART: >10% weight loss; >1 cm deviation in external contour; >5% variations compared to the accepted doses; and variations of the planned doses in the periphery of the target volumes. The authors mention the value of ART for HNC radiotherapy and the ability of the method to decrease toxicity and improve local control. Hybrid adaptation, including image guidance combined with offline re-planning, is considered the optimal option to respond to organ motion and tumor shrinkage. Using fluoro-2-deoxyglucose (FGD) positron emission tomography (PET)-CT imaging, before and after CCRT treatment but also weekly after each chemotherapy session, for HNC cases, Yan and colleagues conceptually propose adaptive dose painting by number (DPbN) based on the voxel dose response matrix obtained by evaluating the voxel dose response matrix and the baseline standardized uptake value (SUV). The conclusions mention the existence of tumor voxels not correlated with baseline SUV as the main regions of radio-resistance. For HNC not related to human papillomavirus (HPV), radio-resistance tumor subvolumes are identified requiring >100 Gy in order to obtain adequate local control. DPbN is considered feasible as an adapted method to radiobiological heterogeneity in order to overcome radio-resistance [39,40,41,42,43,44].
The early prediction of the anatomical variations of the structures’ volume involved in HNC radiotherapy treatment could provide an indication of ART re-planning to avoid tumor coverage errors and radiation overdose of OARs. In order to identify those cases associated with major anatomical variation as a result of the early response to irradiation, a study based on the weekly assessment of CBCT used for IGRT was proposed. A total of 104 Δ-radiomics were extracted from the clinical target volume (CTV) and parotid gland volume. A model composed of 13 features extracted from CTV and 6 features extracted from parotid glands offered a 0.90 accuracy, 0.95 sensitivity, and 0.86 specificity performance for the previously mentioned end-point. The variation of three features, including the gray level matrix features family, were identified as significant in both analyzed structures [45].
An ability evaluation of radiomic features kinetics, extracted from daily mega-voltage CT (MVCT) scans for HNC radiotherapy IGRT used to predict moderate to severe xerostomia, was compared to the dose/volume parameters. The variation Δ of radiomic features extracted from MVCT for the contralateral parotid gland were evaluated as a predictor of moderate to severe xerostomia at 6, 12, and 24 months after treatment. The Δ radiomic model was compared with a pre-treatment dosimetric model and with a mixed dosimetric–radiomic model. For the models internal validation, cross-validation and bootstrapping were used. The Δ radiomic performances were quantified both on training sets and on test sets; area under the curve (AUC) values of 46%, 33%, and 26% for the moderate to severe xerostomia predictive rate at 6, 12, and 24 months, respectively, were obtained by testing the model. Radiomic Δ variations obtained from MVCT, comparing the baseline treatment’s half features, demonstrated superior predictive power compared to the pre-treatment dosimetric models. Pota et al. also mentions the use of the radiomic parameters’ dynamics, extracted from CT images until the halfway point of the treatment, as early predictors of radiotherapy-induced parotid shrinkage, but also for treatment-related toxicity. The fuzzy logic concept, a method of grouping data with the same characteristics into fuzzy sets and a model with the capacity of managing uncertain data and building rule-based classifiers, was proposed by the authors. Weekly CT changes during treatment quantified in Δ radiomic features were assessed on 68 treatment planning CTs and 340 weekly control CTs as a predictor of moderate to severe xerostomia 12 months after radiotherapy. Geometric, intensity, and texture features were evaluated for their predictive power in relation to the reference models based on mean radiation dose for contralateral parotid gland and nadir xerostomia score (Xerbaseline). The Δ variation of the contralateral parotid gland surface was the most predictive feature correlated with moderate and severe xerostomia at 12 months. The Δ changes of the surface of the parotid gland, 3 weeks after the treatment started, demonstrated superior predictive capacity compared to the pre-treatment models. QUS Δ radiomics obtained during CCRT is also proposed as a predictor of recurrence in HNSCC. A total of 31 spectral and texture radiomic features were extracted from a tumor-involved lymph node. The study group included 51 cases treated with a total radiotherapy dose of 70 Gy in 33 daily fractions of IMRT. The radiomic data were extracted before the treatment, and at one and four weeks after the beginning of the treatment. The study included patients with cancers of the oropharynx, larynx, hypopharynx, and of unknown primary; all cases were treated with platinum-based chemotherapy or Cetuximab. In a median follow-up of 38 months (range 7–64 months), relapse was identified in 17 cases. The model predicted the recurrence with an accuracy of 80% and 82% using Δ radiomics at one week and 4 weeks, respectively. The necessity of anticipating the result of chemo-radiotherapy treatment in HNC is also mentioned by Morgan et al., which also highlights the advantage of a response-adapted concept in order to escalate, de-escalate, or modify the treatment. In a retrospective analysis, the authors analyze, in a single institution study, the variation of radiomic features between baseline CT and CBCT, recorded at 1 day and 21 days during radiotherapy treatment. The analysis includes the primary tumor and metastases lymph nodes, with both local or lymph node failure being analyzed. The results were compared evaluating the clinical, radiomic model (including pre-treatment CT and radiomic Δ) and fused model (including both clinical and radiomic data). The fused model seems to offer superior predictive capacity for primary tumor (sensitivity of 78.3% and specificity of 90.9%), as well as for lymph nodes failure (sensitivity of 100.0% and specificity of 68.0%). Even if the study cannot demonstrate a net superiority of the radiomic and Δ radiomic algorithm compared to the clinical model, the fused model, including both clinical Δ radiomic data, is considered superior for the prediction of loco-regional failure including primary tumor, but also lymph nodes, relapse [46,47,48,49,50].
Evaluating the risk of acute xerostomia in a 35 patient lot with stage I-IVB nasopharyngeal cancer treated with IMRT radiotherapy, Liu et al. compared the model based on the variation Δ of CT radiomic features extracted from the parotid glands and a model based on saliva amount quantification. Radiomic features were extracted at fractions 0, 10, 20, and 30. Four patients were subsequently added to the lot for independent testing of the model. The delineation of the parotid gland was performed manually by the radiation oncologist, and the saliva was collected every 10 days during the treatment. Both the amount of saliva and the Radiation Therapy Oncology Group (RTOG) acute xerostomia score were used to quantify acute xerostomia. A total of 1703 radiomic features were extracted from each CT image of the parotid gland. RidgeCV and recursive feature elimination (RFE) methods were used for features selection and feature matrix reduction. Totals of 8 baseline radiomic features and 14 radiomic features extracted after 10 days of treatment were identified as significant for predicting acute xerostomia. The model that compared the variation of radiomic features between the baseline value and the value recorded at 10 treatment fractions offered the best predictive power, with a precision of 0.9220 and a sensitivity of 100% compared to the model based on saliva amount [51]. The main studies that evaluate Δ radiomics as a predictor of response to radiotherapy alone or in combination with systemic therapy, but also to anticipate early treatment related toxicity or the need for ART in HNC, have been summarized in Table 1, [35,37,38,45,46,47,48,49,50,51].

7. Conclusions

Δ radiomics demonstrates its potential as a predictive biomarker of treatment response in HNC and, conceptually, IGRT including CT simulation and CBCT positioning control imaging thus opens up new perspectives for the implementation of radiomics and Δ radiomics in radiation oncology. The use of Δ features of texture, shape, and size, both from the primary tumor and from the tumor-involved lymph nodes, demonstrates the best predictive accuracy. If, in the case of treatment response, promising Δ radiomics results could be obtained, even after 24 h from the start of treatment, for radiation-induced xerostomia the evaluation of Δ radiomics in the middle of treatment could be recommended. The fused models (clinical and Δ radiomics) seem to offer a benefit both in comparison to the clinical model and to the radiomic model. The selection of patients who benefit from induction chemotherapy is underestimated in Δ radiomic studies and may be an unexplored territory with major potential. The advantage offered by “in house” simulation CT and CBCT favors the rapid implementation of Δ radiomics studies in radiotherapy departments. PET-CT Δ radiomics could guide new concepts of dose escalation for radio-resistant sub-volumes based on radiobiological criteria, but could also guide the “next level” of HNC ART.

Author Contributions

All authors have contributed equally. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Gillies, R.J.; Kinahan, P.E.; Hricak, H. Radiomics: Images Are More than Pictures, They Are Data. Radiology 2016, 278, 563–577. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  2. Nardone, V.; Reginelli, A.; Grassi, R.; Boldrini, L.; Vacca, G.; D’Ippolito, E.; Annunziata, S.; Farchione, A.; Belfiore, M.P.; Desideri, I.; et al. Delta radiomics: A systematic review. Radiol. Med. 2021, 126, 1571–1583. [Google Scholar] [CrossRef] [PubMed]
  3. Denaro, N.; Merlano, M.C.; Russi, E.G. Follow-up in Head and Neck Cancer: Do More Does It Mean Do Better? A Systematic Review and Our Proposal Based on Our Experience. Clin. Exp. Otorhinolaryngol. 2016, 9, 287–297. [Google Scholar] [CrossRef]
  4. Sturgis, E.M.; Miller, R.H. Second primary malignancies in the head and neck cancer patient. Ann. Otol. Rhinol. Laryngol. 1995, 104, 946–954. [Google Scholar] [CrossRef] [PubMed]
  5. Fave, X.; Zhang, L.; Yang, J.; Mackin, D.; Balter, P.; Gomez, D.; Followill, D.; Jones, A.K.; Stingo, F.; Liao, Z.; et al. Delta-radiomics features for the prediction of patient outcomes in non–small cell lung cancer. Sci. Rep. 2017, 7, 588. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  6. Jha, A.K.; Mithun, S.; Purandare, N.C.; Kumar, R.; Rangarajan, V.; Wee, L.; Dekker, A. Radiomics: A quantitative imaging biomarker in precision oncology. Nucl. Med. Commun. 2022, 43, 483–493. [Google Scholar] [CrossRef]
  7. Wang, X.; Xie, T.; Luo, J.; Zhou, Z.; Yu, X.; Guo, X. Radiomics predicts the prognosis of patients with locally advanced breast cancer by reflecting the heterogeneity of tumor cells and the tumor microenvironment. Breast Cancer Res. 2022, 24, 20. [Google Scholar] [CrossRef]
  8. Iancu, R.I.; Zara, A.D.; Mirestean, C.C.; Iancu, D.P.T. Radiomics in Head and Neck Cancers Radiotherapy. Promises and Challenges. Maedica (Bucur) 2021, 16, 482–488. [Google Scholar] [CrossRef]
  9. Mirestean, C.C.; Pagute, O.; Buzea, C.; Iancu, R.I.; Iancu, D.T. Radiomic Machine Learning and Texture Analysis—New Horizons for Head and Neck Oncology. Maedica (Bucur) 2019, 14, 126–130. [Google Scholar] [CrossRef]
  10. Wendt, T.G.; Grabenbauer, G.G.; Rödel, C.M.; Thiel, H.J.; Aydin, H.; Rohloff, R.; Wustrow, T.P.; Iro, H.; Popella, C.; Schalhorn, A. Simultaneous radiochemotherapy versus radiotherapy alone in advanced head and neck cancer: A randomized multicenter study. J. Clin. Oncol. 1998, 16, 1318–1324. [Google Scholar] [CrossRef]
  11. Skladowski, K.; Law, M.G.; Maciejewski, B.; Steel, G.G. Planned and unplanned gaps in radiotherapy: The importance of gap position and gap duration. Radiother. Oncol. 1994, 30, 109–120. [Google Scholar] [CrossRef] [PubMed]
  12. Robertson, A.G.; Robertson, C.; Perone, C.; Clarke, K.; Dewar, J.; Elia, M.H.; Hurman, D.; MacDougall, R.H.; Yosef, H.M. Effect of gap length and position on results of treatment of cancer of the larynx in Scotland by radiotherapy: A linear quadratic analysis. Radiother. Oncol. 1998, 48, 165–173. [Google Scholar] [CrossRef] [PubMed]
  13. O’Shea, K.; Coleman, L.; Fahy, L.; Kleefeld, C.; Foley, M.J.; Moore, M. Compensation for radiotherapy treatment interruptions due to a cyberattack: An isoeffective DVH-based dose compensation decision tool. J. Appl. Clin. Med. Phys. 2022, 23, e13716. [Google Scholar] [CrossRef]
  14. Putora, P.M.; Schmuecking, M.; Aebersold, D.; Plasswilm, L. Compensability index for compensation radiotherapy after treatment interruptions. Radiat Oncol. 2012, 7, 208. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  15. Steinmann, D.; Cerny, B.; Karstens, J.H.; Bremer, M. Chemoradiotherapy with weekly cisplatin 40 mg/m2 in 103 head-and-neck cancer patients: A cumulative dose-effect analysis. Strahlenther. Onkol. 2009, 185, 682–688. [Google Scholar] [CrossRef]
  16. Sharma, A.; Kumar, M.; Bhasker, S.; Thakar, A.; Pramanik, R.; Biswas, A.; Kumar, A.; Sikka, K.; Singh, A.C.; Mallick, S.; et al. An open-label, noninferiority phase III RCT of weekly versus three weekly cisplatin and radical radiotherapy in locally advanced head and neck squamous cell carcinoma (ConCERT trial). In Proceedings of the 2022 ASCO Annual Meeting, Chicago, IL, USA, 3–7 June 2022. [Google Scholar]
  17. Lacas, B.; Carmel, A.; Landais, C.; Wong, S.J.; Licitra, L.; Tobias, J.S.; Burtness, B.; Ghi, M.G.; Cohen, E.E.W.; Grau, C.; et al. Meta-analysis of chemotherapy in head and neck cancer (MACH-NC): An update on 107 randomized trials and 19,805 patients, on behalf of MACH-NC Group. Radiother. Oncol. 2021, 156, 281–293. [Google Scholar] [CrossRef]
  18. Available online: https://ascopost.com/issues/september-10-2022/concurrent-chemoradiation-therapy-with-weekly-cisplatin-for-locally-advanced-head-and-neck-squamous-cell-carcinoma/ (accessed on 3 March 2023).
  19. Bonner, J.A.; Harari, P.M.; Giralt, J.; Cohen, R.B.; Jones, C.U.; Sur, R.K.; Raben, D.; Baselga, J.; Spencer, S.A.; Zhu, J.; et al. Radiotherapy plus cetuximab for locoregionally advanced head and neck cancer: 5-year survival data from a phase 3 randomised trial, and relation between cetuximab-induced rash and survival. Lancet Oncol. 2010, 11, 21–28. [Google Scholar] [CrossRef]
  20. Taberna, M.; Oliva, M.; Mesía, R. Cetuximab-Containing Combinations in Locally Advanced and Recurrent or Metastatic Head and Neck Squamous Cell Carcinoma. Front. Oncol. 2019, 9, 383. [Google Scholar] [CrossRef]
  21. Gillison, M.L.; Trotti, A.M.; Harris, J.; Eisbruch, A.; Harari, P.M.; Adelstein, D.J.; Jordan, R.C.K.; Zhao, W.; Sturgis, E.M.; Burtness, B.; et al. Radiotherapy plus cetuximab or cisplatin in human papillomavirus-positive oropharyngeal cancer (NRG Oncology RTOG 1016): A randomised, multicentre, non-inferiority trial. Lancet 2019, 393, 40–50, Erratum in Lancet 2020, 395, 784. [Google Scholar] [CrossRef]
  22. Mehanna, H.; Robinson, M.; Hartley, A.; Kong, A.; Foran, B.; Fulton-Lieuw, T.; Dalby, M.; Mistry, P.; Sen, M.; O’Toole, L.; et al. Radiotherapy plus cisplatin or cetuximab in low-risk human papillomavirus-positive oropharyngeal cancer (De-ESCALaTE HPV): An open-label randomised controlled phase 3 trial. Lancet 2019, 393, 51–60. [Google Scholar] [CrossRef] [Green Version]
  23. Huo, R.X.; Jin, Y.Y.; Zhuo, Y.X.; Ji, X.T.; Cui, Y.; Wu, X.J.; Wang, Y.J.; Zhang, L.; Zhang, W.H.; Cai, Y.M.; et al. Concurrent chemoradiotherapy using gemcitabine and nedaplatin in recurrent or locally advanced head and neck squamous cell carcinoma. World J. Clin. Cases 2022, 10, 3414–3425. [Google Scholar] [CrossRef] [PubMed]
  24. Antonia, S.J.; Villegas, A.; Daniel, D.; Vicente, D.; Murakami, S.; Hui, R.; Yokoi, T.; Chiappori, A.; Lee, K.H.; de Wit, M.; et al. Durvalumab after Chemoradiotherapy in Stage III Non-Small-Cell Lung Cancer. N. Engl. J. Med. 2017, 377, 1919–1929. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  25. Tao, Y.; Biau, J.; Sun, X.S.; Sire, C.; Martin, L.; Alfonsi, M.; Prevost, J.B.; Modesto, A.; Lafond, C.; Tourani, J.M.; et al. Pembrolizumab versus cetuximab concurrent with radiotherapy in patients with locally advanced squamous cell carcinoma of head and neck unfit for cisplatin (GORTEC 2015-01 PembroRad): A multicenter, randomized, phase II trial. Ann. Oncol. 2023, 34, 101–110. [Google Scholar] [CrossRef] [PubMed]
  26. Haddad, R.I.; Posner, M.; Hitt, R.; Cohen, E.E.W.; Schulten, J.; Lefebvre, J.L.; Vermorken, J.B. Induction chemotherapy in locally advanced squamous cell carcinoma of the head and neck: Role, controversy, and future directions. Ann. Oncol. 2018, 29, 1130–1140. [Google Scholar] [CrossRef]
  27. Hitt, R.; Paz-Ares, L.; Brandáriz, A.; Castellano, D.; Peña, C.; Millán, J.M.; Calvo, F.; Ortiz de Urbina, D.; López, E.; Alvarez-Vicent, J.J.; et al. Induction chemotherapy with paclitaxel, cisplatin and 5-fluorouracil for squamous cell carcinoma of the head and neck: Long-term results of a phase II trial. Ann. Oncol. 2002, 13, 1665–1673. [Google Scholar] [CrossRef]
  28. Lowe, N.M.; Bernstein, J.M.; Mais, K.; Garcez, K.; Lee, L.W.; Sykes, A.; Thomson, D.J.; Homer, J.J.; West, C.M.; Slevin, N.J. Taxane, platinum and 5-FU prior to chemoradiotherapy benefits patients with stage IV neck node-positive head and neck cancer and a good performance status. J. Cancer Res. Clin. Oncol. 2018, 144, 389–401. [Google Scholar] [CrossRef]
  29. Wang, X.; Eisbruch, A. IMRT for head and neck cancer: Reducing xerostomia and dysphagia. J. Radiat. Res. 2016, 57 (Suppl. S1), i69–i75. [Google Scholar] [CrossRef] [Green Version]
  30. Lee, N.; Xia, P.; Quivey, J.M.; Sultanem, K.; Poon, I.; Akazawa, C.; Akazawa, P.; Weinberg, V.; Fu, K.K. Intensity-modulated radiotherapy in the treatment of nasopharyngeal carcinoma: An update of the UCSF experience. Int. J. Radiat. Oncol. Biol. Phys. 2002, 53, 12–22. [Google Scholar] [CrossRef]
  31. Little, M.; Schipper, M.; Feng, F.Y.; Vineberg, K.; Cornwall, C.; Murdoch-Kinch, C.A.; Eisbruch, A. Reducing xerostomia after chemo-IMRT for head-and-neck cancer: Beyond sparing the parotid glands. Int. J. Radiat. Oncol. Biol. Phys. 2012, 83, 1007–1014. [Google Scholar] [CrossRef] [Green Version]
  32. Rades, D.; Warwas, B.; Gerull, K.; Pries, R.; Leichtle, A.; Bruchhage, K.L.; Hakim, S.G.; Schild, S.E.; Cremers, F. Prognostic Factors for Complete Recovery From Xerostomia After Radiotherapy of Head-and-Neck Cancers. In Vivo 2022, 36, 1795–1800. [Google Scholar] [CrossRef]
  33. Fernández-Rodríguez, L.J.; Arens-Benites, M.A.; Maldonado-Pijoan, X. Image-Guided Radiation Therapy for Squamous Cell Cancer of the Head and Neck in a Specialized Peruvian Public Hospital. Cureus 2022, 14, e22569. [Google Scholar] [CrossRef] [PubMed]
  34. Kearney, M.; Coffey, M.; Leong, A. A review of Image Guided Radiation Therapy in head and neck cancer from 2009-201—Best Practice Recommendations for RTTs in the Clinic. Technol. Innov. Patient Support. Radiat. Oncol. 2020, 14, 43–50. [Google Scholar] [CrossRef] [PubMed]
  35. Sumner, W.; Kim, S.S.; Vitzthum, L.; Moore, K.; Atwood, T.; Murphy, J.; Miyauchi, S.; Califano, J.A.; Mell, L.K.; Mundt, A.J.; et al. End of treatment cone-beam computed tomography (CBCT) is predictive of radiation response and overall survival in oropharyngeal squamous cell carcinoma. Radiat. Oncol. 2021, 16, 147. [Google Scholar] [CrossRef]
  36. Zumsteg, Z.; DeMarco, J.; Lee, S.P.; Steinberg, M.L.; Lin, C.S.; McBride, W.; Lin, K.; Wang, P.C.; Kupelian, P.; Lee, P. Image guidance during head-and-neck cancer radiation therapy: Analysis of alignment trends with in-room cone-beam computed tomography scans. Int. J. Radiat. Oncol. Biol. Phys. 2012, 83, 712–719. [Google Scholar] [CrossRef]
  37. Tran, W.T.; Suraweera, H.; Quiaoit, K.; DiCenzo, D.; Fatima, K.; Jang, D.; Bhardwaj, D.; Kolios, C.; Karam, I.; Poon, I.; et al. Quantitative ultrasound delta-radiomics during radiotherapy for monitoring treatment responses in head and neck malignancies. Future Sci. OA 2020, 6, FSO624. [Google Scholar] [CrossRef] [PubMed]
  38. Sellami, S.; Bourbonne, V.; Hatt, M.; Tixier, F.; Bouzid, D.; Lucia, F.; Pradier, O.; Goasduff, G.; Visvikis, D.; Schick, U. Predicting response to radiotherapy of head and neck squamous cell carcinoma using radiomics from cone-beam CT images. Acta Oncol. 2022, 61, 73–80. [Google Scholar] [CrossRef]
  39. Yan, D. Adaptive radiotherapy: Merging principle into clinical practice. Semin. Radiat. Oncol. 2010, 20, 79–83. [Google Scholar] [CrossRef]
  40. National Radiotherapy Advisory Group. Radiotherapy: Developing a World Class Service for England. 2007. Available online: https://www.axrem.org.uk/radiotheraphy_papers/DH_Radiotheraphy_developing_first_class_service_NRAG.pdf (accessed on 3 March 2023).
  41. Nierer, L.; Eze, C.; da Silva Mendes, V.; Braun, J.; Thum, P.; von Bestenbostel, R.; Kurz, C.; Landry, G.; Reiner, M.; Niyazi, M.; et al. Dosimetric benefit of MR-guided online adaptive radiotherapy in different tumor entities: Liver, lung, abdominal lymph nodes, pancreas and prostate. Radiat. Oncol. 2022, 17, 53. [Google Scholar] [CrossRef]
  42. Avgousti, R.; Antypas, C.; Armpilia, C.; Simopoulou, F.; Liakouli, Z.; Karaiskos, P.; Kouloulias, V.; Kyrodimos, E.; Moulopoulos, L.A.; Zygogianni, A. Adaptive radiation therapy: When, how and what are the benefits that literature provides? Cancer Radiother. 2022, 26, 622–636. [Google Scholar] [CrossRef]
  43. Oh, S.; Stewart, J.; Moseley, J.; Kelly, V.; Lim, K.; Xie, J.; Fyles, A.; Brock, K.K.; Lundin, A.; Rehbinder, H.; et al. Hybrid adaptive radiotherapy with on-line MRI in cervix cancer IMRT. Radiother. Oncol. 2014, 110, 323–328. [Google Scholar] [CrossRef]
  44. Yan, D.; Chen, S.; Krauss, D.J.; Chen, P.Y.; Chinnaiyan, P.; Wilson, G.D. Tumor Voxel Dose-Response Matrix and Dose Prescription Function Derived Using 18F-FDG PET/CT Images for Adaptive Dose Painting by Number. Int. J. Radiat. Oncol. Biol. Phys. 2019, 104, 207–218. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  45. Iliadou, V.; Kakkos, I.; Karaiskos, P.; Kouloulias, V.; Platoni, K.; Zygogianni, A.; Matsopoulos, G.K. Early Prediction of Planning Adaptation Requirement Indication Due to Volumetric Alterations in Head and Neck Cancer Radiotherapy: A Machine Learning Approach. Cancers 2022, 14, 3573. [Google Scholar] [CrossRef] [PubMed]
  46. Berger, T.; Noble, D.J.; Shelley, L.E.A.; McMullan, T.; Bates, A.; Thomas, S.; Carruthers, L.J.; Beckett, G.; Duffton, A.; Paterson, C.; et al. Predicting radiotherapy-induced xerostomia in head and neck cancer patients using day-to-day kinetics of radiomics features. Phys. Imaging Radiat. Oncol. 2022, 24, 95–101. [Google Scholar] [CrossRef] [PubMed]
  47. Pota, M.; Scalco, E.; Sanguineti, G.; Farneti, A.; Cattaneo, G.M.; Rizzo, G.; Esposito, M. Early prediction of radiotherapy-induced parotid shrinkage and toxicity based on CT radiomics and fuzzy classification. Artif. Intell. Med. 2017, 81, 41–53. [Google Scholar] [CrossRef] [PubMed]
  48. van Dijk, L.V.; Langendijk, J.A.; Zhai, T.-T.; Vedelaar, T.A.; Noordzij, W.; Steenbakkers, R.J.H.M.; Sijtsema, N.M. Delta-radiomics features during radiotherapy improve the prediction of late xerostomia. Sci. Rep. 2019, 9, 12483. [Google Scholar] [CrossRef] [Green Version]
  49. Fatima, K.; Dasgupta, A.; DiCenzo, D.; Kolios, C.; Quiaoit, K.; Saifuddin, M.; Sandhu, M.; Bhardwaj, D.; Karam, I.; Poon, I.; et al. Ultrasound delta-radiomics during radiotherapy to predict recurrence in patients with head and neck squamous cell carcinoma. Clin. Transl. Radiat. Oncol. 2021, 28, 62–70. [Google Scholar] [CrossRef] [PubMed]
  50. Morgan, H.E.; Wang, K.; Dohopolski, M.; Liang, X.; Folkert, M.R.; Sher, D.J.; Wang, J. Exploratory ensemble interpretable model for predicting local failure in head and neck cancer: The additive benefit of CT and intra-treatment cone-beam computed tomography features. Quant. Imaging Med. Surg. 2021, 11, 4781–4796. [Google Scholar] [CrossRef]
  51. Liu, Y.; Shi, H.; Huang, S.; Chen, X.; Zhou, H.; Chang, H.; Xia, Y.; Wang, G.; Yang, X. Early prediction of acute xerostomia during radiation therapy for nasopharyngeal cancer based on delta radiomics from CT images. Quant. Imaging Med. Surg. 2019, 9, 1288–1302. [Google Scholar] [CrossRef]
Table 1. Δ Radiomics in HNC—from treatment response and toxicity to ART prediction.
Table 1. Δ Radiomics in HNC—from treatment response and toxicity to ART prediction.
Δ Radiomics in HNC
Images Used for Radiomic AnalysisNumber of Selected FeaturesEndpointReferences
UltrasoundSingle-feature naive-Bayes classificationPrediction of radiotherapy response at 3 monthsTran et al., 2020 [37]
CBCTTwo-dimensional (2D) maximum and craniocaudal (CC)Prediction of radiotherapy responseSumner at al., 2021 [35]
CBCT1 feature (Coarseness);
Clinical + radiomic model
Progression to radiotherapy for oropharingeal cancerSellami et al., 2022 [38]
CBCT13 features extracted from CTV and 6 features extracted from parotid glandsVolume change in anatomical strictures; necessity for adaptive radiotherapyIliadou et al., 2022 [45]
MVCTCombination of dose/volume and radiomics-based modelPrediction of moderate-to-severe xerostomiaBerger et al., 2022 [46]
CTTexture featuresEarly prediction of radiotherapy-induced parotid shrinkage and toxicity during radiotherapyPota et al., 2017 [47]
Treatment planning CT and evaluation CTGrey tone difference matrix (coarseness), kurtosis, and the median intensityPrediction of late xerostomiavan Dijk et al., 2019 [48]
Ultrasound31 spectral and related texture featuresPrediction of higher risk of recurrenceFatima et al., 2021 [49]
CT and CBCT102 features including 18 first order, 14 shape, 24 gray level co-occurrence matrix (GLCM), 16 gray level run length matrix (GLRLM), 16 gray level size zone matrix (GLSZM), and 14 gray level dependence matrix (GLDM) features.Prediction of local failureMorgan et al., 2021 [50]
CTSaliva amount and 1703 radiomics features based modelPrediction of acute xerostomia during radiation therapy for nasopharyngeal cancerLiu et al., 2019 [51]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Mireștean, C.C.; Iancu, R.I.; Iancu, D.P.T. Image Guided Radiotherapy (IGRT) and Delta (Δ) Radiomics—An Urgent Alliance for the Front Line of the War against Head and Neck Cancers. Diagnostics 2023, 13, 2045. https://doi.org/10.3390/diagnostics13122045

AMA Style

Mireștean CC, Iancu RI, Iancu DPT. Image Guided Radiotherapy (IGRT) and Delta (Δ) Radiomics—An Urgent Alliance for the Front Line of the War against Head and Neck Cancers. Diagnostics. 2023; 13(12):2045. https://doi.org/10.3390/diagnostics13122045

Chicago/Turabian Style

Mireștean, Camil Ciprian, Roxana Irina Iancu, and Dragoș Petru Teodor Iancu. 2023. "Image Guided Radiotherapy (IGRT) and Delta (Δ) Radiomics—An Urgent Alliance for the Front Line of the War against Head and Neck Cancers" Diagnostics 13, no. 12: 2045. https://doi.org/10.3390/diagnostics13122045

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop