Skip to Content
MDPIMDPI

Advanced

RadiationRadiation
PDF
  • Systematic Review
  • Open Access

5 June 2026

19 Pages

Systematic Review of Quantitative Metrics to Standardize Contours on Radiotherapy Clinical Trials

,
,
,
and
1
Northwell Health, Lake Success, NY 11042, USA
2
Zucker School of Medicine at Hofstra/Northwell, Hempstead, NY 11549, USA
3
Department of Radiation Oncology, University of Pennsylvania, Philadelphia, PA 19104, USA
4
Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104, USA
Radiation2026, 6(2), 20;https://doi.org/10.3390/radiation6020020
Version Notes
Review Reports
Order Reprints

Simple Summary

Radiation therapy quality assurance (RTQA) is critical to the success of RT and impacts trial outcomes. We aimed to evaluate the existence of standardized quantitative metrics to grade contours as part of RTQA and to identify barriers to standardization. Our study endpoints were quantitative metrics being used to guide contour scoring. We initially planned quantitative synthesis (meta-analysis) of geometric metric distributions across studies. However, the heterogeneity in evaluation methods, the predominance of qualitative assessment, and the lack of standardized reporting precluded meaningful quantitative pooling. We therefore performed structured qualitative synthesis, thematic analysis, and gap assessment. We found a concerning lack of standardized quantitative metrics to evaluate RT contours, which can negatively impact clinical trial results. Further, there is a lack of methodological consistency required for systematic quantitative synthesis, reinforcing the need for standardization. Effort is needed to create guidelines for quantitatively evaluating RT contours to use across clinical trials to improve the quality and validity of RTQA.

Abstract

Radiation therapy quality assurance (RTQA) is critical to the success of RT and impacts trial outcomes. We aimed to evaluate the existence of standardized quantitative metrics to grade contours as part of RTQA and to identify barriers to standardization. A systematic review was conducted according to PRISMA guidelines. PubMed, Web of Science, and Embase were used to search relevant studies published from 2014–2024. Our endpoints were quantitative metrics used to guide contour scoring. We initially planned quantitative synthesis (meta-analysis) of geometric metric distributions across studies. However, the heterogeneity in evaluation methods, predominance of qualitative assessment, and lack of standardized reporting precluded meaningful quantitative pooling. We therefore performed structured qualitative synthesis, thematic analysis, and gap assessment. A total of 37 studies were included, 20 of which were prospective and 17 were retrospective. Of the prospective, 12 reported the results of benchmark case/dummy run reviews, 4 were a combination of benchmark case/dummy runs and individual QA analyses done pre- or post-treatment, 1 used knowledge-based planning to evaluate plans in real-time, and 3 involved real-time prospective QA. Only one reported the use of an objective QA scoring system. There is a concerning lack of standardized quantitative metrics to evaluate RT contours on clinical trials and a lack of methodological consistency required for systematic quantitative synthesis, reinforcing the need for standardization. Effort is needed to create guidelines for quantitatively evaluating RT contours to use across clinical trials to improve the quality and validity of RTQA.

1. Introduction

Approximately 50% of cancer patients receive radiation therapy (RT) as a component of their cancer care [1]. Clinical trials are essential to establish best practices in delivering RT in the treatment of cancer patients. One of the foundations of successful RT delivery is accurate contouring of both target volumes and organs at risk (OARs). However, contour delineation can be highly variable among providers and is consistently shown to account for the majority of protocol deviations encountered on clinical trials [2,3]. Studies have shown that major protocol deviations can adversely impact trial outcomes [4,5,6].
To ensure uniform and accurate RT delivery on cooperative group trials, radiation therapy quality assurance (RTQA) programs have been widely implemented both in the United States and abroad [7,8,9,10]. Clinical trial RTQA programs consist of multiple components including accurate contour delineation, treatment planning, and treatment delivery. This review focuses strictly on contour evaluation processes as distinct from plan dosimetric quality. Contour geometric accuracy refers to the spatial agreement between delineated structures and their true anatomical boundaries or consensus definitions, thus measuring the precision of delineation itself, independent of subsequent dose calculations. Geometric accuracy can be measured using metrics such as the Dice similarity coefficient (DSC), Hausdorff distance (HD), mean distance to agreement, and volume ratios. It represents the foundation upon which all treatment planning is built. Plan dosimetric quality refers to how well the calculated dose distribution meets protocol-specified constraints and objectives, and is evaluated using dose-volume parameters such as planning target volume (PTV) coverage metrics (V95%, D95%, conformity indices) and organ-at-risk constraints (Dmean, Dmax, VxGy). Dosimetric quality depends on both the accuracy of the underlying contours and the quality of the optimization and planning process, with contours being the foundation on which subsequent treatment planning is based on [9].
Despite the increasing implementation of RTQA involving contour review across cooperative group clinical trials, it is not uniform and can be highly variable even among strictly designed phase III randomized controlled studies [11]. While there has been a surge in the development and use of comprehensive guidelines to delineate target volume and OAR contours [12], there is a lack of guidance on how to assess and score the quality of submitted contours. Herein, we conducted a review to systematically assess quantitative metrics being used to guide RTQA evaluation of target volume and OAR contours on cooperative group clinical trials. While we acknowledge that the final treatment plan/dosimetry are ultimately the clinical concern, we contend that standardized geometric evaluation is a prerequisite for meaningful interpretation of dosimetric findings and attribution of protocol deviations.

2. Materials and Methods

A systematic review was conducted according to the guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) to investigate evidence for RTQA performed on randomized controlled trials conducted by a major cooperative group in the United States or abroad specifically involving assessment of target volume or OAR contours or both. The search was performed by a professional librarian in PubMed, Web of Science, and Embase. Relevant studies conducted in the English language between 2004 and 2024 were included. Search terms were broad to reflect the variations of nomenclature for “radiation therapy” and were combined with “contour” or “organ at risk” or “target coverage” or “dosimetric coverage” and major cooperative groups in the United States and abroad. The references were imported into Covidence and two authors independently screened the comprehensive list of articles. The PRISMA Flow Diagram is presented in Figure 1. This review was performed in accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines and has not been registered. A PRISMA checklist is provided in the Supplement (File S1).
Figure 1. Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA).
Two authors (LT, HG) independently screened the comprehensive list of articles, and one (LT) assessed full-text documents to identify only full-length articles and excluded review articles, abstracts, editorials, and letters. The remaining records were retrieved and full texts were reviewed for eligibility. Our study endpoints were quantitative metrics being used to guide RTQA scoring of target volume and OAR contours.
Our primary endpoint was the identification and characterization of quantitative geometric metrics used to evaluate contour accuracy in cooperative group clinical trials. We defined quantitative geometric metrics as numerical measurements that can be systematically applied to evaluate spatial agreement between submitted contours and reference standards, including but not limited to: overlap indices (Dice similarity coefficient, Jaccard index, conformity index); distance metrics (Hausdorff distance, mean distance to agreement, surface distance); volume comparisons (volume ratio, added/missed volume); and centroid displacement. We explicitly excluded metrics of dosimetric quality (e.g., PTV coverage parameters such as V95% or D95%, and OAR dose constraints such as Dmean or VxGy) as these assess planning optimization and dose calculation rather than contouring accuracy per se. Studies that assessed both contour geometry and dosimetry were included only when contour-specific evaluation methods were separately reported and could be independently analyzed.
We characterized evaluation methods as objective when they utilized computer-calculated metrics with defined thresholds that could be applied consistently across evaluators, versus subjective when they relied primarily on expert visual review or qualitative assessment without quantifiable criteria. Studies employing “comparison to gold standard” approaches were categorized based on whether this comparison was qualitative (expert assessment of similarity) or quantitative (calculation of geometric indices).
We initially planned quantitative synthesis (meta-analysis) of geometric metric distributions across studies. However, preliminary analysis revealed that the heterogeneity in evaluation methods, the predominance of qualitative assessment approaches, and the lack of standardized reporting precluded meaningful quantitative pooling of results. This limitation itself represents a key finding: the field currently lacks the methodological consistency required for systematic quantitative synthesis, reinforcing the imperative for standardization. Thus, our aim was to provide interpretive synthesis of the current landscape of contour assessment on cooperative group trials.

3. Results

Our initial search yielded a total of 1592 studies of which 509 were found to be duplicates and were removed (Figure 1). The remaining 1083 studies were screened for relevancy and 966 were found to be irrelevant and were excluded. The remaining 117 full text studies were assessed for eligibility, and 79 were excluded: 47 were the wrong study design (primarily guidelines and retrospective inter-observer studies), 19 were only available as abstracts, 7 reported the wrong outcomes, 5 were conducted in the wrong setting, and 2 reported on the wrong intervention. The remaining 37 studies were included.

3.1. Study Characteristics

Twenty studies were conducted prospectively (Table 1). Twelve of these reported results of benchmark cases or dummy runs performed as part of the credentialing process for centers participating in cooperative group trials, with comprehensive assessments of target volume and/or OAR contouring [13,14,15,16,17,18,19,20,21,22,23,24,25]. Most of these studies reported substantial variability in radiotherapy contouring and planning, despite the availability of comprehensive protocols to guide treatment [13,14,15,18,19,20,21,22,26]. Of the remaining seven prospective QA studies, four reported a combination of dummy run reviews and individual QA analyses [27,28,29,30], and three involved real-time prospective individual case review RTQA [31,32,33]. Lastly, one study involved prospective real-time RTQA using knowledge-based planning (KBP) in which a treatment plan was created by using the KBP model; this was then compared with the submitted plan for each OAR dose constraint, which was then evaluated subjectively by a trial radiation oncologist who assessed contours for adequacy [31].
Table 1. Prospective radiation therapy quality assurance data.
Seventeen studies were retrospective QA analyses of patients treated on cooperative group trials (Table 2). Twelve of these involved retrospective reviews of contouring (and plans) for patients treated on major cooperative group trials, without providing quantitative metrics used to review the contours [18,26,35,36,37,38,39,40,41,42,43,44]. Three studies dealt with assessing contouring guidelines for either OARs or target volumes on clinical trials, attempting to provide quantitative guidance on contour scoring [45,46,47].
The remaining two studies used computer programs to perform more quantitative assessments of contours as part of the RTQA process. The study by Thor et al. compared doses between deep-learning (DL) segmented hearts and trial hearts and found that, using a coherent heart definition as enabled through their open-source DL algorithm, the trial heart doses in RTOG 0617 were significantly higher than previously reported [48]. The only study to systematically analyze overall RT plan quality for an entire cohort of patients treated in a cooperative group clinical trial was conducted by Olch et al. in patients treated on the Children’s Oncology Group study ACNS1123 [49]. The authors used a commercially available plan-quality-scoring algorithm to create a scoring system. PTVs could earn up to 70 points, while OARs could earn up to 30 points, and scores were adjusted based on the difficulty in achieving the structure dose requirements. The brain stem, PTV boost, and cochlea lost the most points. The authors concluded that future clinical trials could potentially use this method to reduce plan quality variability, which may improve outcomes.
Table 2. Retrospective radiation therapy quality assurance data.
A summary of the contour evaluation methods utilized in each paper is provided in Table 3 and Figure 2.
Table 3. Summary of contour evaluation methods across all studies (n = 37).
Figure 2. Breakdown of RTQA trials and use of quantitative metrics for contour assessment.

3.2. Thematic Synthesis of Findings

Beyond the individual study characteristics delineated above, our systematic review revealed four major themes emerging consistently across the 37 included studies, regardless of disease site, cooperative group, or geographic region: (1) delineation as the predominant source of protocol deviations; (2) predominance of the qualitative “gold standard comparison” paradigm; (3) a persistent gap between identification and action; and (4) asymmetric attention to target volumes vs. organs at risk.
Across multiple studies spanning different disease sites and trial designs, contour delineation was consistently identified as the primary contributor to protocol deviations, typically accounting for 50–70% of all quality assurance failures. Christiaens et al. reported that in the EORTC 1219-DAHANCA 29 trial for head and neck cancer, delineation—not planning—remained the primary reason for protocol deviations [19]. Similarly, Stelmes et al. found that OAR contours specifically caused protocol failure in 53% of cases in the EORTC 1420 trial [14]. This pattern persisted across both prospective benchmark assessments and retrospective individual case reviews, across both target volumes and organs at risk, and across both simple and complex anatomical sites. The consistency of this finding despite methodological differences suggests a fundamental, cross-cutting challenge in radiotherapy quality assurance that transcends individual trial designs or disease-specific factors.
We found that 32 of 37 studies (86%) employed some variant of comparison to “gold standard” expert contours as their primary evaluation method. However, in the vast majority of these (28/32, 88%), this comparison was qualitative in nature—relying on expert reviewers to visually assess submitted contours and determine acceptability based on clinical judgment. Only four studies performed quantitative geometric analysis of the comparison to gold standard contours. This reveals that the field remains largely entrenched in a subjective, qualitative paradigm for contour assessment.
A comparative analysis of the ten studies using quantitative metrics revealed the heterogeneity in contour quality assessment (Table 4). Nine of the ten studies used quantitative metrics solely for research analysis rather than pass/fail determinations. Nine of the ten did not define evidence-based thresholds for acceptability, leaving interpretation subjective. DSC was the most common metric (five studies), but reported values and interpretations varied widely. Only Olch et al. developed a comprehensive scoring system with defined point allocations. There was no metric standardization across disease sites or cooperative groups.
Table 4. Comparative analysis of studies employing quantitative geometric metrics (Number = 10).
A striking pattern emerged of studies repeatedly identifying contour variability as problematic and calling for improvements, yet subsequent trials continued to encounter similar challenges without evidence of systematic amelioration (Table 5). There was a gradual increase in quantitative metric use from 12% (2009–2013) to 38% (2022–2024). The first comprehensive scoring system appeared only in the 2023 publication by Olch et al. [49]. Despite a trend toward quantification, qualitative approaches remain predominant. Thus, the field appears to be in a transition phase but lacks coordinated standardization of effort.
Table 5. Evolution of contour evaluation approaches by time period.
Our analysis revealed an imbalance in quality assurance focus, with target volume evaluation receiving substantially more attention than organ-at-risk assessment (Table 6). Among the 37 studies, 35 (95%) included target volume evaluation while 24 (65%) included OAR assessment. More strikingly, quantitative geometric metrics, when used at all, were almost exclusively applied to target volumes rather than OARs.
Table 6. Evaluation method stratified by quality assurance type.

4. Discussion

To our knowledge, this is the first systematic review of the literature assessing the existence of standardized quantitative metrics to evaluate contours as part of the cooperative group clinical trial RTQA process. While there are other systematic reviews evaluating QA programs for RT delivery [50,51], none specifically address what quantitative metrics are being used to evaluate RT contours. While the need for improved contour quality assurance has been repeatedly articulated in the literature, our systematic review reveals that quantitative standardization has not materialized despite decades of recognition. Analysis of patterns across our 37 included studies suggests multiple interconnected barriers operating at structural, methodological, and organizational levels.
Several of the studies evaluated in this systematic review identified the variability encountered among contours of patients treated on clinical trials, concluding that this variability is concerning and improvements are needed in RTQA moving forward [14,25,27,49]. While the need for improvements in contouring has been repeatedly shown and each study individually uses metrics to assess the quality of contours being reported, there is a surprising lack of literature describing how to quantitatively assess contours. This failure to provide a uniform quantitative methodology to assess RT contours on clinical trials is due to several barriers.
Unlike dosimetric constraints, which must be documented for protocol approval, contour evaluation methods remain at the discretion of individual trials or quality assurance centers. This lack of regulatory impetus removes a primary driver for standardization. Implementing quantitative contour analysis requires resources which may not be universally available. Further, there is an absence of validated, evidence-based thresholds for geometric indices. Our review found that even when studies employed quantitative metrics such as the Dice similarity coefficient, none defined what constitutes an “acceptable” value based on correlation with clinical outcomes or dosimetric impact. Lack of consensus regarding which metrics are most clinically meaningful further impedes progress. Disease-site heterogeneity compounds this challenge. Finally, the relationship between geometric contour accuracy and clinical outcomes remains incompletely characterized. Quality assurance programs have functioned using qualitative expert review for decades. Changing these processes requires demonstrating not only that quantitative approaches are superior, but that the benefit justifies the resource investment. Without compelling evidence of outcome improvement or efficiency gains, organizational momentum favors continuation of familiar methods.
Beyond the absence of validated thresholds, the metrics themselves carry well-documented limitations that influence their suitability across structures and disease sites, and these limitations are infrequently acknowledged in the trials we reviewed [52]. The Dice similarity coefficient (DSC), the most commonly reported metric in our review, is mathematically biased by structure size: for a fixed absolute boundary error, DSC decreases as structure volume decreases, making it systematically less favorable for small organs at risk (e.g., optic chiasm, cochlea, hippocampus) than for bulky targets, even when delineation quality is comparable [53]. DSC further conflates inclusion and exclusion errors and is relatively insensitive to localized boundary deviations on otherwise large structures. The Hausdorff distance (HD), by contrast, captures the worst-case boundary error and is therefore informative for structures in which a single excursion can be clinically meaningful (e.g., brainstem, spinal cord), but it is highly sensitive to outliers, segmentation noise, and slice thickness; the 95th-percentile Hausdorff distance (HD95) and the mean surface distance are commonly recommended as more robust alternatives [52,53]. For tubular and curvilinear structures (e.g., esophagus, spinal cord, optic nerves), distance-based metrics generally outperform volume-overlap metrics, whereas for large, compact volumes (e.g., parotid glands, prostate, liver) overlap metrics tend to be more discriminating. Surface DSC and added path length have been proposed specifically to bridge the disconnect between DSC and the editing effort required to render a contour clinically acceptable [54,55]. The volume ratio and centroid displacement provide complementary information about systematic over- or under-contouring and gross localization errors, respectively, but neither alone characterizes shape agreement. These differential properties imply that no single metric is appropriate across all structures, and that thoughtful, structure-specific metric selection—guided by structure size, geometry, and the clinical relevance of boundary regions—is itself a prerequisite for meaningful standardization across disease sites.
To provide context for the heterogeneity noted above, we summarize approximate ranges of geometric metric values reported across the benchmark and inter-observer variability studies in our review and in the broader contouring literature; these are offered as descriptive observations rather than prescriptive acceptability thresholds. For target volumes, reported DSC values most commonly fall in the 0.65–0.85 range, with higher values (≥0.80) typically observed for well-bounded, high-contrast targets such as prostate and liver lesions, intermediate values (0.70–0.80) for many head-and-neck and lung targets, and lower values (0.50–0.70) for diffuse or microscopic-spread CTVs such as those in pancreatic and esophageal disease [52] (consistent with the median GTV DSC of 0.67 reported by Gwynne et al. in esophageal cancer and the broad DSC range reported by Fokas et al. in pancreatic cancer). For organs at risk, reported DSC values span a wide range driven largely by structure size and tissue contrast, with parotid glands typically in the 0.75–0.85 range, large well-contrasted structures such as lungs and kidneys commonly in the 0.85–0.95 range, and small or low-contrast structures such as the optic chiasm, cochlea, and hippocampus more often in the 0.50–0.75 range [52,56,57]. Reported HD and HD95 values vary similarly by site, frequently approximating 3–10 mm for HD95 in well-circumscribed structures and exceeding 10–20 mm for complex, diffuse, or large CTVs. We emphasize that these ranges reflect a heterogeneous literature with variable reference standards, image modalities, slice thicknesses, and observer expertise, and they are reported here strictly to characterize the current landscape. They should not be interpreted as evidence-based acceptability criteria, but rather as a descriptive snapshot of the field intended to motivate—and ultimately be replaced by—the prospectively generated, disease-site-specific metric distributions described in our Phase 1 proposal below.
The study by Olch et al. [49] represents the only comprehensive, objective scoring system identified in our review, yet it has not been adopted by any other trial in the literature despite publication in 2023. This “Olch exception” provides insight into adoption barriers. The system requires: (1) commercial software (PlanIQ) not universally available; (2) creation of disease-site-specific scoring templates—a time-intensive process; (3) consensus on point allocation weighting—requiring multi-institutional agreement; and (4) validation that scores correlate with clinically meaningful outcomes. These prerequisites, while achievable, represent substantial implementation barriers that may explain limited adoption despite demonstrated feasibility.
Our focus on geometric contour accuracy, distinct from dosimetric plan quality, requires explicit justification given that dosimetric consequences ultimately determine clinical impact. Adequate contouring of both target volumes and OARs is critically important to the successful delivery of RT, and delivery of high-quality RT is necessary to validate clinical trial outcomes. Studies have shown that clinical trial outcomes are impacted by non-protocol-compliant RT delivery [4,5]. The contouring component of RT delivery has consistently been shown to contribute to the plurality of protocol deviations encountered in clinical trial QA analyses [25,32,44]. We contend that geometric standardization is not merely one approach to quality assurance, but rather a necessary prerequisite for meaningful dosimetric evaluation.
The study by Thor et al. [48] provides compelling illustration of this principle. Using deep-learning auto-segmentation to create geometrically consistent cardiac contours, they reanalyzed dosimetry from RTOG 0617, a landmark phase III trial comparing standard versus high-dose radiation for locally advanced non-small cell lung cancer. When standardized heart contours replaced the original clinician-drawn contours, reported cardiac doses increased significantly, not because treatment plans changed, but because the measured anatomical structure differed. Given that the dose-escalation arm showed unexpected increased mortality potentially related to cardiac dose [58], this geometric variation may have confounded the trial’s dosimetric analysis and clinical interpretation. Importantly, the original trial met its protocol-specified cardiac dose constraints based on the submitted contours, yet those constraints were measured on geometrically variable structures. This case exemplifies a fundamental problem: without geometric standardization, dosimetric evaluation lacks a stable reference frame. If different reviewers or institutions contour the same structure differently, then “meeting” a dosimetric constraint becomes a function of how the structure was drawn, not how it was treated. This case also illustrates the significant impact of accurate OAR contouring on downstream planning adequacy and ultimately on patient outcomes. Several studies specifically documented that OAR delineation failures constituted a major source of protocol deviations—53% in Stelmes et al. [14] and substantial proportions in other trials [2,44]. Yet, in practice, there is a greater emphasis on target volume accuracy. This finding represents a potential blind spot in quality assurance practices: while target volumes receive intensive scrutiny, OAR evaluation remains more cursory despite contributing significantly to protocol failures and patient outcomes.
Geometric standardization also offers practical advantages: it is computationally straightforward (compared to dose calculation verification), can be performed immediately upon contour submission before plan generation, and provides clear, structure-specific feedback to submitting centers. These attributes make geometric evaluation particularly suitable for real-time prospective quality assurance, potentially allowing correction before treatment initiation.
In summary, we position geometric contour accuracy not as the ultimate quality assurance endpoint, but as the essential foundation upon which all subsequent quality assurance must be built. Standardizing geometric evaluation is the necessary first step toward reliable dosimetric assessment and ultimately toward correlation of treatment quality with clinical outcomes.
It should be noted that our study has several limitations. While we did perform a systematic review of the literature, because there was such inconsistency in contour evaluation methods among studies, a more formal analysis such as a meta-analysis or statistical synthesis of our results could not be performed. As such, we were not able to explicitly define a comprehensive set of quantitative metrics used across studies. We also acknowledge that not all quantitative metrics are completely objective, as they may still involve some subjective elements in their application or interpretation. However, we attempted, in so far as we were able, to identify what would be considered uniform quantitative metrics for contour scoring in the literature.

Future Directions

Based on our systematic findings, we propose a phased implementation framework with specific, actionable steps. First (Phase 1), we recommend that baseline quantitative data are established without disrupting current workflows and using existing software. All new cooperative group trials should mandate reporting of geometric indices (DSC, HD95, volume ratio) for submitted contours compared to protocol reference contours. These should include both target volume and OAR contours. Data collection should be implemented as quality assurance metric reporting, not pass/fail criteria initially. We recommend the utilization of existing software (available in MIM, Velocity, Eclipse, RayStation, open-source tools). The end result of Phase I would be generation of a database of geometric metric distributions across trials.
The second step (Phase 2) would establish evidence-based acceptability criteria. This can be accomplished by creation of disease-site specific working groups to analyze Phase 1 data to determine metric distributions and to correlate geometric metrics with: (a) dosimetric impact, (b) expert reviewer classifications, and (c) clinical outcomes where available. The working groups would then establish preliminary acceptability thresholds informed by: 10th–90th percentile ranges from compliant cases, dosimetric sensitivity analysis (geometric deviation required to cause clinically meaningful dose change), and inter-observer variability benchmarks. We expect the outcomes of Phase 2 to be published consensus thresholds for priority disease sites.
As the third step (Phase 3), we recommend formal implementation of a scoring system into clinical trial RTQA. This would involve adapting an Olch-style scoring framework for each disease site using Phase 2 thresholds, weighting structures by clinical importance (informed by dose–outcome literature), accounting for structure-specific difficulty (e.g., sharp vs. gradual boundaries, motion effects), and implementing automated calculation with a manual expert override option for justified deviations. Both target volumes and OARs should be evaluated and scored. The end result of Phase 3 would be a validated scoring system implemented routinely for new clinical trials.
The next step (Phase 4) would be to refine thresholds based on clinical correlation. This would involve prospective correlation of quantitative contour scores with: overall protocol adherence rates, dosimetric quality metrics, and clinical outcomes (local control, toxicity) in completed trials. The goal would be to iteratively refine thresholds based on outcome data and identify which geometric metrics are most predictive of clinically meaningful deviations. The end result of Phase 4 would be evidence-based, outcome-validated evaluation standards.
Finally, the final phase (Phase 5) would be to achieve global harmonization and continuous improvement. This would involve cross-cooperative group consensus on minimum quantitative QA standards, integration into institutional credentialing requirements (similar to IROC phantom credentialing), and real-time automated feedback systems providing immediate quantitative assessment at submission. The end result of Phase 5 would be globally harmonized, evidence-based contour quality assurance practices.
Once quality assurance practices are standardized across clinical trials and the optimal methods of quantitative standardization are developed and validated as per the processes above, these standards can be implemented into clinical practice. Existing software, in conjunction with developing artificial intelligence platforms, can be used to generate contours and evaluate their accuracy as part of RTQA in local RT clinics. Ideally, this would be done following contour creation, but prior to plan generation, as these downstream steps impact the accuracy of subsequent planning steps [58].

5. Conclusions

In this systematic review, a plethora of evidence was found highlighting variability in contours across clinical trials in multiple disease sites conducted by cooperative groups globally. This variability was assessed and identified through subjective metrics used to score contours on clinical trials. We found a lack of evidence describing quantitative metrics used to score contours, indicating a major gap in radiation therapy quality assurance practices conducted by cooperative groups. This concerning lack of standardized quantitative metrics used to evaluate RT target and OAR contours can negatively impact clinical trial results. Effort is needed to create guidelines for objectively evaluating RT contours to be used across clinical trials to improve the quality and validity of RTQA.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/radiation6020020/s1. File S1: PRISMA checklist. Ref. [59] has been cited in Supplementary Materials.

Author Contributions

Conceptualization, L.T.T., H.G., Y.X.; methodology, L.T.T., Y.X.; formal analysis, L.T.T., Y.X.; investigation, L.T.T., Y.X.; data curation, J.L.; writing—original draft preparation, L.T.T., J.J.K.; writing—review and editing, all authors; visualization, L.T.T., Y.X.; supervision, Y.X. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Data Availability Statement

All of this data is publicly available in the research databases cites above.

Acknowledgments

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process: During the revision of this work the author(s) used Claude to effectively address reviewer comments. After using this tool/service, the author(s) reviewed and edited the content as needed. The authors have reviewed and edited the output and take full responsibility for the content of this publication.

Conflicts of Interest

The authors declare there are no conflicts of interest.

Abbreviations

The following abbreviations are used in this manuscript:
RTQARadiation therapy quality assurance
RTRadiation therapy

References

  1. Delaney, G.; Jacob, S.; Featherstone, C.; Barton, M. The role of radiotherapy in cancer treatment. Cancer 2005, 104, 1129–1137. [Google Scholar] [CrossRef] [Scilit]
  2. Tchelebi, L.; Winter, K.; Abrams, R.; Safran, H.; Regine, W.; McNulty, S.; Wu, A.; Du, K.; Seaward, S.; Bian, S.; et al. Analysis of Radiation Therapy Quality Assurance in NRG Oncology RTOG 0848. Int. J. Radiat. Oncol. 2021, 111, S40. [Google Scholar] [CrossRef] [Scilit]
  3. Cox, S.; Cleves, A.; Clementel, E.; Miles, E.; Staffurth, J.; Gwynne, S. Impact of deviations in target volume delineation—Time for a new RTQA approach? Radiother. Oncol. 2019, 137, 1–8. [Google Scholar] [CrossRef] [Scilit]
  4. Ohri, N.; Shen, X.; Dicker, A.P.; Doyle, L.A.; Harrison, A.S.; Showalter, T.N. Radiotherapy Protocol Deviations and Clinical Outcomes: A Meta-analysis of Cooperative Group Clinical Trials. JNCI J. Natl. Cancer Inst. 2013, 105, 387–393. [Google Scholar] [CrossRef] [Scilit]
  5. Abrams, R.A.; Winter, K.A.; Regine, W.F.; Safran, H.; Hoffman, J.P.; Lustig, R.; Konski, A.A.; Benson, A.B.; Macdonald, J.S.; Rich, T.A.; et al. Failure to Adhere to Protocol Specified Radiation Therapy Guidelines Was Associated with Decreased Survival in RTOG 9704—A Phase III Trial of Adjuvant Chemotherapy and Chemoradiotherapy for Patients with Resected Adenocarcinoma of the Pancreas. Int. J. Radiat. Oncol. 2012, 82, 809–816. [Google Scholar] [CrossRef] [Scilit]
  6. Fairchild, A.; Straube, W.; Laurie, F.; Followill, D. Does Quality of Radiation Therapy Predict Outcomes of Multicenter Cooperative Group Trials? A Literature Review. Int. J. Radiat. Oncol. 2013, 87, 246–260. [Google Scholar] [CrossRef] [Scilit]
  7. Albert, J.M.; Das, P. Quality Indicators in Radiation Oncology. Int. J. Radiat. Oncol. 2013, 85, 904–911. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  8. de Balincourt, C.; Lacombe, D.; Coens, C.; Dulk, M.D.; Machiels, J.-P.; Weber, D.; Spahn, M.; Salgado, R.; Kasper, B.; Hau, P.; et al. Multidisciplinary quality assurance and control in oncological trials: Perspectives from European Organisation for Research and Treatment of Cancer (EORTC). Eur. J. Cancer 2017, 86, 91–100. [Google Scholar] [CrossRef] [Scilit]
  9. Bekelman, J.E.; Deye, J.A.; Vikram, B.; Bentzen, S.M.; Bruner, D.; Curran, W.J.; Dignam, J.; Efstathiou, J.A.; FitzGerald, T.; Hurkmans, C.; et al. Redesigning Radiotherapy Quality Assurance: Opportunities to Develop an Efficient, Evidence-Based System to Support Clinical Trials—Report of the National Cancer Institute Work Group on Radiotherapy Quality Assurance. Int. J. Radiat. Oncol. 2012, 83, 782–790. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  10. McDowell, L.J.; Corry, J. A Call to Arms: Radiation Therapy Quality Assurance in the Next Generation of Clinical Trials. Int. J. Radiat. Oncol. 2018, 102, 1590–1591. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  11. Corrigan, K.L.; Kry, S.; Howell, R.M.; Kouzy, R.; Jaoude, J.A.; Patel, R.R.; Jhingran, A.; Taniguchi, C.; Koong, A.C.; McAleer, M.F.; et al. The radiotherapy quality assurance gap among phase III cancer clinical trials. Radiother. Oncol. 2021, 166, 51–57. [Google Scholar] [CrossRef] [Scilit]
  12. Lin, D.; Lapen, K.; Sherer, M.V.; Kantor, J.; Zhang, Z.; Boyce, L.M.; Bosch, W.; Korenstein, D.; Gillespie, E.F. A Systematic Review of Contouring Guidelines in Radiation Oncology: Analysis of Frequency, Methodology, and Delivery of Consensus Recommendations. Int. J. Radiat. Oncol. 2020, 107, 827–835. [Google Scholar] [CrossRef] [Scilit]
  13. Wills, L.; Maggs, R.; Lewis, G.; Jones, G.; Nixon, L.; Staffurth, J.; Crosby, T. Quality assurance of the SCOPE 1 trial in oesophageal radiotherapy. Radiat. Oncol. 2017, 12, 179. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  14. Stelmes, J.-J.; Vu, E.; Grégoire, V.; Simon, C.; Clementel, E.; Kazmierska, J.; Grant, W.; Ozsahin, M.; Tomsej, M.; Vieillevigne, L.; et al. Quality assurance of radiotherapy in the ongoing EORTC 1420 “Best of” trial for early stage oropharyngeal, supraglottic and hypopharyngeal carcinoma: Results of the benchmark case procedure. Radiat. Oncol. 2021, 16, 81. [Google Scholar] [CrossRef] [Scilit]
  15. Okamoto, H.; Murakami, N.; Isohashi, F.; Kasamatsu, T.; Hasumi, Y.; Iijima, K.; Nishioka, S.; Nakamura, S.; Nakamura, M.; Nishio, T.; et al. Dummy-run for standardizing plan quality of intensity-modulated radiotherapy for postoperative uterine cervical cancer: Japan Clinical Oncology Group study (JCOG1402). Radiat. Oncol. 2019, 14, 133. [Google Scholar] [CrossRef] [Scilit]
  16. Musat, E.; Roelofs, E.; Bar-Deroma, R.; Fenton, P.; Gulyban, A.; Collette, L.; Stupp, R.; Weber, D.C.; Davis, J.B.; Aird, E.; et al. Dummy run and conformity indices in the ongoing EORTC low-grade glioma trial 22033-26033: First evaluation of quality of radiotherapy planning. Radiother. Oncol. 2010, 95, 218–224. [Google Scholar] [CrossRef] [Scilit]
  17. Gondi, V.; Cui, Y.; Mehta, M.P.; Manfredi, D.; Xiao, Y.; Galvin, J.M.; Rowley, H.; Tome, W.A. Real-Time Pretreatment Review Limits Unacceptable Deviations on a Cooperative Group Radiation Therapy Technique Trial: Quality Assurance Results of RTOG 0933. Int. J. Radiat. Oncol. 2015, 91, 564–570. [Google Scholar] [CrossRef] [Scilit]
  18. Gwynne, S.; Spezi, E.; Wills, L.; Nixon, L.; Hurt, C.; Joseph, G.; Evans, M.; Griffiths, G.; Crosby, T.; Staffurth, J. Toward Semi-automated Assessment of Target Volume Delineation in Radiotherapy Trials: The SCOPE 1 Pretrial Test Case. Int. J. Radiat. Oncol. 2012, 84, 1037–1042. [Google Scholar] [CrossRef] [Scilit]
  19. Christiaens, M.; Collette, S.; Overgaard, J.; Gregoire, V.; Kazmierska, J.; Castadot, P.; Giralt, J.; Grant, W.; Tomsej, M.; Bar-Deroma, R.; et al. Quality assurance of radiotherapy in the ongoing EORTC 1219-DAHANCA-29 trial for HPV/p16 negative squamous cell carcinoma of the head and neck: Results of the benchmark case procedure. Radiother. Oncol. 2017, 123, 424–430. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  20. Fairchild, A.; Langendijk, J.A.; Nuyts, S.; Scrase, C.; Tomsej, M.; Schuring, D.; Gulyban, A.; Ghosh, S.; Weber, D.C.; Budach, W. Quality assurance for the EORTC 22071–26071 study: Dummy run prospective analysis. Radiat. Oncol. 2014, 9, 248. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  21. Lambrecht, M.; Clementel, E.; Sonke, J.-J.; Nestle, U.; Adebahr, S.; Guckenberger, M.; Andratschke, N.; Weber, D.C.; Verheij, M.; Hurkmans, C.W. Radiotherapy quality assurance of SBRT for patients with centrally located lung tumours within the multicentre phase II EORTC Lungtech trial: Benchmark case results. Radiother. Oncol. 2019, 132, 63–69. [Google Scholar] [CrossRef] [Scilit]
  22. Fokas, E.; Clifford, C.; Spezi, E.; Joseph, G.; Branagan, J.; Hurt, C.; Nixon, L.; Abrams, R.; Staffurth, J.; Mukherjee, S. Comparison of investigator-delineated gross tumor volumes and quality assurance in pancreatic cancer: Analysis of the pretrial benchmark case for the SCALOP trial. Radiother. Oncol. 2015, 117, 432–437. [Google Scholar] [CrossRef] [Scilit]
  23. Sassowsky, M.; Gut, P.; Hölscher, T.; Hildebrandt, G.; Müller, A.-C.; Najafi, Y.; Kohler, G.; Kranzbühler, H.; Guckenberger, M.; Zwahlen, D.R.; et al. Use of EORTC Target Definition Guidelines for Dose-Intensified Salvage Radiation Therapy for Recurrent Prostate Cancer: Results of the Quality Assurance Program of the Randomized Trial SAKK 09/10. Int. J. Radiat. Oncol. 2013, 87, 534–541. [Google Scholar] [CrossRef] [Scilit]
  24. Fenton, P.A.; Hurkmans, C.; Gulyban, A.; van der Leer, J.; Matzinger, O.; Poortmans, P.; Collette, L.; Bolla, M. Quality assurance of the EORTC 22043-30041 trial in post-operative radiotherapy in prostate cancer: Results of the Dummy Run procedure. Radiother. Oncol. 2013, 107, 346–351. [Google Scholar] [CrossRef] [Scilit]
  25. Cloak, K.; Jameson, M.G.; Paneghel, A.; Wiltshire, K.; Kneebone, A.; Pearse, M.; Sidhom, M.; Tang, C.; Fraser-Browne, C.; Holloway, L.C.; et al. Contour variation is a primary source of error when delivering post prostatectomy radiotherapy: Results of the Trans-Tasman Radiation Oncology Group 08.03 Radiotherapy Adjuvant Versus Early Salvage (RAVES) benchmarking exercise. J. Med. Imaging Radiat. Oncol. 2019, 63, 390–398. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  26. Fokas, E.; Spezi, E.; Patel, N.; Hurt, C.; Nixon, L.; Chu, K.; Joseph, G.; Staffurth, J.; Abrams, R.; Mukherjee, S. P-222 Analysis of tumour contours and radiotherapy planning of “on-trial” patients undergoing chemoradiotherapy (CRT) in SCALOP trial: Does pre-trial Radiotherapy Quality Assurance (RTQA) improve the quality of “on-trial” radiotherapy? Ann. Oncol. 2016, 27, ii64–ii65. [Google Scholar] [CrossRef] [Scilit]
  27. Matzinger, O.; Poortmans, P.; Giraud, J.-Y.; Maingon, P.; Budiharto, T.; Bergh, A.C.M.V.D.; Davis, J.B.; Musat, E.; Ataman, F.; Huyskens, D.P.; et al. Quality assurance in the 22991 EORTC ROG trial in localized prostate cancer: Dummy run and individual case review. Radiother. Oncol. 2009, 90, 285–290. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  28. Kearvell, R.; Haworth, A.; Ebert, M.A.; Murray, J.; Hooton, B.; Richardson, S.; Joseph, D.J.; Lamb, D.; Spry, N.A.; Duchesne, G.; et al. Quality improvements in prostate radiotherapy: Outcomes and impact of comprehensive quality assurance during the TROG 03.04 ‘RADAR’ trial. J. Med. Imaging Radiat. Oncol. 2013, 57, 247–257. [Google Scholar] [CrossRef] [Scilit]
  29. Corry, J.; Moore, A.; Kenny, L.; Wratten, C.; Fua, T.; Lin, C.; Porceddu, S.; Liu, C.; Ruemelin, M.; Sharkey, A.; et al. Radiotherapy quality assurance in the TROG 12.01 randomised trial and its impact on loco-regional failure. Front. Oncol. 2024, 13, 1333098. [Google Scholar] [CrossRef] [Scilit]
  30. Okamoto, H.; Murakami, N.; Isohashi, F.; Kasamatsu, T.; Hasumi, Y.; Kobayashi, H.; Ishikawa, M.; Nakamura, M.; Nishio, T.; Igaki, H.; et al. Plan quality association between dummy run and individual case review in a prospective multi-institutional clinical trial of postoperative cervical cancer patients treated with intensity-modulated radiotherapy: Japan clinical Oncology Group study (JCOG1402). Radiother. Oncol. 2023, 183, 109630. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  31. Hardcastle, N.; Cook, O.; Ray, X.; Moore, A.; Moore, K.L.; Pryor, D.; Rossi, A.; Foroudi, F.; Kron, T.; Siva, S. Personalising treatment plan quality review with knowledge-based planning in the TROG 15.03 trial for stereotactic ablative body radiotherapy in primary kidney cancer. Radiat. Oncol. 2021, 16, 142. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  32. Tchelebi, L.T.; Winter, K.A.; Abrams, R.A.; Safran, H.P.; Regine, W.F.; McNulty, S.; Wu, A.; Du, K.L.; Seaward, S.A.; Bian, S.X.; et al. Analysis of Radiation Therapy Quality Assurance in NRG Oncology RTOG 0848. Int. J. Radiat. Oncol. 2023, 118, 107–114. [Google Scholar] [CrossRef] [Scilit]
  33. Lukovic, J.; Moore, A.J.; Lee, M.T.; Willis, D.; Ahmed, S.; Akra, M.; Hortobagyi, E.; Kron, T.; Joon, D.L.; Liu, A.; et al. The Feasibility of Quality Assurance in the TOPGEAR International Phase 3 Clinical Trial of Neoadjuvant Chemoradiation Therapy for Gastric Cancer (an Intergroup Trial of the AGITG/TROG/NHMRC CTC/EORTC/CCTG). Int. J. Radiat. Oncol. 2023, 117, 1096–1106. [Google Scholar] [CrossRef] [Scilit]
  34. Trada, Y.; Kneebone, A.; Paneghel, A.; Pearse, M.; Sidhom, M.; Tang, C.; Wiltshire, K.; Haworth, A.; Fraser-Browne, C.; Martin, J. Optimizing Radiation Therapy Quality Assurance in Clinical Trials: A TROG 08.03 RAVES Substudy. Int. J. Radiat. Oncol. 2015, 93, 1045–1051. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  35. Mir, R.; Groom, N.; Mistry, H.B.; Wilson, E.; Faivre-Finn, C. Association between radiotherapy protocol variations and outcome in the CONVERT trial. Clin. Transl. Radiat. Oncol. 2022, 39, 100560. [Google Scholar] [CrossRef] [Scilit]
  36. Haas, R.; Stelmes, J.; Zaffaroni, F.; Sauvé, N.; Clementel, E.; Bar-Deroma, R.; Le Péchoux, C.; Litière, S.; Marreaud, S.; Alyamani, N.; et al. Critical impact of radiotherapy protocol compliance and quality in the treatment of retroperitoneal sarcomas: Results from the EORTC 62092-22092 STRASS trial. Cancer 2022, 128, 2796–2805. [Google Scholar] [CrossRef] [Scilit]
  37. Fairchild, A.; Weber, D.C.; Bar-Deroma, R.; Gulyban, A.; Fenton, P.A.; Stupp, R.; Baumert, B.G. Quality assurance in the EORTC 22033–26033/CE5 phase III randomized trial for low grade glioma: The digital individual case review. Radiother. Oncol. 2012, 103, 287–292. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  38. Toita, T.; Kato, S.; Ishikura, S.; Tsujino, K.; Kodaira, T.; Uno, T.; Hatano, K.; Sakurai, H.; Niibe, Y.; Kazumoto, T.; et al. Radiotherapy quality assurance of the Japanese Gynecologic Oncology Group study (JGOG1066): A cooperative phase II study of concurrent chemoradiotherapy for uterine cervical cancer. Int. J. Clin. Oncol. 2011, 16, 379–386. [Google Scholar] [CrossRef] [Scilit]
  39. Beck, M.; Sassowsky, M.; Schär, S.; Mathier, E.; Halter, M.; Zwahlen, D.R.; Hölscher, T.; Arnold, W.; Polat, B.; Hildebrandt, G.; et al. Adherence to Contouring and Treatment Planning Requirements Within a Multicentric Trial: Results of the Quality Assurance of the SAKK 09/10 trial. Int. J. Radiat. Oncol. 2022, 113, 80–91. [Google Scholar] [CrossRef] [Scilit]
  40. Abrunhosa-Branquinho, A.N.; Bar-Deroma, R.; Collette, S.; Clementel, E.; Liu, Y.; Hurkmans, C.W.; Feuvret, L.; Van Beek, K.; Bent, M.v.D.; Baumert, B.G.; et al. Radiotherapy quality assurance for the RTOG 0834/EORTC 26053-22054/NCIC CTG CEC.1/CATNON intergroup trial “concurrent and adjuvant temozolomide chemotherapy in newly diagnosed non-1p/19q deleted anaplastic glioma”: Individual case review analysis. Radiother. Oncol. 2018, 127, 292–298. [Google Scholar] [CrossRef] [Scilit]
  41. Zhong, H.; Men, K.; Wang, J.; van Soest, J.; Rosenthal, D.; Dekker, A.; Zhang, Z.; Xiao, Y. The Impact of Clinical Trial Quality Assurance on Outcome in Head and Neck Radiotherapy Treatment. In Proceedings of the 59th Annual Scientific Meeting of the American-Society-for-Radiation-Oncology (ASTRO), San Diego, CA, USA, 24–27 September 2017; p. 792. [Google Scholar]
  42. Sanuki-Fujimoto, N.; Ishikura, S.; Hayakawa, K.; Kubota, K.; Nishiwaki, Y.; Tamura, T. Radiotherapy quality assurance review in a multi-center randomized trial of limited-disease small cell lung cancer: The Japan Clinical Oncology Group (JCOG) trial 0202. Radiat. Oncol. 2009, 4, 16. [Google Scholar] [CrossRef] [Scilit]
  43. Schiff, J.P.; Lee, Y.; Wang, Y.; Perkins, S.M.; Kessel, S.K.; Fitzgerald, T.J.; Larrier, N.A.; Michalski, J.M. An Analysis of Major Target Deviations in Craniospinal Irradiation Treatment Plans for Patients with Intermediate-Risk Medulloblastoma Within a Phase 3 Clinical Trial (Children’s Oncology Group Study ACNS0331). Adv. Radiat. Oncol. 2022, 8, 101083. [Google Scholar] [CrossRef] [Scilit]
  44. Khaw, P.; Do, V.; Lim, K.; Cunninghame, J.; Dixon, J.; Vassie, J.; Bailey, M.; Johnson, C.; Kahl, K.; Gordon, C.; et al. Radiotherapy Quality Assurance in the PORTEC-3 (TROG 08.04) Trial. Clin. Oncol. 2022, 34, 198–204. [Google Scholar] [CrossRef] [Scilit]
  45. Yang, H.; Mir, R.; Díez, P.; Tsang, Y.; Conibear, J.; Simões, R.; Cox, S.; Webster, A.; Nabi, Z.; Eaton, D.; et al. Provision of Organ at Risk Contouring Guidance in UK Radiotherapy Clinical Trials. Clin. Oncol. 2020, 32, e60–e66. [Google Scholar] [CrossRef] [Scilit]
  46. Mir, R.; Kelly, S.M.; Xiao, Y.; Moore, A.; Clark, C.H.; Clementel, E.; Corning, C.; Ebert, M.; Hoskin, P.; Hurkmans, C.W.; et al. Organ at risk delineation for radiation therapy clinical trials: Global Harmonization Group consensus guidelines. Radiother. Oncol. 2020, 150, 30–39. [Google Scholar] [CrossRef] [Scilit]
  47. Harmon, G.; Chan, D.; Lee, B.; Miller, C.; Gorbonos, A.; Gupta, G.; Quek, M.; Woods, M.; Savir-Baruch, B.; Harkenrider, M.M.; et al. Validating Modern NRG Oncology Pelvic Nodal and Groupe Francophone de Radiothérapie Urologique Prostate Bed Contouring Guidelines for Post-Prostatectomy Salvage Radiation: A Secondary Analysis of the LOCATE Trial. Int. J. Radiat. Oncol. 2021, 111, 1195–1203. [Google Scholar] [CrossRef] [Scilit]
  48. Thor, M.; Apte, A.; Haq, R.; Iyer, A.; LoCastro, E.; Deasy, J.O. Using Auto-Segmentation to Reduce Contouring and Dose Inconsistency in Clinical Trials: The Simulated Impact on RTOG 0617. Int. J. Radiat. Oncol. 2020, 109, 1619–1626. [Google Scholar] [CrossRef] [Scilit]
  49. Olch, A.J.; Gopalakrishnan, M.; Murphy, E.S.; MacDonald, S.M.; Hua, C.-H. Toward Systematic Assessment and Improvement of Radiation Therapy Plan Quality of Cooperative Group Trial Submissions: A Report From the Children’s Oncology Group. Pract. Radiat. Oncol. 2023, 13, e374–e382. [Google Scholar] [CrossRef] [Scilit]
  50. Brooks, C.; Miles, E.; Hoskin, P.J. Radiotherapy trial quality assurance processes: A systematic review. Lancet Oncol. 2024, 25, e104–e113. [Google Scholar] [CrossRef] [Scilit]
  51. Vinod, S.K.; Jameson, M.G.; Min, M.; Holloway, L.C. Uncertainties in volume delineation in radiation oncology: A systematic review and recommendations for future studies. Radiother. Oncol. 2016, 121, 169–179. [Google Scholar] [CrossRef] [Scilit]
  52. Sherer, M.V.; Lin, D.; Elguindi, S.; Duke, S.; Tan, L.-T.; Cacicedo, J.; Dahele, M.; Gillespie, E.F. Metrics to evaluate the performance of auto-segmentation for radiation treatment planning: A critical review. Radiother. Oncol. 2021, 160, 185–191. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  53. Taha, A.A.; Hanbury, A. Metrics for evaluating 3D medical image segmentation: Analysis, selection, and tool. BMC Med. Imaging 2015, 15, 29. [Google Scholar] [CrossRef] [Scilit]
  54. Nikolov, S.; Blackwell, S.; Zverovitch, A.; Mendes, R.; Livne, M.; De Fauw, J.; Patel, Y.; Meyer, C.; Askham, H.; Romera-Paredes, B.; et al. Clinically Applicable Segmentation of Head and Neck Anatomy for Radiotherapy: Deep Learning Algorithm Development and Validation Study. J. Med. Internet Res. 2021, 23, e26151. [Google Scholar] [CrossRef] [Scilit]
  55. Vaassen, F.; Hazelaar, C.; Vaniqui, A.; Gooding, M.; van der Heyden, B.; Canters, R.; van Elmpt, W. Evaluation of measures for assessing time-saving of automatic organ-at-risk segmentation in radiotherapy. Phys. Imaging Radiat. Oncol. 2020, 13, 1–6. [Google Scholar] [CrossRef] [Scilit]
  56. Brouwer, C.L.; Steenbakkers, R.J.H.M.; van den Heuvel, E.; Duppen, J.C.; Navran, A.; Bijl, H.P.; Chouvalova, O.; Burlage, F.R.; Meertens, H.; Langendijk, J.A.; et al. 3D Variation in delineation of head and neck organs at risk. Radiat. Oncol. 2012, 7, 32. [Google Scholar] [CrossRef] [Scilit]
  57. Brouwer, C.L.; Steenbakkers, R.J.; Bourhis, J.; Budach, W.; Grau, C.; Grégoire, V.; van Herk, M.; Lee, A.; Maingon, P.; Nutting, C.; et al. CT-based delineation of organs at risk in the head and neck region: DAHANCA, EORTC, GORTEC, HKNPCSG, NCIC CTG, NCRI, NRG Oncology and TROG consensus guidelines. Radiother. Oncol. 2015, 117, 83–90. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  58. Cox, B.W.; Teckie, S.; Kapur, A.; Chou, H.; Potters, L. Prospective Peer Review in Radiation Therapy Treatment Planning: Long-Term Results From a Longitudinal Study. Pr. Radiat. Oncol. 2020, 10, e199–e206. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  59. Page, M.J.; McKenzie, J.E.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E.; et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ 2021, 372, n71. [Google Scholar] [CrossRef] [Scilit]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Article Metrics

Citations

Article Access Statistics

Journal Statistics
Multiple requests from the same IP address are counted as one view.
Itraconazole (ITRA) with Standard Radiotherapy (RT) and Temozolomide (TMZ) in Patients with Newly Diagnosed Glioblastoma Multiforme (GBM) ITRA-RAD: Phase I Clinical Study
Previous
Mepilex Dressings in Managing Radiation-Induced Moist Desquamation in Head and Neck Cancer
Next