Next Article in Journal
Mepilex Dressings in Managing Radiation-Induced Moist Desquamation in Head and Neck Cancer
Previous Article in Journal
Itraconazole (ITRA) with Standard Radiotherapy (RT) and Temozolomide (TMZ) in Patients with Newly Diagnosed Glioblastoma Multiforme (GBM) ITRA-RAD: Phase I Clinical Study
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Radiation
Volume 6
Issue 2
10.3390/radiation6020020
radiation-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Systematic Review

Systematic Review of Quantitative Metrics to Standardize Contours on Radiotherapy Clinical Trials

by
Leila T. Tchelebi
1,2,*,
Janice Lester
1,
Joshua J. Kim
2,
Huaizhi Geng
3 and
Ying Xiao
3,4
1
Northwell Health, Lake Success, NY 11042, USA
2
Zucker School of Medicine at Hofstra/Northwell, Hempstead, NY 11549, USA
3
Department of Radiation Oncology, University of Pennsylvania, Philadelphia, PA 19104, USA
4
Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104, USA
*
Author to whom correspondence should be addressed.
Radiation 2026, 6(2), 20; https://doi.org/10.3390/radiation6020020
Submission received: 22 April 2026 / Revised: 11 May 2026 / Accepted: 12 May 2026 / Published: 5 June 2026
Browse Figures
Review Reports Versions Notes

Simple Summary

Radiation therapy quality assurance (RTQA) is critical to the success of RT and impacts trial outcomes. We aimed to evaluate the existence of standardized quantitative metrics to grade contours as part of RTQA and to identify barriers to standardization. Our study endpoints were quantitative metrics being used to guide contour scoring. We initially planned quantitative synthesis (meta-analysis) of geometric metric distributions across studies. However, the heterogeneity in evaluation methods, the predominance of qualitative assessment, and the lack of standardized reporting precluded meaningful quantitative pooling. We therefore performed structured qualitative synthesis, thematic analysis, and gap assessment. We found a concerning lack of standardized quantitative metrics to evaluate RT contours, which can negatively impact clinical trial results. Further, there is a lack of methodological consistency required for systematic quantitative synthesis, reinforcing the need for standardization. Effort is needed to create guidelines for quantitatively evaluating RT contours to use across clinical trials to improve the quality and validity of RTQA.

Abstract

Radiation therapy quality assurance (RTQA) is critical to the success of RT and impacts trial outcomes. We aimed to evaluate the existence of standardized quantitative metrics to grade contours as part of RTQA and to identify barriers to standardization. A systematic review was conducted according to PRISMA guidelines. PubMed, Web of Science, and Embase were used to search relevant studies published from 2014–2024. Our endpoints were quantitative metrics used to guide contour scoring. We initially planned quantitative synthesis (meta-analysis) of geometric metric distributions across studies. However, the heterogeneity in evaluation methods, predominance of qualitative assessment, and lack of standardized reporting precluded meaningful quantitative pooling. We therefore performed structured qualitative synthesis, thematic analysis, and gap assessment. A total of 37 studies were included, 20 of which were prospective and 17 were retrospective. Of the prospective, 12 reported the results of benchmark case/dummy run reviews, 4 were a combination of benchmark case/dummy runs and individual QA analyses done pre- or post-treatment, 1 used knowledge-based planning to evaluate plans in real-time, and 3 involved real-time prospective QA. Only one reported the use of an objective QA scoring system. There is a concerning lack of standardized quantitative metrics to evaluate RT contours on clinical trials and a lack of methodological consistency required for systematic quantitative synthesis, reinforcing the need for standardization. Effort is needed to create guidelines for quantitatively evaluating RT contours to use across clinical trials to improve the quality and validity of RTQA.

1. Introduction

Approximately 50% of cancer patients receive radiation therapy (RT) as a component of their cancer care [1]. Clinical trials are essential to establish best practices in delivering RT in the treatment of cancer patients. One of the foundations of successful RT delivery is accurate contouring of both target volumes and organs at risk (OARs). However, contour delineation can be highly variable among providers and is consistently shown to account for the majority of protocol deviations encountered on clinical trials [2,3]. Studies have shown that major protocol deviations can adversely impact trial outcomes [4,5,6].
To ensure uniform and accurate RT delivery on cooperative group trials, radiation therapy quality assurance (RTQA) programs have been widely implemented both in the United States and abroad [7,8,9,10]. Clinical trial RTQA programs consist of multiple components including accurate contour delineation, treatment planning, and treatment delivery. This review focuses strictly on contour evaluation processes as distinct from plan dosimetric quality. Contour geometric accuracy refers to the spatial agreement between delineated structures and their true anatomical boundaries or consensus definitions, thus measuring the precision of delineation itself, independent of subsequent dose calculations. Geometric accuracy can be measured using metrics such as the Dice similarity coefficient (DSC), Hausdorff distance (HD), mean distance to agreement, and volume ratios. It represents the foundation upon which all treatment planning is built. Plan dosimetric quality refers to how well the calculated dose distribution meets protocol-specified constraints and objectives, and is evaluated using dose-volume parameters such as planning target volume (PTV) coverage metrics (V95%, D95%, conformity indices) and organ-at-risk constraints (Dmean, Dmax, VxGy). Dosimetric quality depends on both the accuracy of the underlying contours and the quality of the optimization and planning process, with contours being the foundation on which subsequent treatment planning is based on [9].
Despite the increasing implementation of RTQA involving contour review across cooperative group clinical trials, it is not uniform and can be highly variable even among strictly designed phase III randomized controlled studies [11]. While there has been a surge in the development and use of comprehensive guidelines to delineate target volume and OAR contours [12], there is a lack of guidance on how to assess and score the quality of submitted contours. Herein, we conducted a review to systematically assess quantitative metrics being used to guide RTQA evaluation of target volume and OAR contours on cooperative group clinical trials. While we acknowledge that the final treatment plan/dosimetry are ultimately the clinical concern, we contend that standardized geometric evaluation is a prerequisite for meaningful interpretation of dosimetric findings and attribution of protocol deviations.

2. Materials and Methods

A systematic review was conducted according to the guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) to investigate evidence for RTQA performed on randomized controlled trials conducted by a major cooperative group in the United States or abroad specifically involving assessment of target volume or OAR contours or both. The search was performed by a professional librarian in PubMed, Web of Science, and Embase. Relevant studies conducted in the English language between 2004 and 2024 were included. Search terms were broad to reflect the variations of nomenclature for “radiation therapy” and were combined with “contour” or “organ at risk” or “target coverage” or “dosimetric coverage” and major cooperative groups in the United States and abroad. The references were imported into Covidence and two authors independently screened the comprehensive list of articles. The PRISMA Flow Diagram is presented in Figure 1. This review was performed in accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines and has not been registered. A PRISMA checklist is provided in the Supplement (File S1).
Two authors (LT, HG) independently screened the comprehensive list of articles, and one (LT) assessed full-text documents to identify only full-length articles and excluded review articles, abstracts, editorials, and letters. The remaining records were retrieved and full texts were reviewed for eligibility. Our study endpoints were quantitative metrics being used to guide RTQA scoring of target volume and OAR contours.
Our primary endpoint was the identification and characterization of quantitative geometric metrics used to evaluate contour accuracy in cooperative group clinical trials. We defined quantitative geometric metrics as numerical measurements that can be systematically applied to evaluate spatial agreement between submitted contours and reference standards, including but not limited to: overlap indices (Dice similarity coefficient, Jaccard index, conformity index); distance metrics (Hausdorff distance, mean distance to agreement, surface distance); volume comparisons (volume ratio, added/missed volume); and centroid displacement. We explicitly excluded metrics of dosimetric quality (e.g., PTV coverage parameters such as V95% or D95%, and OAR dose constraints such as Dmean or VxGy) as these assess planning optimization and dose calculation rather than contouring accuracy per se. Studies that assessed both contour geometry and dosimetry were included only when contour-specific evaluation methods were separately reported and could be independently analyzed.
We characterized evaluation methods as objective when they utilized computer-calculated metrics with defined thresholds that could be applied consistently across evaluators, versus subjective when they relied primarily on expert visual review or qualitative assessment without quantifiable criteria. Studies employing “comparison to gold standard” approaches were categorized based on whether this comparison was qualitative (expert assessment of similarity) or quantitative (calculation of geometric indices).
We initially planned quantitative synthesis (meta-analysis) of geometric metric distributions across studies. However, preliminary analysis revealed that the heterogeneity in evaluation methods, the predominance of qualitative assessment approaches, and the lack of standardized reporting precluded meaningful quantitative pooling of results. This limitation itself represents a key finding: the field currently lacks the methodological consistency required for systematic quantitative synthesis, reinforcing the imperative for standardization. Thus, our aim was to provide interpretive synthesis of the current landscape of contour assessment on cooperative group trials.

3. Results

Our initial search yielded a total of 1592 studies of which 509 were found to be duplicates and were removed (Figure 1). The remaining 1083 studies were screened for relevancy and 966 were found to be irrelevant and were excluded. The remaining 117 full text studies were assessed for eligibility, and 79 were excluded: 47 were the wrong study design (primarily guidelines and retrospective inter-observer studies), 19 were only available as abstracts, 7 reported the wrong outcomes, 5 were conducted in the wrong setting, and 2 reported on the wrong intervention. The remaining 37 studies were included.

3.1. Study Characteristics

Twenty studies were conducted prospectively (Table 1). Twelve of these reported results of benchmark cases or dummy runs performed as part of the credentialing process for centers participating in cooperative group trials, with comprehensive assessments of target volume and/or OAR contouring [13,14,15,16,17,18,19,20,21,22,23,24,25]. Most of these studies reported substantial variability in radiotherapy contouring and planning, despite the availability of comprehensive protocols to guide treatment [13,14,15,18,19,20,21,22,26]. Of the remaining seven prospective QA studies, four reported a combination of dummy run reviews and individual QA analyses [27,28,29,30], and three involved real-time prospective individual case review RTQA [31,32,33]. Lastly, one study involved prospective real-time RTQA using knowledge-based planning (KBP) in which a treatment plan was created by using the KBP model; this was then compared with the submitted plan for each OAR dose constraint, which was then evaluated subjectively by a trial radiation oncologist who assessed contours for adequacy [31].
Seventeen studies were retrospective QA analyses of patients treated on cooperative group trials (Table 2). Twelve of these involved retrospective reviews of contouring (and plans) for patients treated on major cooperative group trials, without providing quantitative metrics used to review the contours [18,26,35,36,37,38,39,40,41,42,43,44]. Three studies dealt with assessing contouring guidelines for either OARs or target volumes on clinical trials, attempting to provide quantitative guidance on contour scoring [45,46,47].
The remaining two studies used computer programs to perform more quantitative assessments of contours as part of the RTQA process. The study by Thor et al. compared doses between deep-learning (DL) segmented hearts and trial hearts and found that, using a coherent heart definition as enabled through their open-source DL algorithm, the trial heart doses in RTOG 0617 were significantly higher than previously reported [48]. The only study to systematically analyze overall RT plan quality for an entire cohort of patients treated in a cooperative group clinical trial was conducted by Olch et al. in patients treated on the Children’s Oncology Group study ACNS1123 [49]. The authors used a commercially available plan-quality-scoring algorithm to create a scoring system. PTVs could earn up to 70 points, while OARs could earn up to 30 points, and scores were adjusted based on the difficulty in achieving the structure dose requirements. The brain stem, PTV boost, and cochlea lost the most points. The authors concluded that future clinical trials could potentially use this method to reduce plan quality variability, which may improve outcomes.
Table 2. Retrospective radiation therapy quality assurance data.
Table 2. Retrospective radiation therapy quality assurance data.
Author, yearRegionDisease SiteStudy TypeRelevant Clinical TrialQA ApproachContour Evaluation MethodQuantitative Metrics UsedKey Findings
Mir,
2020 [46]		Multi-nationalMulti-siteOAR Contouring Guideline Development for Clinical TrialsN/AGlobal Quality Assurance of Radiation Therapy Clinical Trials Harmonization Group working group (GHG) review of OAR structures across clinical trials Expert reviewNone reportedConsensus guidance on clinical trial OARs with peer-reviewed descriptions
Yang, 2020 [45]Europe/
UK		Multi-siteOAR Contouring Guideline Development for Clinical TrialsNIHRCRN trialsTwo-round Delphi assessment to determine OAR descriptions Expert reviewNone reportedConsensus guidance on clinical trial OARs with peer-reviewed descriptions
Harmon, 2021 [47]Europe/
UK		ProstateContouring guideline validation LOCATE trialPatterns of recurrence used to evaluate validity of contouring guidelines on trialExpert reviewNone reportedValidated updated contouring guidelines
Mir,
2023 [35]		Europe/
UK		LungRetrospective case reviewCONVERT TrialExpert generated contours were compared to the patient contoursExpert reviewNone reportedSignificant non-compliance was found with respect to OAR contours.
Haas, 2022 [36]Europe/EORTCRetroperitoneal sarcomaRetrospective case reviewEORTC 62092–22092 STRASS Expert review of cases treated on trial Expert reviewNone reportedImproved recurrence-free survival for patients treated without protocol non-compliance
Fairchild, 2012 [37]Europe/EORTCCNSRetrospective case review EORTC 22033–26033/NCIC CE5 Phase IIIExpert review of cases treated on trialExpert reviewNone reportedSignificant protocol violations detected with respect to target volume and OAR contours
Toita, 2011 [38]Japan/JGOGCervicalRetrospective case reviewJGOG1066Expert review of cases treated on trialExpert reviewNone reportedRadiotherapy protocol compliance was favorable except for the radiation boost indications
Beck, 2022 [39]Europe/EORTCProstateRetrospective case reviewEORTC SAKK 09/10 Expert review of cases treated on trialExpert reviewNone reportedSignificant protocol violations detected with respect to target volume and OAR contours
Abrunhosa-Branquinho, 2018 [40]US/Europe/
RTOG/EORTC		CNSRetrospective case review RTOG 0834/EORTC 26053–22054/NCIC CTG CEC.1/
CATNON intergroup trial		Expert review of cases treated on trialExpert reviewNone reportedSignificant protocol violations detected, particularly for PTV delineation
Zhong, 2019 [41]US/NRG/RTOGHNSCCRetrospective case reviewNRG/RTOG 0522Expert review of cases treated on trial, correlated with clinical outcomesExpert review None reportedLocal control was significantly different between the per-protocol and variation-acceptable patients in target and OAR contouring
Sanuki, 2009 [42]Japan/JCOGLungRetrospective case reviewJCOG 0202 Phase IIIExpert review of cases treated on trialExpert reviewNone reportedSubstantial contour violations reported
Schiff, 2023 [43]US/ACNSPeds/CNSRetrospective case reviewACNS 0331 Phase IIIExpert review of cases treated on trial, correlated with clinical outcomes. Expert reviewNone reportedSubstantial contour violations detected, but were not correlated with clinical outcomes
Fokas, 2016 [26]Europe/
UK		PancreasRetrospective case reviewSCALOP TrialExpert review of cases treated on trial Expert review with use of quantitative metricsJaccard Conformity Index (JCI) and Geographical Miss Index (GMI) used to compare accuracy of investigators’ GTV and PTV to trials team-defined gold standardSubstantial contour violations detected
Gwynne, 2012 [18]UKEsophagusRetrospective case reviewSCOPE 1 trialExpert review of cases treated on trial Expert review with use of quantitative metricsJaccard Conformity Index (JCI), Dice coefficient, van’t Riet index, Geographical Miss Index (GMI), and discordance index were used to compare GTV patient contours to expert contours JCI was the most useful single metric to detect inter-observer variation in contours
Khaw, 2022 [44]TROGEndometrialRetrospective case reviewPORTEC-3 (TROG 08.04)Expert review of cases treated on trialExpert reviewNone reported Authors found low rates of both minor and major deviations in contouring and dosimetry
Olch, 2023 [49]Children’s Oncology GroupPeds/CNSComputer program used to retrospectively review trial casesACNS 1123Commercially available plan quality scoring algorithm was used to create a scoring system using the protocol dosimetric requirements Computer programPoints system (70 for PTV, 30 for OAR)Objective scoring system used to evaluate plan quality
Thor, 2021 [48]US/NRG/RTOGLungAI program used to generate OAR contours RTOG 0617Comparison of Artificial Intelligence (AI)-generated and MD-generated heartsAI and Deep learning modelsComputer generated deep learning model used to assess OAR heart contours Trial heart doses were found to be significantly higher than previously reported, which may have led to an even more rapid mortality accumulation
Abbreviations: GHG, Global Quality Assurance of Radiation Therapy Clinical Trials Harmonization Group; UK, United Kingdom; OAR, organs at risk; QA, quality assurance; RTQA, radiation therapy quality assurance; AAPM, American Association of Physicists in Medicine; ASTRO, American Society for Radiation Oncology; NIHR CRN, National Institute of Health Research Clinical Research Network; RTOG, Radiation Therapy Oncology Group; GFRU, Groupe Francophone de Radiothérapie Urologique; EORTC, European Organization for Research and Treatment of Cancer; RT, radiation therapy; CI, confidence interval; JGOG, Japanese Gynecological Oncology Group; CTV, clinical treatment volume; PTV, planning treatment volume; JCOG, Japan Clinical Oncology Group; GTV, gross tumor volume; IQR, inter-quartile rank.
A summary of the contour evaluation methods utilized in each paper is provided in Table 3 and Figure 2.

3.2. Thematic Synthesis of Findings

Beyond the individual study characteristics delineated above, our systematic review revealed four major themes emerging consistently across the 37 included studies, regardless of disease site, cooperative group, or geographic region: (1) delineation as the predominant source of protocol deviations; (2) predominance of the qualitative “gold standard comparison” paradigm; (3) a persistent gap between identification and action; and (4) asymmetric attention to target volumes vs. organs at risk.
Across multiple studies spanning different disease sites and trial designs, contour delineation was consistently identified as the primary contributor to protocol deviations, typically accounting for 50–70% of all quality assurance failures. Christiaens et al. reported that in the EORTC 1219-DAHANCA 29 trial for head and neck cancer, delineation—not planning—remained the primary reason for protocol deviations [19]. Similarly, Stelmes et al. found that OAR contours specifically caused protocol failure in 53% of cases in the EORTC 1420 trial [14]. This pattern persisted across both prospective benchmark assessments and retrospective individual case reviews, across both target volumes and organs at risk, and across both simple and complex anatomical sites. The consistency of this finding despite methodological differences suggests a fundamental, cross-cutting challenge in radiotherapy quality assurance that transcends individual trial designs or disease-specific factors.
We found that 32 of 37 studies (86%) employed some variant of comparison to “gold standard” expert contours as their primary evaluation method. However, in the vast majority of these (28/32, 88%), this comparison was qualitative in nature—relying on expert reviewers to visually assess submitted contours and determine acceptability based on clinical judgment. Only four studies performed quantitative geometric analysis of the comparison to gold standard contours. This reveals that the field remains largely entrenched in a subjective, qualitative paradigm for contour assessment.
A comparative analysis of the ten studies using quantitative metrics revealed the heterogeneity in contour quality assessment (Table 4). Nine of the ten studies used quantitative metrics solely for research analysis rather than pass/fail determinations. Nine of the ten did not define evidence-based thresholds for acceptability, leaving interpretation subjective. DSC was the most common metric (five studies), but reported values and interpretations varied widely. Only Olch et al. developed a comprehensive scoring system with defined point allocations. There was no metric standardization across disease sites or cooperative groups.
A striking pattern emerged of studies repeatedly identifying contour variability as problematic and calling for improvements, yet subsequent trials continued to encounter similar challenges without evidence of systematic amelioration (Table 5). There was a gradual increase in quantitative metric use from 12% (2009–2013) to 38% (2022–2024). The first comprehensive scoring system appeared only in the 2023 publication by Olch et al. [49]. Despite a trend toward quantification, qualitative approaches remain predominant. Thus, the field appears to be in a transition phase but lacks coordinated standardization of effort.
Our analysis revealed an imbalance in quality assurance focus, with target volume evaluation receiving substantially more attention than organ-at-risk assessment (Table 6). Among the 37 studies, 35 (95%) included target volume evaluation while 24 (65%) included OAR assessment. More strikingly, quantitative geometric metrics, when used at all, were almost exclusively applied to target volumes rather than OARs.

4. Discussion

To our knowledge, this is the first systematic review of the literature assessing the existence of standardized quantitative metrics to evaluate contours as part of the cooperative group clinical trial RTQA process. While there are other systematic reviews evaluating QA programs for RT delivery [50,51], none specifically address what quantitative metrics are being used to evaluate RT contours. While the need for improved contour quality assurance has been repeatedly articulated in the literature, our systematic review reveals that quantitative standardization has not materialized despite decades of recognition. Analysis of patterns across our 37 included studies suggests multiple interconnected barriers operating at structural, methodological, and organizational levels.
Several of the studies evaluated in this systematic review identified the variability encountered among contours of patients treated on clinical trials, concluding that this variability is concerning and improvements are needed in RTQA moving forward [14,25,27,49]. While the need for improvements in contouring has been repeatedly shown and each study individually uses metrics to assess the quality of contours being reported, there is a surprising lack of literature describing how to quantitatively assess contours. This failure to provide a uniform quantitative methodology to assess RT contours on clinical trials is due to several barriers.
Unlike dosimetric constraints, which must be documented for protocol approval, contour evaluation methods remain at the discretion of individual trials or quality assurance centers. This lack of regulatory impetus removes a primary driver for standardization. Implementing quantitative contour analysis requires resources which may not be universally available. Further, there is an absence of validated, evidence-based thresholds for geometric indices. Our review found that even when studies employed quantitative metrics such as the Dice similarity coefficient, none defined what constitutes an “acceptable” value based on correlation with clinical outcomes or dosimetric impact. Lack of consensus regarding which metrics are most clinically meaningful further impedes progress. Disease-site heterogeneity compounds this challenge. Finally, the relationship between geometric contour accuracy and clinical outcomes remains incompletely characterized. Quality assurance programs have functioned using qualitative expert review for decades. Changing these processes requires demonstrating not only that quantitative approaches are superior, but that the benefit justifies the resource investment. Without compelling evidence of outcome improvement or efficiency gains, organizational momentum favors continuation of familiar methods.
Beyond the absence of validated thresholds, the metrics themselves carry well-documented limitations that influence their suitability across structures and disease sites, and these limitations are infrequently acknowledged in the trials we reviewed [52]. The Dice similarity coefficient (DSC), the most commonly reported metric in our review, is mathematically biased by structure size: for a fixed absolute boundary error, DSC decreases as structure volume decreases, making it systematically less favorable for small organs at risk (e.g., optic chiasm, cochlea, hippocampus) than for bulky targets, even when delineation quality is comparable [53]. DSC further conflates inclusion and exclusion errors and is relatively insensitive to localized boundary deviations on otherwise large structures. The Hausdorff distance (HD), by contrast, captures the worst-case boundary error and is therefore informative for structures in which a single excursion can be clinically meaningful (e.g., brainstem, spinal cord), but it is highly sensitive to outliers, segmentation noise, and slice thickness; the 95th-percentile Hausdorff distance (HD95) and the mean surface distance are commonly recommended as more robust alternatives [52,53]. For tubular and curvilinear structures (e.g., esophagus, spinal cord, optic nerves), distance-based metrics generally outperform volume-overlap metrics, whereas for large, compact volumes (e.g., parotid glands, prostate, liver) overlap metrics tend to be more discriminating. Surface DSC and added path length have been proposed specifically to bridge the disconnect between DSC and the editing effort required to render a contour clinically acceptable [54,55]. The volume ratio and centroid displacement provide complementary information about systematic over- or under-contouring and gross localization errors, respectively, but neither alone characterizes shape agreement. These differential properties imply that no single metric is appropriate across all structures, and that thoughtful, structure-specific metric selection—guided by structure size, geometry, and the clinical relevance of boundary regions—is itself a prerequisite for meaningful standardization across disease sites.
To provide context for the heterogeneity noted above, we summarize approximate ranges of geometric metric values reported across the benchmark and inter-observer variability studies in our review and in the broader contouring literature; these are offered as descriptive observations rather than prescriptive acceptability thresholds. For target volumes, reported DSC values most commonly fall in the 0.65–0.85 range, with higher values (≥0.80) typically observed for well-bounded, high-contrast targets such as prostate and liver lesions, intermediate values (0.70–0.80) for many head-and-neck and lung targets, and lower values (0.50–0.70) for diffuse or microscopic-spread CTVs such as those in pancreatic and esophageal disease [52] (consistent with the median GTV DSC of 0.67 reported by Gwynne et al. in esophageal cancer and the broad DSC range reported by Fokas et al. in pancreatic cancer). For organs at risk, reported DSC values span a wide range driven largely by structure size and tissue contrast, with parotid glands typically in the 0.75–0.85 range, large well-contrasted structures such as lungs and kidneys commonly in the 0.85–0.95 range, and small or low-contrast structures such as the optic chiasm, cochlea, and hippocampus more often in the 0.50–0.75 range [52,56,57]. Reported HD and HD95 values vary similarly by site, frequently approximating 3–10 mm for HD95 in well-circumscribed structures and exceeding 10–20 mm for complex, diffuse, or large CTVs. We emphasize that these ranges reflect a heterogeneous literature with variable reference standards, image modalities, slice thicknesses, and observer expertise, and they are reported here strictly to characterize the current landscape. They should not be interpreted as evidence-based acceptability criteria, but rather as a descriptive snapshot of the field intended to motivate—and ultimately be replaced by—the prospectively generated, disease-site-specific metric distributions described in our Phase 1 proposal below.
The study by Olch et al. [49] represents the only comprehensive, objective scoring system identified in our review, yet it has not been adopted by any other trial in the literature despite publication in 2023. This “Olch exception” provides insight into adoption barriers. The system requires: (1) commercial software (PlanIQ) not universally available; (2) creation of disease-site-specific scoring templates—a time-intensive process; (3) consensus on point allocation weighting—requiring multi-institutional agreement; and (4) validation that scores correlate with clinically meaningful outcomes. These prerequisites, while achievable, represent substantial implementation barriers that may explain limited adoption despite demonstrated feasibility.
Our focus on geometric contour accuracy, distinct from dosimetric plan quality, requires explicit justification given that dosimetric consequences ultimately determine clinical impact. Adequate contouring of both target volumes and OARs is critically important to the successful delivery of RT, and delivery of high-quality RT is necessary to validate clinical trial outcomes. Studies have shown that clinical trial outcomes are impacted by non-protocol-compliant RT delivery [4,5]. The contouring component of RT delivery has consistently been shown to contribute to the plurality of protocol deviations encountered in clinical trial QA analyses [25,32,44]. We contend that geometric standardization is not merely one approach to quality assurance, but rather a necessary prerequisite for meaningful dosimetric evaluation.
The study by Thor et al. [48] provides compelling illustration of this principle. Using deep-learning auto-segmentation to create geometrically consistent cardiac contours, they reanalyzed dosimetry from RTOG 0617, a landmark phase III trial comparing standard versus high-dose radiation for locally advanced non-small cell lung cancer. When standardized heart contours replaced the original clinician-drawn contours, reported cardiac doses increased significantly, not because treatment plans changed, but because the measured anatomical structure differed. Given that the dose-escalation arm showed unexpected increased mortality potentially related to cardiac dose [58], this geometric variation may have confounded the trial’s dosimetric analysis and clinical interpretation. Importantly, the original trial met its protocol-specified cardiac dose constraints based on the submitted contours, yet those constraints were measured on geometrically variable structures. This case exemplifies a fundamental problem: without geometric standardization, dosimetric evaluation lacks a stable reference frame. If different reviewers or institutions contour the same structure differently, then “meeting” a dosimetric constraint becomes a function of how the structure was drawn, not how it was treated. This case also illustrates the significant impact of accurate OAR contouring on downstream planning adequacy and ultimately on patient outcomes. Several studies specifically documented that OAR delineation failures constituted a major source of protocol deviations—53% in Stelmes et al. [14] and substantial proportions in other trials [2,44]. Yet, in practice, there is a greater emphasis on target volume accuracy. This finding represents a potential blind spot in quality assurance practices: while target volumes receive intensive scrutiny, OAR evaluation remains more cursory despite contributing significantly to protocol failures and patient outcomes.
Geometric standardization also offers practical advantages: it is computationally straightforward (compared to dose calculation verification), can be performed immediately upon contour submission before plan generation, and provides clear, structure-specific feedback to submitting centers. These attributes make geometric evaluation particularly suitable for real-time prospective quality assurance, potentially allowing correction before treatment initiation.
In summary, we position geometric contour accuracy not as the ultimate quality assurance endpoint, but as the essential foundation upon which all subsequent quality assurance must be built. Standardizing geometric evaluation is the necessary first step toward reliable dosimetric assessment and ultimately toward correlation of treatment quality with clinical outcomes.
It should be noted that our study has several limitations. While we did perform a systematic review of the literature, because there was such inconsistency in contour evaluation methods among studies, a more formal analysis such as a meta-analysis or statistical synthesis of our results could not be performed. As such, we were not able to explicitly define a comprehensive set of quantitative metrics used across studies. We also acknowledge that not all quantitative metrics are completely objective, as they may still involve some subjective elements in their application or interpretation. However, we attempted, in so far as we were able, to identify what would be considered uniform quantitative metrics for contour scoring in the literature.

Future Directions

Based on our systematic findings, we propose a phased implementation framework with specific, actionable steps. First (Phase 1), we recommend that baseline quantitative data are established without disrupting current workflows and using existing software. All new cooperative group trials should mandate reporting of geometric indices (DSC, HD95, volume ratio) for submitted contours compared to protocol reference contours. These should include both target volume and OAR contours. Data collection should be implemented as quality assurance metric reporting, not pass/fail criteria initially. We recommend the utilization of existing software (available in MIM, Velocity, Eclipse, RayStation, open-source tools). The end result of Phase I would be generation of a database of geometric metric distributions across trials.
The second step (Phase 2) would establish evidence-based acceptability criteria. This can be accomplished by creation of disease-site specific working groups to analyze Phase 1 data to determine metric distributions and to correlate geometric metrics with: (a) dosimetric impact, (b) expert reviewer classifications, and (c) clinical outcomes where available. The working groups would then establish preliminary acceptability thresholds informed by: 10th–90th percentile ranges from compliant cases, dosimetric sensitivity analysis (geometric deviation required to cause clinically meaningful dose change), and inter-observer variability benchmarks. We expect the outcomes of Phase 2 to be published consensus thresholds for priority disease sites.
As the third step (Phase 3), we recommend formal implementation of a scoring system into clinical trial RTQA. This would involve adapting an Olch-style scoring framework for each disease site using Phase 2 thresholds, weighting structures by clinical importance (informed by dose–outcome literature), accounting for structure-specific difficulty (e.g., sharp vs. gradual boundaries, motion effects), and implementing automated calculation with a manual expert override option for justified deviations. Both target volumes and OARs should be evaluated and scored. The end result of Phase 3 would be a validated scoring system implemented routinely for new clinical trials.
The next step (Phase 4) would be to refine thresholds based on clinical correlation. This would involve prospective correlation of quantitative contour scores with: overall protocol adherence rates, dosimetric quality metrics, and clinical outcomes (local control, toxicity) in completed trials. The goal would be to iteratively refine thresholds based on outcome data and identify which geometric metrics are most predictive of clinically meaningful deviations. The end result of Phase 4 would be evidence-based, outcome-validated evaluation standards.
Finally, the final phase (Phase 5) would be to achieve global harmonization and continuous improvement. This would involve cross-cooperative group consensus on minimum quantitative QA standards, integration into institutional credentialing requirements (similar to IROC phantom credentialing), and real-time automated feedback systems providing immediate quantitative assessment at submission. The end result of Phase 5 would be globally harmonized, evidence-based contour quality assurance practices.
Once quality assurance practices are standardized across clinical trials and the optimal methods of quantitative standardization are developed and validated as per the processes above, these standards can be implemented into clinical practice. Existing software, in conjunction with developing artificial intelligence platforms, can be used to generate contours and evaluate their accuracy as part of RTQA in local RT clinics. Ideally, this would be done following contour creation, but prior to plan generation, as these downstream steps impact the accuracy of subsequent planning steps [58].

5. Conclusions

In this systematic review, a plethora of evidence was found highlighting variability in contours across clinical trials in multiple disease sites conducted by cooperative groups globally. This variability was assessed and identified through subjective metrics used to score contours on clinical trials. We found a lack of evidence describing quantitative metrics used to score contours, indicating a major gap in radiation therapy quality assurance practices conducted by cooperative groups. This concerning lack of standardized quantitative metrics used to evaluate RT target and OAR contours can negatively impact clinical trial results. Effort is needed to create guidelines for objectively evaluating RT contours to be used across clinical trials to improve the quality and validity of RTQA.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/radiation6020020/s1. File S1: PRISMA checklist. Ref. [59] has been cited in Supplementary Materials.

Author Contributions

Conceptualization, L.T.T., H.G., Y.X.; methodology, L.T.T., Y.X.; formal analysis, L.T.T., Y.X.; investigation, L.T.T., Y.X.; data curation, J.L.; writing—original draft preparation, L.T.T., J.J.K.; writing—review and editing, all authors; visualization, L.T.T., Y.X.; supervision, Y.X. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

All of this data is publicly available in the research databases cites above.

Acknowledgments

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process: During the revision of this work the author(s) used Claude to effectively address reviewer comments. After using this tool/service, the author(s) reviewed and edited the content as needed. The authors have reviewed and edited the output and take full responsibility for the content of this publication.

Conflicts of Interest

The authors declare there are no conflicts of interest.

Abbreviations

The following abbreviations are used in this manuscript:
RTQARadiation therapy quality assurance
RTRadiation therapy

References

  1. Delaney, G.; Jacob, S.; Featherstone, C.; Barton, M. The role of radiotherapy in cancer treatment. Cancer 2005, 104, 1129–1137. [Google Scholar] [CrossRef] [Scilit]
  2. Tchelebi, L.; Winter, K.; Abrams, R.; Safran, H.; Regine, W.; McNulty, S.; Wu, A.; Du, K.; Seaward, S.; Bian, S.; et al. Analysis of Radiation Therapy Quality Assurance in NRG Oncology RTOG 0848. Int. J. Radiat. Oncol. 2021, 111, S40. [Google Scholar] [CrossRef] [Scilit]
  3. Cox, S.; Cleves, A.; Clementel, E.; Miles, E.; Staffurth, J.; Gwynne, S. Impact of deviations in target volume delineation—Time for a new RTQA approach? Radiother. Oncol. 2019, 137, 1–8. [Google Scholar] [CrossRef] [Scilit]
  4. Ohri, N.; Shen, X.; Dicker, A.P.; Doyle, L.A.; Harrison, A.S.; Showalter, T.N. Radiotherapy Protocol Deviations and Clinical Outcomes: A Meta-analysis of Cooperative Group Clinical Trials. JNCI J. Natl. Cancer Inst. 2013, 105, 387–393. [Google Scholar] [CrossRef] [Scilit]
  5. Abrams, R.A.; Winter, K.A.; Regine, W.F.; Safran, H.; Hoffman, J.P.; Lustig, R.; Konski, A.A.; Benson, A.B.; Macdonald, J.S.; Rich, T.A.; et al. Failure to Adhere to Protocol Specified Radiation Therapy Guidelines Was Associated with Decreased Survival in RTOG 9704—A Phase III Trial of Adjuvant Chemotherapy and Chemoradiotherapy for Patients with Resected Adenocarcinoma of the Pancreas. Int. J. Radiat. Oncol. 2012, 82, 809–816. [Google Scholar] [CrossRef] [Scilit]
  6. Fairchild, A.; Straube, W.; Laurie, F.; Followill, D. Does Quality of Radiation Therapy Predict Outcomes of Multicenter Cooperative Group Trials? A Literature Review. Int. J. Radiat. Oncol. 2013, 87, 246–260. [Google Scholar] [CrossRef] [Scilit]
  7. Albert, J.M.; Das, P. Quality Indicators in Radiation Oncology. Int. J. Radiat. Oncol. 2013, 85, 904–911. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  8. de Balincourt, C.; Lacombe, D.; Coens, C.; Dulk, M.D.; Machiels, J.-P.; Weber, D.; Spahn, M.; Salgado, R.; Kasper, B.; Hau, P.; et al. Multidisciplinary quality assurance and control in oncological trials: Perspectives from European Organisation for Research and Treatment of Cancer (EORTC). Eur. J. Cancer 2017, 86, 91–100. [Google Scholar] [CrossRef] [Scilit]
  9. Bekelman, J.E.; Deye, J.A.; Vikram, B.; Bentzen, S.M.; Bruner, D.; Curran, W.J.; Dignam, J.; Efstathiou, J.A.; FitzGerald, T.; Hurkmans, C.; et al. Redesigning Radiotherapy Quality Assurance: Opportunities to Develop an Efficient, Evidence-Based System to Support Clinical Trials—Report of the National Cancer Institute Work Group on Radiotherapy Quality Assurance. Int. J. Radiat. Oncol. 2012, 83, 782–790. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  10. McDowell, L.J.; Corry, J. A Call to Arms: Radiation Therapy Quality Assurance in the Next Generation of Clinical Trials. Int. J. Radiat. Oncol. 2018, 102, 1590–1591. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  11. Corrigan, K.L.; Kry, S.; Howell, R.M.; Kouzy, R.; Jaoude, J.A.; Patel, R.R.; Jhingran, A.; Taniguchi, C.; Koong, A.C.; McAleer, M.F.; et al. The radiotherapy quality assurance gap among phase III cancer clinical trials. Radiother. Oncol. 2021, 166, 51–57. [Google Scholar] [CrossRef] [Scilit]
  12. Lin, D.; Lapen, K.; Sherer, M.V.; Kantor, J.; Zhang, Z.; Boyce, L.M.; Bosch, W.; Korenstein, D.; Gillespie, E.F. A Systematic Review of Contouring Guidelines in Radiation Oncology: Analysis of Frequency, Methodology, and Delivery of Consensus Recommendations. Int. J. Radiat. Oncol. 2020, 107, 827–835. [Google Scholar] [CrossRef] [Scilit]
  13. Wills, L.; Maggs, R.; Lewis, G.; Jones, G.; Nixon, L.; Staffurth, J.; Crosby, T. Quality assurance of the SCOPE 1 trial in oesophageal radiotherapy. Radiat. Oncol. 2017, 12, 179. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  14. Stelmes, J.-J.; Vu, E.; Grégoire, V.; Simon, C.; Clementel, E.; Kazmierska, J.; Grant, W.; Ozsahin, M.; Tomsej, M.; Vieillevigne, L.; et al. Quality assurance of radiotherapy in the ongoing EORTC 1420 “Best of” trial for early stage oropharyngeal, supraglottic and hypopharyngeal carcinoma: Results of the benchmark case procedure. Radiat. Oncol. 2021, 16, 81. [Google Scholar] [CrossRef] [Scilit]
  15. Okamoto, H.; Murakami, N.; Isohashi, F.; Kasamatsu, T.; Hasumi, Y.; Iijima, K.; Nishioka, S.; Nakamura, S.; Nakamura, M.; Nishio, T.; et al. Dummy-run for standardizing plan quality of intensity-modulated radiotherapy for postoperative uterine cervical cancer: Japan Clinical Oncology Group study (JCOG1402). Radiat. Oncol. 2019, 14, 133. [Google Scholar] [CrossRef] [Scilit]
  16. Musat, E.; Roelofs, E.; Bar-Deroma, R.; Fenton, P.; Gulyban, A.; Collette, L.; Stupp, R.; Weber, D.C.; Davis, J.B.; Aird, E.; et al. Dummy run and conformity indices in the ongoing EORTC low-grade glioma trial 22033-26033: First evaluation of quality of radiotherapy planning. Radiother. Oncol. 2010, 95, 218–224. [Google Scholar] [CrossRef] [Scilit]
  17. Gondi, V.; Cui, Y.; Mehta, M.P.; Manfredi, D.; Xiao, Y.; Galvin, J.M.; Rowley, H.; Tome, W.A. Real-Time Pretreatment Review Limits Unacceptable Deviations on a Cooperative Group Radiation Therapy Technique Trial: Quality Assurance Results of RTOG 0933. Int. J. Radiat. Oncol. 2015, 91, 564–570. [Google Scholar] [CrossRef] [Scilit]
  18. Gwynne, S.; Spezi, E.; Wills, L.; Nixon, L.; Hurt, C.; Joseph, G.; Evans, M.; Griffiths, G.; Crosby, T.; Staffurth, J. Toward Semi-automated Assessment of Target Volume Delineation in Radiotherapy Trials: The SCOPE 1 Pretrial Test Case. Int. J. Radiat. Oncol. 2012, 84, 1037–1042. [Google Scholar] [CrossRef] [Scilit]
  19. Christiaens, M.; Collette, S.; Overgaard, J.; Gregoire, V.; Kazmierska, J.; Castadot, P.; Giralt, J.; Grant, W.; Tomsej, M.; Bar-Deroma, R.; et al. Quality assurance of radiotherapy in the ongoing EORTC 1219-DAHANCA-29 trial for HPV/p16 negative squamous cell carcinoma of the head and neck: Results of the benchmark case procedure. Radiother. Oncol. 2017, 123, 424–430. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  20. Fairchild, A.; Langendijk, J.A.; Nuyts, S.; Scrase, C.; Tomsej, M.; Schuring, D.; Gulyban, A.; Ghosh, S.; Weber, D.C.; Budach, W. Quality assurance for the EORTC 22071–26071 study: Dummy run prospective analysis. Radiat. Oncol. 2014, 9, 248. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  21. Lambrecht, M.; Clementel, E.; Sonke, J.-J.; Nestle, U.; Adebahr, S.; Guckenberger, M.; Andratschke, N.; Weber, D.C.; Verheij, M.; Hurkmans, C.W. Radiotherapy quality assurance of SBRT for patients with centrally located lung tumours within the multicentre phase II EORTC Lungtech trial: Benchmark case results. Radiother. Oncol. 2019, 132, 63–69. [Google Scholar] [CrossRef] [Scilit]
  22. Fokas, E.; Clifford, C.; Spezi, E.; Joseph, G.; Branagan, J.; Hurt, C.; Nixon, L.; Abrams, R.; Staffurth, J.; Mukherjee, S. Comparison of investigator-delineated gross tumor volumes and quality assurance in pancreatic cancer: Analysis of the pretrial benchmark case for the SCALOP trial. Radiother. Oncol. 2015, 117, 432–437. [Google Scholar] [CrossRef] [Scilit]
  23. Sassowsky, M.; Gut, P.; Hölscher, T.; Hildebrandt, G.; Müller, A.-C.; Najafi, Y.; Kohler, G.; Kranzbühler, H.; Guckenberger, M.; Zwahlen, D.R.; et al. Use of EORTC Target Definition Guidelines for Dose-Intensified Salvage Radiation Therapy for Recurrent Prostate Cancer: Results of the Quality Assurance Program of the Randomized Trial SAKK 09/10. Int. J. Radiat. Oncol. 2013, 87, 534–541. [Google Scholar] [CrossRef] [Scilit]
  24. Fenton, P.A.; Hurkmans, C.; Gulyban, A.; van der Leer, J.; Matzinger, O.; Poortmans, P.; Collette, L.; Bolla, M. Quality assurance of the EORTC 22043-30041 trial in post-operative radiotherapy in prostate cancer: Results of the Dummy Run procedure. Radiother. Oncol. 2013, 107, 346–351. [Google Scholar] [CrossRef] [Scilit]
  25. Cloak, K.; Jameson, M.G.; Paneghel, A.; Wiltshire, K.; Kneebone, A.; Pearse, M.; Sidhom, M.; Tang, C.; Fraser-Browne, C.; Holloway, L.C.; et al. Contour variation is a primary source of error when delivering post prostatectomy radiotherapy: Results of the Trans-Tasman Radiation Oncology Group 08.03 Radiotherapy Adjuvant Versus Early Salvage (RAVES) benchmarking exercise. J. Med. Imaging Radiat. Oncol. 2019, 63, 390–398. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  26. Fokas, E.; Spezi, E.; Patel, N.; Hurt, C.; Nixon, L.; Chu, K.; Joseph, G.; Staffurth, J.; Abrams, R.; Mukherjee, S. P-222 Analysis of tumour contours and radiotherapy planning of “on-trial” patients undergoing chemoradiotherapy (CRT) in SCALOP trial: Does pre-trial Radiotherapy Quality Assurance (RTQA) improve the quality of “on-trial” radiotherapy? Ann. Oncol. 2016, 27, ii64–ii65. [Google Scholar] [CrossRef] [Scilit]
  27. Matzinger, O.; Poortmans, P.; Giraud, J.-Y.; Maingon, P.; Budiharto, T.; Bergh, A.C.M.V.D.; Davis, J.B.; Musat, E.; Ataman, F.; Huyskens, D.P.; et al. Quality assurance in the 22991 EORTC ROG trial in localized prostate cancer: Dummy run and individual case review. Radiother. Oncol. 2009, 90, 285–290. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  28. Kearvell, R.; Haworth, A.; Ebert, M.A.; Murray, J.; Hooton, B.; Richardson, S.; Joseph, D.J.; Lamb, D.; Spry, N.A.; Duchesne, G.; et al. Quality improvements in prostate radiotherapy: Outcomes and impact of comprehensive quality assurance during the TROG 03.04 ‘RADAR’ trial. J. Med. Imaging Radiat. Oncol. 2013, 57, 247–257. [Google Scholar] [CrossRef] [Scilit]
  29. Corry, J.; Moore, A.; Kenny, L.; Wratten, C.; Fua, T.; Lin, C.; Porceddu, S.; Liu, C.; Ruemelin, M.; Sharkey, A.; et al. Radiotherapy quality assurance in the TROG 12.01 randomised trial and its impact on loco-regional failure. Front. Oncol. 2024, 13, 1333098. [Google Scholar] [CrossRef] [Scilit]
  30. Okamoto, H.; Murakami, N.; Isohashi, F.; Kasamatsu, T.; Hasumi, Y.; Kobayashi, H.; Ishikawa, M.; Nakamura, M.; Nishio, T.; Igaki, H.; et al. Plan quality association between dummy run and individual case review in a prospective multi-institutional clinical trial of postoperative cervical cancer patients treated with intensity-modulated radiotherapy: Japan clinical Oncology Group study (JCOG1402). Radiother. Oncol. 2023, 183, 109630. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  31. Hardcastle, N.; Cook, O.; Ray, X.; Moore, A.; Moore, K.L.; Pryor, D.; Rossi, A.; Foroudi, F.; Kron, T.; Siva, S. Personalising treatment plan quality review with knowledge-based planning in the TROG 15.03 trial for stereotactic ablative body radiotherapy in primary kidney cancer. Radiat. Oncol. 2021, 16, 142. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  32. Tchelebi, L.T.; Winter, K.A.; Abrams, R.A.; Safran, H.P.; Regine, W.F.; McNulty, S.; Wu, A.; Du, K.L.; Seaward, S.A.; Bian, S.X.; et al. Analysis of Radiation Therapy Quality Assurance in NRG Oncology RTOG 0848. Int. J. Radiat. Oncol. 2023, 118, 107–114. [Google Scholar] [CrossRef] [Scilit]
  33. Lukovic, J.; Moore, A.J.; Lee, M.T.; Willis, D.; Ahmed, S.; Akra, M.; Hortobagyi, E.; Kron, T.; Joon, D.L.; Liu, A.; et al. The Feasibility of Quality Assurance in the TOPGEAR International Phase 3 Clinical Trial of Neoadjuvant Chemoradiation Therapy for Gastric Cancer (an Intergroup Trial of the AGITG/TROG/NHMRC CTC/EORTC/CCTG). Int. J. Radiat. Oncol. 2023, 117, 1096–1106. [Google Scholar] [CrossRef] [Scilit]
  34. Trada, Y.; Kneebone, A.; Paneghel, A.; Pearse, M.; Sidhom, M.; Tang, C.; Wiltshire, K.; Haworth, A.; Fraser-Browne, C.; Martin, J. Optimizing Radiation Therapy Quality Assurance in Clinical Trials: A TROG 08.03 RAVES Substudy. Int. J. Radiat. Oncol. 2015, 93, 1045–1051. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  35. Mir, R.; Groom, N.; Mistry, H.B.; Wilson, E.; Faivre-Finn, C. Association between radiotherapy protocol variations and outcome in the CONVERT trial. Clin. Transl. Radiat. Oncol. 2022, 39, 100560. [Google Scholar] [CrossRef] [Scilit]
  36. Haas, R.; Stelmes, J.; Zaffaroni, F.; Sauvé, N.; Clementel, E.; Bar-Deroma, R.; Le Péchoux, C.; Litière, S.; Marreaud, S.; Alyamani, N.; et al. Critical impact of radiotherapy protocol compliance and quality in the treatment of retroperitoneal sarcomas: Results from the EORTC 62092-22092 STRASS trial. Cancer 2022, 128, 2796–2805. [Google Scholar] [CrossRef] [Scilit]
  37. Fairchild, A.; Weber, D.C.; Bar-Deroma, R.; Gulyban, A.; Fenton, P.A.; Stupp, R.; Baumert, B.G. Quality assurance in the EORTC 22033–26033/CE5 phase III randomized trial for low grade glioma: The digital individual case review. Radiother. Oncol. 2012, 103, 287–292. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  38. Toita, T.; Kato, S.; Ishikura, S.; Tsujino, K.; Kodaira, T.; Uno, T.; Hatano, K.; Sakurai, H.; Niibe, Y.; Kazumoto, T.; et al. Radiotherapy quality assurance of the Japanese Gynecologic Oncology Group study (JGOG1066): A cooperative phase II study of concurrent chemoradiotherapy for uterine cervical cancer. Int. J. Clin. Oncol. 2011, 16, 379–386. [Google Scholar] [CrossRef] [Scilit]
  39. Beck, M.; Sassowsky, M.; Schär, S.; Mathier, E.; Halter, M.; Zwahlen, D.R.; Hölscher, T.; Arnold, W.; Polat, B.; Hildebrandt, G.; et al. Adherence to Contouring and Treatment Planning Requirements Within a Multicentric Trial: Results of the Quality Assurance of the SAKK 09/10 trial. Int. J. Radiat. Oncol. 2022, 113, 80–91. [Google Scholar] [CrossRef] [Scilit]
  40. Abrunhosa-Branquinho, A.N.; Bar-Deroma, R.; Collette, S.; Clementel, E.; Liu, Y.; Hurkmans, C.W.; Feuvret, L.; Van Beek, K.; Bent, M.v.D.; Baumert, B.G.; et al. Radiotherapy quality assurance for the RTOG 0834/EORTC 26053-22054/NCIC CTG CEC.1/CATNON intergroup trial “concurrent and adjuvant temozolomide chemotherapy in newly diagnosed non-1p/19q deleted anaplastic glioma”: Individual case review analysis. Radiother. Oncol. 2018, 127, 292–298. [Google Scholar] [CrossRef] [Scilit]
  41. Zhong, H.; Men, K.; Wang, J.; van Soest, J.; Rosenthal, D.; Dekker, A.; Zhang, Z.; Xiao, Y. The Impact of Clinical Trial Quality Assurance on Outcome in Head and Neck Radiotherapy Treatment. In Proceedings of the 59th Annual Scientific Meeting of the American-Society-for-Radiation-Oncology (ASTRO), San Diego, CA, USA, 24–27 September 2017; p. 792. [Google Scholar]
  42. Sanuki-Fujimoto, N.; Ishikura, S.; Hayakawa, K.; Kubota, K.; Nishiwaki, Y.; Tamura, T. Radiotherapy quality assurance review in a multi-center randomized trial of limited-disease small cell lung cancer: The Japan Clinical Oncology Group (JCOG) trial 0202. Radiat. Oncol. 2009, 4, 16. [Google Scholar] [CrossRef] [Scilit]
  43. Schiff, J.P.; Lee, Y.; Wang, Y.; Perkins, S.M.; Kessel, S.K.; Fitzgerald, T.J.; Larrier, N.A.; Michalski, J.M. An Analysis of Major Target Deviations in Craniospinal Irradiation Treatment Plans for Patients with Intermediate-Risk Medulloblastoma Within a Phase 3 Clinical Trial (Children’s Oncology Group Study ACNS0331). Adv. Radiat. Oncol. 2022, 8, 101083. [Google Scholar] [CrossRef] [Scilit]
  44. Khaw, P.; Do, V.; Lim, K.; Cunninghame, J.; Dixon, J.; Vassie, J.; Bailey, M.; Johnson, C.; Kahl, K.; Gordon, C.; et al. Radiotherapy Quality Assurance in the PORTEC-3 (TROG 08.04) Trial. Clin. Oncol. 2022, 34, 198–204. [Google Scholar] [CrossRef] [Scilit]
  45. Yang, H.; Mir, R.; Díez, P.; Tsang, Y.; Conibear, J.; Simões, R.; Cox, S.; Webster, A.; Nabi, Z.; Eaton, D.; et al. Provision of Organ at Risk Contouring Guidance in UK Radiotherapy Clinical Trials. Clin. Oncol. 2020, 32, e60–e66. [Google Scholar] [CrossRef] [Scilit]
  46. Mir, R.; Kelly, S.M.; Xiao, Y.; Moore, A.; Clark, C.H.; Clementel, E.; Corning, C.; Ebert, M.; Hoskin, P.; Hurkmans, C.W.; et al. Organ at risk delineation for radiation therapy clinical trials: Global Harmonization Group consensus guidelines. Radiother. Oncol. 2020, 150, 30–39. [Google Scholar] [CrossRef] [Scilit]
  47. Harmon, G.; Chan, D.; Lee, B.; Miller, C.; Gorbonos, A.; Gupta, G.; Quek, M.; Woods, M.; Savir-Baruch, B.; Harkenrider, M.M.; et al. Validating Modern NRG Oncology Pelvic Nodal and Groupe Francophone de Radiothérapie Urologique Prostate Bed Contouring Guidelines for Post-Prostatectomy Salvage Radiation: A Secondary Analysis of the LOCATE Trial. Int. J. Radiat. Oncol. 2021, 111, 1195–1203. [Google Scholar] [CrossRef] [Scilit]
  48. Thor, M.; Apte, A.; Haq, R.; Iyer, A.; LoCastro, E.; Deasy, J.O. Using Auto-Segmentation to Reduce Contouring and Dose Inconsistency in Clinical Trials: The Simulated Impact on RTOG 0617. Int. J. Radiat. Oncol. 2020, 109, 1619–1626. [Google Scholar] [CrossRef] [Scilit]
  49. Olch, A.J.; Gopalakrishnan, M.; Murphy, E.S.; MacDonald, S.M.; Hua, C.-H. Toward Systematic Assessment and Improvement of Radiation Therapy Plan Quality of Cooperative Group Trial Submissions: A Report From the Children’s Oncology Group. Pract. Radiat. Oncol. 2023, 13, e374–e382. [Google Scholar] [CrossRef] [Scilit]
  50. Brooks, C.; Miles, E.; Hoskin, P.J. Radiotherapy trial quality assurance processes: A systematic review. Lancet Oncol. 2024, 25, e104–e113. [Google Scholar] [CrossRef] [Scilit]
  51. Vinod, S.K.; Jameson, M.G.; Min, M.; Holloway, L.C. Uncertainties in volume delineation in radiation oncology: A systematic review and recommendations for future studies. Radiother. Oncol. 2016, 121, 169–179. [Google Scholar] [CrossRef] [Scilit]
  52. Sherer, M.V.; Lin, D.; Elguindi, S.; Duke, S.; Tan, L.-T.; Cacicedo, J.; Dahele, M.; Gillespie, E.F. Metrics to evaluate the performance of auto-segmentation for radiation treatment planning: A critical review. Radiother. Oncol. 2021, 160, 185–191. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  53. Taha, A.A.; Hanbury, A. Metrics for evaluating 3D medical image segmentation: Analysis, selection, and tool. BMC Med. Imaging 2015, 15, 29. [Google Scholar] [CrossRef] [Scilit]
  54. Nikolov, S.; Blackwell, S.; Zverovitch, A.; Mendes, R.; Livne, M.; De Fauw, J.; Patel, Y.; Meyer, C.; Askham, H.; Romera-Paredes, B.; et al. Clinically Applicable Segmentation of Head and Neck Anatomy for Radiotherapy: Deep Learning Algorithm Development and Validation Study. J. Med. Internet Res. 2021, 23, e26151. [Google Scholar] [CrossRef] [Scilit]
  55. Vaassen, F.; Hazelaar, C.; Vaniqui, A.; Gooding, M.; van der Heyden, B.; Canters, R.; van Elmpt, W. Evaluation of measures for assessing time-saving of automatic organ-at-risk segmentation in radiotherapy. Phys. Imaging Radiat. Oncol. 2020, 13, 1–6. [Google Scholar] [CrossRef] [Scilit]
  56. Brouwer, C.L.; Steenbakkers, R.J.H.M.; van den Heuvel, E.; Duppen, J.C.; Navran, A.; Bijl, H.P.; Chouvalova, O.; Burlage, F.R.; Meertens, H.; Langendijk, J.A.; et al. 3D Variation in delineation of head and neck organs at risk. Radiat. Oncol. 2012, 7, 32. [Google Scholar] [CrossRef] [Scilit]
  57. Brouwer, C.L.; Steenbakkers, R.J.; Bourhis, J.; Budach, W.; Grau, C.; Grégoire, V.; van Herk, M.; Lee, A.; Maingon, P.; Nutting, C.; et al. CT-based delineation of organs at risk in the head and neck region: DAHANCA, EORTC, GORTEC, HKNPCSG, NCIC CTG, NCRI, NRG Oncology and TROG consensus guidelines. Radiother. Oncol. 2015, 117, 83–90. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  58. Cox, B.W.; Teckie, S.; Kapur, A.; Chou, H.; Potters, L. Prospective Peer Review in Radiation Therapy Treatment Planning: Long-Term Results From a Longitudinal Study. Pr. Radiat. Oncol. 2020, 10, e199–e206. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  59. Page, M.J.; McKenzie, J.E.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E.; et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ 2021, 372, n71. [Google Scholar] [CrossRef] [Scilit]
Radiation 06 00020 g001
Figure 1. Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA).
Figure 1. Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA).
Radiation 06 00020 g001
Radiation 06 00020 g002
Figure 2. Breakdown of RTQA trials and use of quantitative metrics for contour assessment.
Figure 2. Breakdown of RTQA trials and use of quantitative metrics for contour assessment.
Radiation 06 00020 g002
Table 1. Prospective radiation therapy quality assurance data.
Table 1. Prospective radiation therapy quality assurance data.
Author, YearRegionDisease Site Study TypeRelevant Clinical TrialQA ApproachContour Evaluation MethodQuantitative Metrics Used Key Findings
Wills, 2017 [13]UKEsophagusBenchmark Case/
Dummy Run		SCOPE 1 trialComparison to protocol reference standardExpert reviewNone reportedSubstantial variability in contours despite existence of a comprehensive protocol
Stelmes, 2021 [14]Europe/
EORTC		Head and NeckBenchmark Case/
Dummy Run		EORTC 1420 Comparison to gold standard expert delineationExpert reviewNone reportedOAR volume delineations were the cause for protocol failure in 53% of cases
Okamoto, 2019 [15]Japan/
JCOG		CervixBenchmark Case/
Dummy Run		JCOG1402Comparison to protocol reference standardExpert reviewThe Dice similarity coefficient (DSC) and the Hausdorff distance (HD) were calculated to assess inter-observer variability for delineation in the CTV and scored categorically (per protocol/acceptable variation/unacceptable)Substantial variability in contours despite existence of a comprehensive protocol
Musat, 2010 [16]Europe/
EORTC		CNSBenchmark Case/
Dummy Run		EORTC 22033–26033Comparison to protocol reference standard Expert reviewNone reportedOnly two (5%) centers had to repeat target volume delineations
Gondi, 2015 [17]US/NRG/
RTOG 		CNSBenchmark Case/
Dummy Run		RTOG 0933 Comparison to gold standard expert delineationExpert review with use of quantitative metricsHausdorff distance was calculated for the submitted hippocampal contour and scored categorically (per protocol/acceptable variation/unacceptable) Real-time contour review limits unacceptable deviations
Christiaens, 2017 [19]Europe/
EORTC		HNSCCBenchmark Case/
Dummy Run		EORTC 1219-DAHANCA-29Comparison to gold standard expert delineationExpert review with use of quantitative metricsThe Sørensen–Dice Similarity Index and the Hausdorff distance were calculated retrospectively and used to evaluate the overlap between the participating center’s delineations of CTVs and the expert contoursContour delineation remains primary reason for protocol deviations
Fairchild, 2014 [20]Europe/
EORTC		HNSCCBenchmark Case/
Dummy Run		EORTC 22071–26071 Comparison to gold standard expert delineationExpert reviewNone reportedSubstantial variability in contours despite existence of a comprehensive protocol
Lambrecht, 2019 [21]Europe/
EORTC		LungBenchmark Case/
Dummy Run		EORTC Lungtech TrialComparison to protocol reference standardExpert reviewNone reportedContour delineation remains primary reason for protocol deviations
Fokas, 2015 [22]Europe/
UK		PancreasBenchmark Case/
Dummy Run		SCALOP TrialComparison to gold standard expert delineationExpert review with use of quantitative metricsWhole-volume Jaccard Conformity Index (JCI) and Geographic Miss Index (GMI) were generated for both GTV and PTV comparison. JCI is defined as the volume of the gold standard contour encompassed by the investigator contour as a fraction of their combined volume. There was considerable variation in GTV delineation
Sassowsky, 2013 [23]EuropeProstateBenchmark Case/
Dummy Run		SAKK 09/10Comparison to protocol reference standardExpert review with use of quantitative metricsInter-observer agreement on target volume and OAR delineation was assessed by the apparent volume overlap, which was corrected for agreement by chance using Fleiss’ κSubstantial variability in contours on first attempt despite existence of a comprehensive protocol
Fenton, 2013 [24]Europe/
EORTC		ProstateBenchmark Case/
Dummy Run		EORTC 22043–30041 Comparison to protocol reference standardExpert reviewNone reportedSubstantial variability in contours despite existence of a comprehensive protocol
Cloak, 2019 [25]TROGProstateBenchmark Case/
Dummy Run		TROG 08.03 RAVES Comparison to protocol reference standardExpert review with use of quantitative metricsQuantitative geometric parameters including volume, Hausdorff distance (HD), mean distance to agreement (MDA) and Dice similarity coefficient (DSC) were analyzed for targets and organs at risk. Variability was found most notably in CTV and rectum delineation as a primary source of protocol error.
Matzinger, 2009 [27]Europe/
EORTC		ProstateBenchmark Case/
Dummy Run + Individual case review		EORTC 22991Comparison to protocol reference standardExpert reviewNone reportedVolume delineation varied significantly between investigators; however, no major protocol deviations were observed in individual case reviews
Kearvell, 2013 [28]TROGProstateBenchmark Case/
Dummy Run + Individual case review		TROG 03.04 ‘RADAR’ Comparison to protocol reference standardExpert reviewNone reportedSubstantial variability in contours despite existence of a comprehensive protocol
Corry, 2024 [29]TROGHead and NeckBenchmark Case/
Dummy Run + Individual case review + post-treatment RTQA review		TROG 12.01 Comparison to protocol reference standardExpert reviewNone reportedMajor protocol deviations were found indicating importance of pre-treatment RTQA
Okamoto, 2023 [30]Japan/
JCOG		CervixBenchmark Case/
Dummy Run + post-treatment RTQA review		JCOG1402Comparison to gold standard expert delineationExpert review with use of quantitative metricsNone reportedFrequent deviations were observed in clinical target volume (CTV) delineations
Hardcastle, 2021 [31]TROGKidneyProspective real-time individual case review TROG 15.03 FASTRACK II Comparison to protocol reference standardExpert reviewNone reported (Note: KBP modeling was used for plan and DVH review but not contour review)Real time plan review can improve treatment plan quality
Tchelebi, 2024 [32]US/NRG/RTOGPancreasProspective real-time individual case review RTOG 0848 Phase IIIComparison to protocol reference standardExpert reviewNone reportedSubstantial variability in contours despite existence of a comprehensive protocol
Lukovic, 2023 [33]International: AGITG, TROG, EORTC, CCTGGastricProspective real-time individual case reviewTOPGEAR International Phase IIIComparison to protocol reference standard Expert reviewNone reportedSubstantial variability in CTV contours despite existence of a comprehensive protocol
Trada, 2015 [34]TROGProstateProspective real-time individual case reviewTROG 08.03
RAVES		Comparison to protocol reference standardExpert reviewNone reportedThe majority of protocol deviations were due to contouring and dosimetric violations
Abbreviations: UK, United Kingdom; SCOPE, Study of Chemoradiotherapy in OesoPhageal cancer with Erbitux; CRT, chemoradiation; QA, quality assurance; RTQA, radiation therapy quality assurance; GTV, gross target volume; cc, cubic centimeter; PTV, planning target volume; V20Gy, volume receiving 20 Gray; V40Gy, volume receiving 40 Gray; OAR, organ at risk; EORTC, European Organization for Research and Treatment of Cancer; IOV, inter-observer variability; IMRT, intensity-modulated radiation therapy; JCOG, Japan Clinical Oncology Group; CTV, clinical treatment volume; RTOG, Radiation Therapy Oncology Group; SBRT, stereotactic body radiation therapy; IQR, inter-quartile rank; TROG, Trans-Tasman Radiation Oncology Group; KBP, knowledge-based planning; TOPGEAR, Trial of Preoperative Therapy for Gastric and Esophagogastric Junction Adenocarcinoma; AGITG, Australasian Gastro-Intestinal Trials Group; CCTG, Canadian Clinical Trials Group; RAVES, Radiation Therapy-Adjuvant vs. Early Salvage; OR, Odds Ratio.
Table 3. Summary of contour evaluation methods across all studies (n = 37).
Table 3. Summary of contour evaluation methods across all studies (n = 37).
Contour Evaluation MethodNumber of Studies (%)
Expert review only26 (70)
Expert review with use of quantitative metrics6 (16)
AI-assisted contour review1 (3)
Computer based objective scoring system for contour review1 (3)
Abbreviations: AI, artificial intelligence.
Table 4. Comparative analysis of studies employing quantitative geometric metrics (Number = 10).
Table 4. Comparative analysis of studies employing quantitative geometric metrics (Number = 10).
StudyYearDisease SiteMetric(s) UsedThresholds Defined?ApplicationKey Finding
Gwynne et al. [18]2012EsophagealVolume overlap, DSC, conformity indexNoResearch analysis of benchmarkMedian DSC 0.67 for GTV; high variability
Sassowsky et al. [23]2013ProstateConformity index, volume comparisonNoBenchmark case assessmentSubstantial volume differences vs. reference
Gondi et al. [17]2015BrainVisual comparison with quantitative volumetric analysisNoReal-time review35% unacceptable deviations initially
Fokas et al. [26]2016PancreaticDSC, volume ratiosNoRetrospective analysisDSC range 0.48–0.85; no correlation with protocol adherence
Christiaens et al. [19]2017Head & NeckVolume, DSCNoBenchmark assessmentDelineation primary deviation cause
Okamoto et al. [15]2019CervicalVolumetric comparison, distance metricsNoDummy run standardizationImproved consistency after feedback
Cloak et al. [25]2019ProstateVolume analysis, spatial comparisonNoBenchmark (RAVES trial)Contour variation primary error source
Thor et al. [48]2021LungAuto-segmentation comparisonNoRetrospective reanalysisHeart doses underestimated due to contour inconsistency
Okamoto et al. [30]2023CervicalVolume metricsNoIndividual case review correlation with dummy runAssociation between dummy run and on-trial performance
Olch et al. [49]2023CNSComprehensive scoring systemYesPlan quality assessmentOnly study with objective scoring thresholds
Table 5. Evolution of contour evaluation approaches by time period.
Table 5. Evolution of contour evaluation approaches by time period.
Time PeriodTotal StudiesQualitative Expert Review OnlyQuantitative Metrics UsedObjective Scoring SystemPredominant Approach
2009–201387 (88%)1 (12%)0 (0%)Gold standard comparison, qualitative
2014–2017118 (73%)3 (27%)0 (0%)Expert review with emerging metrics
2018–2021106 (60%)4 (40%)0 (0%)Mixed qualitative/quantitative
2022–202484 (50%)3 (38%)1 (12%)Increasing quantification
Table 6. Evaluation method stratified by quality assurance type.
Table 6. Evaluation method stratified by quality assurance type.
QA TypeNumber of StudiesStudies Using Quantitative MetricsPercentage
Dummy Run/Benchmark Only12650%
Individual Case Review (Prospective)800%
Individual Case Review (Retrospective)12325%
Combined Dummy Run + Individual Review5120%
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Tchelebi, L.T.; Lester, J.; Kim, J.J.; Geng, H.; Xiao, Y. Systematic Review of Quantitative Metrics to Standardize Contours on Radiotherapy Clinical Trials. Radiation 2026, 6, 20. https://doi.org/10.3390/radiation6020020

AMA Style

Tchelebi LT, Lester J, Kim JJ, Geng H, Xiao Y. Systematic Review of Quantitative Metrics to Standardize Contours on Radiotherapy Clinical Trials. Radiation. 2026; 6(2):20. https://doi.org/10.3390/radiation6020020

Chicago/Turabian Style

Tchelebi, Leila T., Janice Lester, Joshua J. Kim, Huaizhi Geng, and Ying Xiao. 2026. "Systematic Review of Quantitative Metrics to Standardize Contours on Radiotherapy Clinical Trials" Radiation 6, no. 2: 20. https://doi.org/10.3390/radiation6020020

APA Style

Tchelebi, L. T., Lester, J., Kim, J. J., Geng, H., & Xiao, Y. (2026). Systematic Review of Quantitative Metrics to Standardize Contours on Radiotherapy Clinical Trials. Radiation, 6(2), 20. https://doi.org/10.3390/radiation6020020

Article Metrics

Back to TopTop