Optimization of Replanning Processes for Volumetric Modulated Arc Therapy Plans at Risk of QA Failure Predicted by a Machine Learning Model

Featured Application

Automatable replanning strategies can improve patient-specific quality assurance outcomes of radiotherapy plans identified at risk by machine learning models without compromising the dosimetric quality.

Abstract

Patient-specific quality assurance (PSQA) procedures ensure the safe delivery of volumetric modulated arc therapy (VMAT) plans. PSQA requires extensive time and resources and may cause treatment delays if replanning is needed due to failures. Recently, our group developed a machine learning (ML) model predicting gamma passing rate (GPR) for VMAT arcs. This study explores automatable replanning strategies for plans identified at risk of failure, aiming to improve deliverability while maintaining dosimetric quality. Between 2022 and 2023, our ML model analyzed 1252 VMAT plans. Ten patients having a predicted GPR (pGPR) <95% were selected. Replanning strategies consisted of limiting monitor units (MUlimit) and employing the aperture shape controller (ASC) tool. Re-optimized plans were compared with the originals in terms of dose volume constraints (DVCs) for the target and organs-at-risk (OARs), and deliverability using the modulation complexity score (MCS), pGPR, and measured GPR (mGPR). Forty-five re-optimizations were performed. Replanning led to an increase in DVCs for OARs and a reduction for the target. Complexity decreased, reflected by the increase in the MCS from 0.17 to 0.21 (MUlimit) and 0.20 (ASC). The deliverability improved, with the pGPR increasing from 93.3% to 94.4% (MUlimit) and 95.1% (ASC), and the mGPR from 99.3% to 99.7% (MUlimit) and 99.8% (ASC). Limiting the MUs or utilizing the ASC reduced the complexity of plans and improved the GPR without compromising the dosimetric quality. These strategies can be used to automate replanning procedures, reduce the workload related to PSQA, and improve patient safety.

1. Introduction

Patient-specific quality assurance (PSQA) procedures based on measurements are recommended by the American Association of Physicists in Medicine Task Group 218 (AAPM TG218) to ensure that intensity-modulated radiation therapy (IMRT) treatment plans can be delivered as intended [1]. Typically, the treatment plan dose is recalculated on a phantom and then compared to the dose measured using suitable detectors [2]. The most common procedure to compare the planned and measured dose distributions is the ɣ-index analysis, which quantifies their agreement by considering both the dose and the physical distance [3]. The fraction of points having ɣ < 1 is referred to as the gamma passing rate (GPR). The AAPM TG218 recommends, for IMRT plans, the universal action limits of a GPR ≥ 90%, with 3%/2 mm and a 10% dose threshold.

One major challenge of traditional measurement-based PSQA is the extensive time required to complete the procedures. Additionally, the necessity for replanning and the consequent increase in workload due to PSQA failures may lead to postponing the patient treatment schedules. Consequently, numerous research initiatives have explored the adoption of machine learning (ML) models based on plan complexity indices as virtual QA (VQA) approaches to accurately predict the GPR for IMRT treatments [4,5]. Among various IMRT techniques, volumetric modulated arc therapy (VMAT) is becoming the most common, where adjustments in gantry speed, the positioning of the multi-leaf collimator, and the dose rate are dynamically modulated for enhanced treatment precision [6,7].

Recently, our group trained a tree-based ensemble ML model on a large dataset of over 12,000 VMAT arcs with GPRs measured through electronic portal imaging device (EPID) dosimetry [8]. The model was validated on an internal test set of 1871 VMAT arcs. To this aim, 19 numerical features describing the complexity of an arc were considered to predict the GPR. These complexity parameters quantified the irregularity in a multileaf collimator (MLC) shape, monitor units (MUs), leaves’ travel motion and speed, gantry speed and acceleration, dose rate, and field size. All of these factors contribute to the uncertainties related to the ability of the treatment planning system (TPS) to accurately model the dose distribution and the linac’s delivery capability, which might compromise the PSQA result [9,10,11,12].

In this study, we investigated the possibility of developing an automatic re-optimization process for VMAT plans identified at risk of failing the PSQA by our ML model with the aim to improve their deliverability, i.e., reduce the complexity and increase the GPR, while preserving a high dosimetric quality. Our central assumption is that VQA combined with automation could improve the patient’s safety and reduce the overall workload. In particular, we considered different treatment regions, comprising the head and neck, mediastinum, esophagus, thorax, and prostate, and considered several replanning strategies, including the reduction of the delivered MUs and simplification of the MLC aperture shape during optimization.

2. Materials and Methods

Between 2022 and 2023, we analyzed all 1252 VMAT plans delivered by two Varian TrueBeam machines at our institute, equipped with Varian Millenium 120-leaf MLCs. Treatment plans were optimized using the Eclipse TPS v15 (Varian Medical Systems, Palo Alto, CA, USA) with the Photon Optimizer (PO) and dose calculated with the Analytical Anisotropic Algorithm or Acuros XB algorithm. These plans were not part of the dataset used to develop our ML model [8]. Briefly, the model (xgboost) is an ensemble of boosted trees predicting the GPR of a VMAT arc with 3%/1 mm gamma criteria.

In order to collect a sample size reasonable for analysis, ten patients from different regions were selected using the following filtering criteria:

• Average predicted GPR (pGPR): plans with a pGPR less than 95% averaged over the arcs, using a 3%/1 mm gamma criteria and a 10% dose threshold, were considered.
• Minimum field size: fields larger than 5 × 5 cm² were considered. Preliminary investigations showed that the ML model underestimated the GPR for arcs with smaller field apertures. Moreover, a larger field size was crucial to mitigate the uncertainties in small field dosimetry and degradation of the EPID center [11,13,14].
• Case selection: brain and breast cases were excluded due to their specific requirements and the complexity of their treatment plans, which necessitate the intervention of an experienced planner. In particular, most brain cases are optimized with a dedicated algorithm (HyperArc; Varian Medical Systems, Palo Alto, CA, USA), while for breast cases a non-automatable procedure is required to account for the patient’s breathing motion [15].

Table 1. Dose prescription and clinical constraints of the ten treatment plans identified. The constraints on the PTV were the same for all treatments: V95% ≥ 95%, V100% ≥ 50%, and V105% < 2%. The constraint for the CTV was V95% ≥ 98%.

Treatment	Number of Cases	Prescription	Clinical Constraints
Head and neck	4	69.96 Gy/33 fr	Lenses Dmax < 6 Gy D0.03cc < 6 Gy Cochlea Dmean < 45 Gy Brainstem Dmax < 54 Gy Temporal lobe Dmax < 65 Gy D0.03cc < 65 Gy Parotids Dmean < 26 Gy Oral cavity Dmean < 40 Gy	Mandible D2% < 70 Gy Submandibular Dmean < 35 Gy Thyroid V50 Gy < 70% Esophagus Dmean < 45 Gy Spinal cord D0.03cc < 45 Gy Dmax < 45 Gy Brachial plexus Dmax < 66 Gy D0.03cc < 66 Gy
Esophagus	1	55 Gy/25 fr	Spinal cord D0.03cc < 40 Gy Dmax < 45 Gy Dmax < 40 Gy Heart V30 Gy < 30% Dmean < 30 Gy	Liver V30 Gy < 20% Dmean < 25 Gy V20 Gy < 30% Stomach Dmax < 54 Gy Dmean < 30 Gy
Mediastinum	1	20 Gy/10 fr	Esophagus D0.1cc < 48 Gy Dmax < 48 Gy D5cc < 40 Gy	Spinal cord Dmax < 36 Gy D0.1cc < 36 Gy
Thorax	1	30 Gy/15 fr	Esophagus Dmean < 27 Gy D0.1cc < 54 Gy V42 Gy < 32% D5cc < 45 Gy Dmax < 54 Gy	Spinal cord Dmax < 40 Gy D0.1cc < 40 Gy D5cc < 39 Gy Heart D15cc < 42 Gy D0.1cc < 48.9 Gy Dmax < 48.9 Gy Dmean < 29 Gy
Prostate	3	67.5 Gy/25 fr	Small bowel V36 Gy < 195cc Femoral heads V40 Gy < 10%	Rectum D1cc < 68 Gy Dmax < 68 Gy V50 Gy < 30% V40 Gy < 50% Bladder V50 Gy < 50% Penile bulb Dmean < 40 Gy

Abbreviations: CTV = clinical target volume, PTV = planning target volume.

summarizes the characteristics of the selected plans, all optimized with PO and dose calculated with AcurosXB. The treatment plans included various anatomical regions, from head to pelvis, and dose prescriptions. All constraints were defined according to the available literature and our internal experience.

The 10 plans were randomly split into two groups to optimize the replanning procedure. In particular, in the first phase, three plans were considered to identify the best strategies that balanced dosimetric quality, complexity, and deliverability. In the second phase, the remaining seven plans were replanned using the selected strategies to assess the overall improvement of the plans.

The first three plans were head and neck, mediastinum, and thorax cases. For each plan, eight replanning strategies were considered, acting on different TPS parameters:

• Manual re-optimization (MRO): performed by an expert planner who had the flexibility to modify the plan’s parameters (e.g., jaw apertures, isocenter position) and to adjust weights and constraints during the optimization.
• Optimization restart (NoChanges): restarting the optimization process from scratch without any changes during the optimization.
• MU limitation (MUlimit): capping the maximum value of MUs to 70% of the original plan’s value.
• Aperture Shape Controller (ASC) levels—a specific tool of Eclipse to limit plan complexity: utilizing the five available ASC levels, i.e., very low (ASC_VL), low (ASC_L), medium (ASC_M), high (ASC_H), and very high (ASC_VH).

Except for the MRO strategy, which served as a control sample, all plans were optimized with unaltered constraints and no changes during the optimization to enable automatic execution without human intervention. MU limits were imposed based on previous works showing an improved plan deliverability without compromising the overall clinical plan quality [16,17]. On the other hand, preliminary studies have shown that similar results can be achieved using the ASC [18,19]. ASC was recently introduced in the Eclipse TPS for the PO algorithm to favor apertures of minimal local curvature. The ASC strength is controlled by the user via a calculation option that can be set to five different values, ranging from very low to very high. An increasing strength applies a larger penalization weight to the total cost function of PO. As a result, the occurrence of disconnected apertures reduces, effectively limiting the complexity of MLC movement and modulation.

The modulation complexity score (MCS) was selected to characterize the complexity of the VMAT arcs [20]. MCS combines information on the variability in leaf positions, irregularity in field shape, segment weight, and area, into a single score ranging from 0 (high complexity) to 1 (open rectangular field, no complexity).

For each plan, PSQA was measured using the EPID attached to the machines’ gantry and the analyses were performed with Portal Dosimetry software v15.6 (Varian Medical Systems, Palo Alto, CA, USA). At the start of each QA session, the EPID was re-calibrated. Then, the gamma analysis of the images obtained from the PSQA was executed automatically for each arc. The measured GPR (mGPR) with 3%/1 mm and a 10% dose threshold was extracted using an in-house script based on the Eclipse Scripting API (ESAPI). Similarly, a second tool was used to extract the points of the DVH that corresponded to the constraints of each specific region treated (see Table 1). In addition, a range of volume percentage statistics (VX% for X = 10, 20, …, 100) for the BODY were extracted to evaluate the overall dose conformity.

Based on the analysis on the three plans, three replanning processes were selected, corresponding to strategies which effectively balanced improved deliverability and DVH quality. The remaining seven plans were re-optimized according to the selected strategies and the same analysis was conducted.

The newly optimized plans were compared with the original ones to assess potential enhancements in terms of:

• clinical quality, through the dose volume constraint (DVC) specific to the treatment region;
• deliverability of the arcs, through the MCS, the pGPR, and the mGPR.

For each of these parameters, the median value and the interquartile range (IQR) were evaluated. To aggregate DVC comparisons between re-optimized and original plans for different treatment sites, the DVC differences were normalized to the original plan’s value:

$$DV{C}_{norm}=\frac{DV{C}_{re-opt}-DV{C}_{orig}}{DV{C}_{orig}}\times 100$$

Furthermore, the DVC comparisons were grouped by clinical/planning target volume (CTV/PTV), organs at risk (OARs), and BODY. During the first phase, an initial exploratory analysis was conducted on the $DV{C}_{norm}$, MCS, and GPR to identify plans reaching the optimal balance between improved deliverability and dosimetric quality. Then, the results obtained with the three selected optimization strategies were compared with the clinical plans through the Wilcoxon signed rank test, with a significance level of 0.05.

3. Results

Figure 1 shows an example of a head and neck case replanned using the selected strategies.

Figure 2 shows the $DV{C}_{norm}$ for CTV/PTV, BODY, and OARs for the first three patients replanned using all eight different strategies (i.e., 32 plans for a total of 81 arcs). Similarly, Figure 3 shows the box plots of the MCS, pGPR, and mGPR. Overall, we observed an average increase in DVC_norm for OARs (+6.5%) and BODY (+1.5%), with a decrease for CTV/PTV (−1.3%).

The following optimization strategies were discarded according to these reasons:

• NoChanges: it produced arcs with the highest complexity (MCS = 0.14) and worse pGPR = 92.5%.
• ASC_VL: it produced high complexity arcs (MCS = 0.16) with the lowest pGPR = 92.5% and mGPR = 98.8% among the ASC settings.
• ASC_VH: despite it yielding the highest pGPR = 95.5% and mGPR = 99.3%, it was the worst method with respect to the DVC_norm values, with −2.4% for CTV/PTV, +7.0% for OARs, and +2.6% for BODY.
• ASC_M: this strategy was comparable to ASC_H in terms of all the considered parameters.

In order to explore polarized settings, we eventually opted to consider ASC_L and ASC_H for the subsequent analysis. Furthermore, we included the MUlimit as it produced arcs with minimal complexity (MCS = 0.21) and because of its widespread implementation across many commercial TPSs. Therefore, the remaining seven patients were re-optimized with these three strategies and the analysis repeated, totaling 128 arcs from 40 treatment plans.

Table 2. Change in DVCs and deliverability for the different replanning strategies. Median values with the 1st and 3rd quartile are reported. Significant differences between replanned and clinical plans are marked with *.

Parameter	Clinical	MUlimit	ASC_L	ASC_H
DVCnorm CTV/PTV (%)	-	−1.5 * [−2.4, −1.2]	−0.6 [−0.9, −0.4]	−0.8 * [−1.2, −0.3]
DVCnorm OARs (%)	-	1.9 * [1.2, 4.6]	1.5 [−0.4, 2.7]	2.3 * [0.1, 4.7]
DVCnorm BODY (%)	-	1.2 [0.1, 2.8]	0.6 [−0.1, 2.0]	0.9 [0.7, 2.4]
MCS	0.17 [0.13, 0.18]	0.21 * [0.17, 0.24]	0.19 * [0.14, 0.21]	0.20 * [0.15, 0.24]
pGPR 3%/1 mm (%)	93.3 [89.8, 94.9]	94.4 [91.2, 96.5]	95.0 * [92.4, 97.3]	95.1 * [93.5, 96.1]
mGPR 3%/1 mm (%)	99.3 [98.3, 99.8]	99.7 [99.2, 99.9]	99.5 * [99.2, 99.9]	99.8 * [99.3, 99.9]

Abbreviations: ASC = aperture shape controller, DVC = dose volume constraints, MCS = modulation complexity score, mGPR = measured gamma passing rate, MU = monitor unit, pGPR = predicted gamma passing rate.

summarizes the change in the DVC and deliverability among the different replanning procedures. For all of them, we observed a median increase in DVCs for OARs and BODY, and a median reduction in the CTV/PTV. In particular, we found significant differences in the DVC_norm of CTV/PTV and OARs for the MUlimit and ASC_H strategies. Figure 4 shows the box plots of the DVC_norm for all of the considered structures. We highlight that the replanning processes did not compromise the overall dosimetric quality of the plans since the clinical constraints were still respected for all cases.

Figure 5 shows the box plots of the MCS, pGPR, and mGPR for all of the analyzed strategies. The re-optimization strategies produced a significant decrease in the complexity of the arcs, as indicated by the increase in MCS from a median value of 0.17 [inter quartile range (IQR) = 0.05] to 0.21 [IQR = 0.07], 0.19 [IQR = 0.07] and 0.20 [IQR = 0.09] for, respectively, the MUlimit, ASC_L, and ASC_H. In particular, a larger variability was observed in the latter approach. The reduced complexity was reflected in an improved deliverability. The pGPR increased from a median of 93.3% [IQR = 5.1%] to 94.4% [IQR = 5.3%] (MUlimit), 95.0% [IQR = 4.9%] (ASC_L), and 95.1% [IQR = 2.6%] (ASC_H), while the median mGPR increased from 99.3% [IQR = 1.6%] to 99.7% [IQR = 0.8%] (MUlimit), 99.5% [IQR = 0.7%] (ASC_L), and 99.8% [IQR = 0.6%] (ASC_H). For both the pGPR and mGPR, the increase was significant when using the ASC setting. Notably, the pGPR was susceptible to larger variations due to the intrinsic uncertainty in the ML model’s predictions.

4. Discussion

Automation in radiotherapy treatment planning has been steadily increasing in recent years. Automatic optimization processes have the potential of improving the efficiency of the radiotherapy workflow by standardizing planning procedures and enhancing the overall quality of generated plans [21,22,23]. In particular, artificial intelligence represents a promising tool for personalized tailored treatments for a wide range of diseases [24,25,26].

This study presented a proof of concept for introducing an automatable replanning strategy using an ML-based VQA. In fact, despite VQA being able to streamline measurement-based PSQA [4,5,8,27], the replans and associated workload due to a failed PSQA can result in delayed clinical treatments. Log file analysis could provide a time-efficient alternative method for detecting both small systematic and random errors in MLC positioning during delivery. However, this approach is regarded as lacking independence from the delivery system and there are still concerns about whether it could offer the same confidence as conventional PSQA methods [28,29].

We selected ten patients found at risk of PSQA failure by our ML model and investigated three replanning procedures consisting of limiting the delivered MUs (MUlimit) or simplifying the MLC aperture shape during optimization (ASC_L and ASC_H). To assess the optimizations outcomes, we considered the plan dosimetric quality, expressed in terms of DVCs, and plan deliverability, quantified by plan complexity with the MCS and GPR. This ensured a comprehensive evaluation of plan quality [30].

Overall, the replanning methods produced an increase in DVCs for OARs and BODY, with a reduction for CTV/PTV. In particular, the largest significant differences were found with the MUlimit and ASC_H for, respectively, CTV/PTV and OARs. Nonetheless, we highlight that the clinical constraints for the re-optimized plans were still respected in all cases. In particular, the target coverage remained within acceptable limits, and doses to OARs were optimized following the ALARA principle (As Low As Reasonably Achievable).

Regarding the deliverability, the three strategies significantly reduced the plan complexity, with MUlimit producing the highest MCS. This trend was confirmed by the mGPR results, which on average increased in all cases, though the changes were not significant for the MUlimit strategy. We hypothesize this was due to external factors, such as EPID degradation and setup, which could have influenced the PSQA measurements and produced low mGPRs (see Figure 5).

As the EPID is a perpendicular composite method, the integrated image may mask some dose delivery errors, such as those in the scattered regions and those due to non-uniform dose rate. Moreover, EPIDs are not ideal absolute dosimeters and require careful calibration due to their smaller thickness compared with the X–ray build-up. Utilizing other devices for PSQA with characteristics other than EPID (e.g., ArcCHECK, Delta4 Panthom+, or MatriXX) could affect the gamma analyses and should be investigated.

Our results are aligned with other studies, which demonstrated that these tools reduce the plan complexity but tend to increase the dose to normal tissue and reduce the dose to the target compared with the reference clinical plan. Specifically, Scaggion et al. investigated the use of MUlimit and ASC_VH in ten prostate and ten oropharynx plans [18]. They reported the ASC was effective for reducing the unnecessary complexity of a plan, with an increased plan deliverability and no loss of plan quality. In another work, Mancosu et al. assessed the effect of the MUlimit in ten patients with breast cancer. The authors concluded that the MUlimit can be used in case of replanning to improve the mGPR, paying attention to the DVCs of OARs which could get compromised [17].

All of the optimization strategies investigated in this study are viable options to replan cases flagged by VQA and generate plans with lower complexity and an improved GPR. Our findings indicate that the optimal approach is the ASC_L, producing significant improvements in deliverability (MCS, pGPR, and mGPR) without compromising the dosimetric quality. Additionally, the MUlimit could be considered as well, being a general approach implemented across many TPSs. We highlight that the GPR criteria and action limit considered in this study (3%/1 mm, 95%) were stricter than those recommended by the AAPM TG-218 (3%/2 mm, 90%). In our clinical experience, only a minor fraction of the treatment plans delivered do not satisfy the recommended criteria [8]. Thus, with a view to continuous improvement in quality, we decided to train our ML model using more stringent criteria for PSQA. Specifically, VQA was added as an additional layer of control to our PSQA program (see Figure S1 in the Supplementary Material).

The ten patients were selected based on the plan average pGPR given by our ML model. However, only one arc was found to be an actual failure (mGPR < 95%). Despite the expected specificity of the model being 0.90, this high false negative rate could be explained by the small sample size and the inherent uncertainty in the model predictions. As VQA is necessary to implement a targeted PSQA approach and reduce the workload of measurement-based PSQA, it is also crucial that the subsequent replannings do not slow down the radiotherapy workflow. Importantly, the automatable replanning strategies that we considered provided a means to re-optimize plans flagged by VQA and improve both the pGPR and mGPR. Given the results of our analysis, after re-optimization, the action limit on the pGPR was set to 90%. All plans were then considered acceptable by VQA.

The automation of the replanning strategies is expected to reduce the QA workload by more than 10 h/month, assuming 10 patients are reoptimized per month and 1 h of work per plan. In busy clinical settings, real-time automatic replannings could be performed in the background before plan approval if additional licenses and computing power are available. Alternatively, replannings could be performed at the end of each working day.

This study has some limitations. The improved deliverability and reduced complexity could improve patient safety due to reduced uncertainties in dose calculation and verification. However, long-term outcomes need to be monitored over a prolonged period in order to provide definitive conclusions on patient quality of life. Our ML model can be applied only to VMAT plans, and the applicability of the model to other treatment deliveries and settings remains an open question. Future investigations will need to address the applicability of the replanning strategies to other treatment regions and increase the number of cases, as well as considering a higher threshold of GPR (e.g., 97.5%) for continuous quality improvement.

5. Conclusions

Limiting the MUs or the irregularity in the MLC aperture shape can reduce the complexity of plans and improve their GPR without compromising the overall dosimetric quality. These strategies can be used to automate replanning procedures for VQA, reduce the workload associated with PSQA failures identified by ML models, and improve patient safety.