Implementation of Machine Learning Models to Ensure Radiotherapy Quality for Multicenter Clinical Trials: Report from a Phase III Lung Cancer Study

Simple Summary Over 50% of all cancer patients receive radiation therapy (RT). The quality of the RT treatment plan is directly related to patient outcomes, such as overall survival and complications related to RT. In this study, we explore a knowledge-based machine learning tool for RT plan quality evaluation on plans submitted to a multicenter non-small-cell lung cancer clinical trial. The results of this study may provide critical information for the analysis of the end points of the trial. This study also demonstrated the feasibility of using this novel tool for RT plan quality assessment in the multicenter clinical trial setting. Abstract The outcome of the patient and the success of clinical trials involving RT is dependent on the quality assurance of the RT plans. Knowledge-based Planning (KBP) models using data from a library of high-quality plans have been utilized in radiotherapy to guide treatment. In this study, we report on the use of these machine learning tools to guide the quality assurance of multicenter clinical trial plans. The data from 130 patients submitted to RTOG1308 were included in this study. Fifty patient cases were used to train separate photon and proton models on a commercially available platform based on principal component analysis. Models evaluated 80 patient cases. Statistical comparisons were made between the KBP plans and the original plans submitted for quality evaluation. Both photon and proton KBP plans demonstrate a statistically significant improvement of quality in terms of organ-at-risk (OAR) sparing. Proton KBP plans, a relatively emerging technique, show more improvements compared with photon plans. The KBP proton model is a useful tool for creating proton plans that adhere to protocol requirements. The KBP tool was also shown to be a useful tool for evaluating the quality of RT plans in the multicenter clinical trial setting.


Introduction
About 50% of all cancer patients receive radiation therapy (RT). RT treatment plans that do not follow specific guidelines are associated with a lower survival probability [1,2], higher probability of disease progression [3] or increased risk of RT-related complications [4][5][6]. Therefore, quality assurance (QA) of RT plans is critical for patient care. Additionally, it is also essential for the success of clinical trials with an RT component.
Accurate delineation of the plan target volumes and adjacent organs at risk (OARs) is the prerequisite of a high-quality plan, which is beyond the scope of this study. Assuming the accuracy of all the delineations, the proper optimization of a plan by delivering uniform prescription doses to the target while mitigating dose deposition on the critical OARs is the main goal and therefore quality evaluation criterion. The treatment plan optimization process is a complicated process involving the skills and experiences of the planners. The Imaging Radiation Oncology Core (IROC) of the National Clinical Trials Network performed a QA review of the plans. All cases were subjected to the population-based dose constraint criteria defined in the trial protocol, which were based on past experiences of clinicians. Although this process identifies the plan that violates protocol constraints, it does not indicate the underlying reasons for the violation or the potential for quality improvement. A Knowledge-Based Planning (KBP) model uses a library of high-quality plans to provide a set of mathematical models between individual anatomy and the lowest achievable dose profiles to the OARs [7,8]. Personalized optimal treatment plans can be realized with the model predictions as plan optimization guidance. The model-generated predictions of dose profiles, when compared with submitted plan doses, can also serve as peer review of the plan quality. KBP for photon therapy is widely adopted in clinical settings [9][10][11][12][13][14], however, it is still in a novel stage for the RT QA in multicenter clinical trial settings. Several reports have shown the potential of this technique for the clinical trial QA of intensity-modulated radiation therapy (IMRT) [3,[15][16][17][18][19][20][21].
The installation of proton accelerators has increased substantially in recent years [22]. The physics of proton beam promises overall lower OAR doses. There is some clinical evidence of the dosimetric advantages of proton therapy vs. conventional photon therapy [23][24][25]. However, large-scale randomized clinical trials are needed to prove these advantages. With the complication of proton planning, proton KBP as an emerging technique is also under investigations [26][27][28][29][30]. No study has been reported for the use of proton KBP for clinal trial QA purposes.
The Radiation Therapy Oncology Group (RTOG) 1308 is a randomized phase III trial that compares overall survival after photon versus proton for inoperable stage II-III nonsmall-cell lung cancer (NSCLC) receiving concurrent chemotherapy and RT. The main goal of the trial is to see if proton therapy can improve overall survival compared with IMRT by lowering the risk of severe OAR toxicity. QA of the treatment plans submitted to both the photon and proton arms is essential for a fair comparison of these two modalities.
In this study, we used the KBP method for the QA of RTOG 1308 plans, with a focus on reporting the general quality of the plans submitted to both the photon and proton arms. Moreover, this is the first experience utilizing the proton KBP model for the QA of multicenter clinical trial proton plans.

Materials and Methods
The process of data selection, model training, and application of planned QA are described in this section.

Initial Data Review and Selection
Data from 210 patients enrolled in RTOG1308 at the time of this study were evaluated according to the IROC QA procedure. The treatment arm (photon or proton), the technique (passive scattering (PS) or intensity-modulated proton therapy (IMPT)), the type of treatment machine, and the dosimetric review in accordance with protocol dose constraints (per protocol: score 1, variation acceptable: score 2, and deviation unacceptable: score 3) were all collected. Table 1 summarizes the protocol's dosimetric constraints for performing the initial plan quality review. The review revealed that there were no deviation unacceptable cases and five score 2 cases among all IMPT cases; 5 deviation unacceptable cases and 9 variation acceptable cases among all PS cases; and 4 deviation unacceptable cases and 6 variation acceptable cases among all photon cases. Following the initial assessment, 130 patient data sets were chosen for this investigation. Fifty score 1 photon cases were randomly selected for model training. Eighty patients were selected as testing cases, and all cases of score 2 and score 3 were included with preferences. Among the 80 testing cases, 20 received IMPT, 20 received PS, and 40 received photon treatments. DICOM CT and RT structures of these 130 patients were imported into Eclipse Treatment Planning System (TPS) (Varian Medical Systems, Palo Alto, CA, USA).

Model Training
Original score 1 photon plans on the 50 training cases were used for initial photon model training (DVH estimation algorithm version 15.7.02).
IMPT plans to treat NSCLC with 2Gy/fraction, 35 fractions were generated manually in these 50 patients with ProBeam beam data. The dose distribution was optimized using fluence-base nonlinear universal proton optimizer (NUPO 16.0.2). The spot spacing was 0.425 of the energy-dependent in-air full-width half-maximum spot size at the isocenter. The multifield simultaneous spot optimization method was selected for all plans. A 5 cm range shifter was used for all fields. The Proton Convolution Superposition algorithm was used for the final dose calculation with a grid of 2.5 × 2.5 × 2.5 mm 3 . A constant relative biological effectiveness (RBE) of 1.1 was applied.
The manually generated 50 IMPT plans were reviewed as per protocol and were included for the initial proton model training (DVH estimation algorithm 16.0.2). Figure 1 illustrates the model training workflow for the photon and proton arms. Initial models were trained with reviewed per protocol plans. Plan optimization iterations were carried out as indicated in Figure 1 to ensure the optimal quality of the model plans. After two optimization iterations, the final models were generated.
The original plans submitted by multiple institutions used a variety of treatment planning systems (Pinnacle3, Elicpse, Raystation, and Elekta XIO) and were based on a variety of treatment machines (Photon: Varian LINACs (Clinac, Trubeam), Elecka. Proton: IBA, Hitachi, Mevion, ProBeam). The original plan beam angles were obtained from the DICOM headers of the submitted RT plans. A plan was created that used the original beam setup with the original RT dose file imported and attached to the plan. The photon plans were duplicated and then reoptimized based on the KBP models using the 10 MV beam models Clinac 23EX 15.6.03 ABX, with Millennium_120 leaf. Additionally, the same settings described in the model training section were used for the proton replan with the original submitted plan beam angles and model-based optimizations. The original plans submitted by multiple institutions used a variety of treatment planning systems (Pinnacle3, Elicpse, Raystation, and Elekta XIO) and were based on a variety of treatment machines (Photon: Varian LINACs (Clinac, Trubeam), Elecka. Proton: IBA, Hitachi, Mevion, ProBeam). The original plan beam angles were obtained from the DICOM headers of the submitted RT plans. A plan was created that used the original beam setup with the original RT dose file imported and attached to the plan. The photon plans were duplicated and then reoptimized based on the KBP models using the 10 MV beam models Clinac 23EX 15.6.03 ABX, with Millennium_120 leaf. Additionally, the same settings described in the model training section were used for the proton replan with the original submitted plan beam angles and model-based optimizations.
Model optimization priorities were manually adjusted on selected testing cases (with challenging anatomy and large tumor volume) for several testing runs.

Plan Quality Review by Models
The submitted plans were compared with the KBP plans dosimetrically. The general quality differences between the originally submitted plans and the KBP plan were analyzed using mean dosimetric points and the t-test. Individual score 3 plans were also examined to determine whether there was a possibility for quality improvement.

Final Model Settings
The KBP platform provides model settings for treatment planning optimization objectives and priorities of the objectives. These model objectives were fine-tuned to produce a plan with uniform target dose coverage and optimized OAR sparing after a single iteration of optimization. The finalized priority settings are reported in Table 2. Both models will be published for researchers and clinicians to access for the optimization of the RT plan and QA purposes. V30Gy ≤ Generated Value Generated priority Line (preferring target) Generated priority Model optimization priorities were manually adjusted on selected testing cases (with challenging anatomy and large tumor volume) for several testing runs.

Plan Quality Review by Models
The submitted plans were compared with the KBP plans dosimetrically. The general quality differences between the originally submitted plans and the KBP plan were analyzed using mean dosimetric points and the t-test. Individual score 3 plans were also examined to determine whether there was a possibility for quality improvement.

Final Model Settings
The KBP platform provides model settings for treatment planning optimization objectives and priorities of the objectives. These model objectives were fine-tuned to produce a plan with uniform target dose coverage and optimized OAR sparing after a single iteration of optimization. The finalized priority settings are reported in Table 2. Both models will be published for researchers and clinicians to access for the optimization of the RT plan and QA purposes.

Proton KBP Model Evaluation
All initial manual IMPT plans built on the 50 training patients met all protocol dose constraints (Table 1). We performed two iterations of plan optimizations, followed by addition of the reoptimized plans for training new models to remove potential dosimetric outliers and enhance the model performance. The results of the dosimetric comparison and the t-test of 50 manual IMPT and the final KBP plans are shown in Table 3. Compared with manual plans, KBP plans demonstrate statistically significant improvements in OAR protection while maintaining the same or greater target coverage, demonstrating the efficacy of the KBP tool for IMPT plan optimizations.

Plan Quality Review
We used the models to reoptimized 40 photon plans and 40 proton plans submitted to RTOG 1308. Additionally, we compared the model-based plans with the original submitted plans to evaluate the quality of the plans from both photon and proton arms.

Photon Plan Quality Review
The dosimetric comparison between the KBP photon plans and the submitted photon plans, together with the results of the t-test, are reported in Table 4. The KBP photon plans demonstrate overall optimized heart and lung doses without significant changes at other dosimetric points. There were six-variation (score 2) and four-deviation (score 3) photon plans among the forty testing plans. Two out of the six score 2 plans were improved to score 1 plans. Only one score 3 plan was improved to score 2. Three out of the four score 3 plans were analyzed to be of good quality; no further optimization was realized.

Proton Plan Quality Review
The critical dosimetric points comparison between KBP IMPT and the original proton plans submitted was listed separately for the original IMPT and PS cases in Table 5 due to the intrinsic difference of PS plans and IMPT plans. KBP IMPT plans demonstrate statistically significant improvements in target coverage (PTV D99%[Gy] indicates target dose coverage) and OAR sparing (especially lower heart and lung doses) for both cohorts of patients. All KBP IMPT plans in the 40 testing patients met all protocol dose constraint criteria given in Table 1, including the original five variation acceptable IMPT plans, nine variation acceptable, and five deviation unacceptable PS plans submitted. Box plots were also generated to show the dosimetric points comparison between the original plans and the KBP IMPT plans for the two cohorts of patients in Figure 2. Box plots show that the original submitted proton plans (both IMPT and PS) vary in target coverage (PTV D99%[Gy]) with several PS plans deviating from the average PTV D99%[Gy] by up to 10 to 15 Gy. KBP IMPT plans provide more uniform target coverage while reducing overall doses to the heart, lungs, and esophagus. Figure 3 shows the screen capture of the 3D dose wash comparison of the original IMPT plan versus the KBP IMPT plan. The KBP IMPT plan significantly reduced dose spillage to normal lung tissue and improved dose uniformity to the tumor.

Discussions
We attempted a comparison between plans generated with photon vs. proton beams for some of the cases. KBP IMPT plans were created in 40 testing patients from the photon cohort, with detailed findings reported in the Appendix A section. Generally, the findings

Discussions
We attempted a comparison between plans generated with photon vs. proton beams for some of the cases. KBP IMPT plans were created in 40 testing patients from the photon cohort, with detailed findings reported in the Appendix A section. Generally, the findings demonstrate the dosimetric superiority of proton therapy compared with photon therapy. However, to realize this dosimetric superiority, optimal proton plan quality is required. Although the plans submitted to both arms are of acceptable quality, proton plans exhibit a greater degree of variation in quality and indicate greater room for improvement. The results of this study may provide critical information for the analysis of the trial end point.
Due to the limitations of the double scatter techniques, some initial PS plans failed to meet the protocol dose constraints. KBP IMPT, on the other hand, easily met the protocol criteria for those patients. This could imply that IMPT has inherent dosimetric advantages over PS in certain patients with difficult anatomy. All proton plans submitted to this trial five years ago utilized the PS method; however, the majority of recent submissions utilized IMPT. This demonstrates the evolution of proton treatment techniques.
IMRT plans may not be subject to significant quality variations caused by treatment machines. However, the quality of the IMPT plan is influenced by the treatment machine's beam quality, spot size, range modulators, and original beam energy ranges. The model used in this investigation was trained using plans constructed from beam models provided by the treatment planning system manufacturer as golden beam models. Due to the limitations of the treatment machines, the plan quality achieved in this study may not be replicable in the enrolling centers. The test results presented in this study just indicate the feasible plan quality with the beam models and techniques; they do not indicate the specific causes of the variation in plan quality.

Conclusions
This study summarizes the general quality of the RT plans submitted to multicenter clinical trials. KBP models were used to conduct a more thorough review of the quality of the plan and the potential for improvement. Proton plans, a relatively emerging technique, show more variation in quality than photon plans, which are consistent with good quality and have little room for improvement with existing approaches. The KBP IMPT model is a useful tool to create IMPT plans that adhere to protocol requirements. The KBP tool was also shown to be a helpful tool for reviewing the quality of RT plans in the multicenter clinical trial setting. Both photon and proton models built using multicenter clinical trial data in this study will be published to researchers as well as clinicians to access for RT plan optimization and QA purposes.

Institutional Review Board Statement:
This study was in accordance with the Declaration of NCI Central Institutional Review Board (NCI CIRB) and was approved by the Ethical Committee of NICI Community Oncology Research Program (NCORP) Ethics Board. Sites participating with the NCI CIRB must get the local IRB approval through NCI CIRB.

Informed Consent Statement:
All patients who participated in the multicenter clinical trial have signed fully informed consent forms provided in the trial. All patient data used in this study followed NCI patient deidentification guidelines.
Data Availability Statement: 3rd Party Data: Restrictions apply to the availability of these data. Data were obtained from Imaging Oncology Core Radiotherapy Quality Assurance team. This is an ongoing trial; no data will be made available to the public before any publication of the end point of this trial. After the closure and publication of the endpoint of this trial, data can be applied via data sharing through NRG oncology.

Acknowledgments:
We would like to give thanks to Pekka Uusitalo and Reynald Vanderstraeten from Varian Medical Systems for providing the knowledge-based planning platform used in this study.

Conflicts of Interest:
Dr. Bradley reports grants and personal fees from Mevion Medical Systems, Inc., Littleton, MA 01460 grants, personal fees, and other from ViewRay, Inc., Oakwood, OH 44146 personal fees and other from Varian Medical Systems, outside the submitted work. "The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results". Other co-authors of this manuscript have no conflict of interest.

Appendix A
As mentioned in the discussion section, KBP IMPT plans were generated on the 40 patients from the photon cohort for plan quality review also. The purpose of this is to study the potential dosimetric advantages of the proton plan over the photon plan for NSCLC. The dosimetric comparison between original photon plans and the KBP IMPT plans are presented in Table A1, and box plots are shown in Figure A1.