Review Reports

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This manuscript investigates the behavior of plan complexity metrics (PCMs) in CBCT-based online adaptive radiotherapy (oART) for prostate cancer delivered with the Ethos platform. The study retrospectively analyzes 451 treatment plans from 21 patients across three anatomical target groups (prostate alone, prostate with seminal vesicles, and prostatic bed).

I find the manuscript worthy of publication, pending minor revisions and clarifications:

1) The abstract contains numerous abbreviations that are not defined at first mention. This significantly affects readability and clarity, especially for readers who may not be deeply familiar with adaptive workflows. 

2) Reference 13 does not report a publication year and could not be identified in the literature. Please verify and complete the reference details to ensure traceability and scientific rigor.

3) The manuscript states that there are no large day-to-day anatomical variations in prostate cancer. This statement is questionable, as prostate anatomy is well known to vary daily due to changes in rectal and bladder filling, which directly affect target shape and position if patient preparation is not strictly reproducible. The rationale for adaptive radiotherapy in prostate cancer is precisely to account for these inter-fraction anatomical deformations. The sentence should be reformulated to avoid implying that prostate anatomy is inherently stable compared to other anatomical sites. In fact, prostate anatomy is often considered more variable than many other treatment sites.

4) A discussion comparing IMRT and VMAT in the context of adaptive workflows would strengthen the methodological justification. While VMAT is mentioned in the context of generalizability, the rationale for choosing IMRT over VMAT in this study should be clarified, particularly considering delivery time.

5)  In Figure 3, the meaning of the different colors is not clearly explained. The figure legend should explicitly describe what each color represents. Currently, the interpretation is not straightforward.

6) What criteria triggered adaptation? The manuscript mentions choosing between scheduled and adapted plans, but it appears that adapted plans were consistently selected. Please clarify whether any scheduled plans were delivered and, if so, under which circumstances.

7) The conclusions state that PCMs may serve as surveillance indicators within a statistical process control framework rather than as substitutes for measurement-based QA. This point should be clarified further. What QA workflow do the authors suggest for centers starting with oART? A more explicit practical recommendation would enhance the clinical relevance of the manuscript.

8) Please revise the reference style throughout the manuscript. In some instances, references are cited with the year in brackets, while in others only the author name is mentioned. Ensure consistent formatting according to journal guidelines.

Author Response

[COM] - 1) The abstract contains numerous abbreviations that are not defined at first mention. This significantly affects readability and clarity, especially for readers who may not be deeply familiar with adaptive workflows.

[ANS]  We thank the reviewer for this suggestion. We clarified the abbreviations in the abstract section.

 

[COM] - 2)  Reference 13 does not report a publication year and could not be identified in the literature. Please verify and complete the reference details to ensure traceability and scientific rigor.

[ANS]  Thank you for pointing this out. Reference 13 corresponds to a manuscript that been recentrly accepted for publication in Medical Physics. We are uploading along the manuscript.

 

[COM] - 3) The manuscript states that there are no large day-to-day anatomical variations in prostate cancer. This statement is questionable, as prostate anatomy is well known to vary daily due to changes in rectal and bladder filling, which directly affect target shape and position if patient preparation is not strictly reproducible. The rationale for adaptive radiotherapy in prostate cancer is precisely to account for these inter-fraction anatomical deformations. The sentence should be reformulated to avoid implying that prostate anatomy is inherently stable compared to other anatomical sites. In fact, prostate anatomy is often considered more variable than many other treatment sites.

[ANS]  We agree with the reviewer and we have amended the text to avoid any misunderstanding.

 

[COM] - 4) A discussion comparing IMRT and VMAT in the context of adaptive workflows would strengthen the methodological justification. While VMAT is mentioned in the context of generalizability, the rationale for choosing IMRT over VMAT in this study should be clarified, particularly considering delivery time.

[ANS]  We agree with the reviewer that the choice of IMRT over VMAT requires further clarification. While VMAT is known for its rapid delivery, the online re-optimization of VMAT plans is more computationally intensive and time-consuming than static-field IMRT. By selecting a 9-field IMRT geometry, we achieved a fast and stable dose re-calculation (approx. 5 minutes) and a total workflow duration (15–20 minutes) that effectively prevents significant intra-fraction bladder filling. We included the explanation at the end of section 2.2

 

[COM] - 5)   In Figure 3, the meaning of the different colors is not clearly explained. The figure legend should explicitly describe what each color represents. Currently, the interpretation is not straightforward.

[ANS]  Figure 3 has been completely revised to clarify the information expressed therein.

 

[COM] - 6) What criteria triggered adaptation? The manuscript mentions choosing between scheduled and adapted plans, but it appears that adapted plans were consistently selected. Please clarify whether any scheduled plans were delivered and, if so, under which circumstances.

[ANS]  The scheduled/adapted selection is based on real-time assessment which follow a traffic-light protocols built upon the clinical endpoint of the treatment. It detailed explanation, in our opinion, lies outside the scope of this work since it is not directly related to the plan dosimetric quality assurance endpoint expressed herein.

 

[COM] - 7) The conclusions state that PCMs may serve as surveillance indicators within a statistical process control framework rather than as substitutes for measurement-based QA. This point should be clarified further. What QA workflow do the authors suggest for centers starting with oART? A more explicit practical recommendation would enhance the clinical relevance of the manuscript.

[ANS]  The reviewer has a point, it is difficult to translate the presented data in an explicit practical reccomandation, so we smoothed out the conclusion section.  

[COM] - 8) Please revise the reference style throughout the manuscript. In some instances, references are cited with the year in brackets, while in others only the author name is mentioned. Ensure consistent formatting according to journal guidelines.

[ANS]  We thank the reviewer for highlighting this issue. The reference style has been revised throughout the manuscript for consistency: author names have been retained, and the year of publication has been omitted.

Reviewer 2 Report

Comments and Suggestions for Authors

General Comments

This manuscript addresses a relevant and timely topic in modern radiotherapy: the characterization of plan complexity and its relationship to quality assurance in AI-assisted CBCT-based online adaptive radiotherapy for prostate cancer. The study is clinically meaningful because measurement-based PSQA cannot realistically be performed for every adapted fraction in the oART setting, and the authors propose plan complexity metrics (PCMs) and SPC-based monitoring as practical complementary tools. The manuscript is generally well structured, the clinical motivation is clear, and the use of linear mixed-effects models, variance decomposition, and LOPO-CV adds methodological value. The dataset includes 451 plans from 21 patients across three anatomical target groups, which is a reasonable basis for a single-institution technical study.

The main findings are that the Ethos IOE produced highly modulated plans, that no significant differences were detected between offline and adapted plans for the selected PCMs, that all PCMs were significantly associated with GPR although with low marginal R² values, and that LOPO-CV supported the feasibility of cohort-informed SPC thresholds. The manuscript also appropriately recognizes that PCMs are not surrogates for PSQA but may be useful as surveillance indicators within a layered QA workflow.

Overall, the work is interesting and publishable. However, several statements should be softened, and some methodological details should be clarified to improve rigor, reproducibility, and interpretive balance.

Specific Comments

Interpretation of the main conclusion should be softened.

The manuscript concludes that plan complexity is governed mainly by the optimization paradigm and patient anatomy rather than daily adaptive changes. While this interpretation is plausible, it is somewhat stronger than the presented evidence fully supports. The absence of statistically significant differences between offline and adapted plans does not necessarily prove that daily adaptation has little effect. A more cautious wording would be preferable, for example stating that no systematic shift in the selected PCMs was observed between offline and adapted plans.

Clarify the PSQA sampling strategy and its implications.

The study acknowledges that the PSQA subset was not randomized and that only five fractions per patient were selected. This should be emphasized more clearly in the Methods and Discussion because this design may affect the generalizability of the PCM–GPR association analysis. The current wording may lead readers to interpret these associations as representative of the entire adaptive treatment course.

Provide more reproducible detail on SPC calculations.

Because SPC is one of the practical outputs of the paper, the description of tolerance and action level derivation should be more explicit. The exact formula used for the action level should be reported, and the authors should clarify whether the same derivation was applied separately for global and local GPR and within each LOPO-CV iteration.

The phrase “high process stability” should be moderated.

The LOPO-CV results are encouraging, but this remains a single-institution cohort of 21 patients, subdivided into three target groups, with very small numbers in the prostatic bed subgroup. Therefore, describing the process as “highly stable” may be too strong. It would be more appropriate to describe the thresholds as showing promising or reasonable stability within this dataset.

The Discussion should further distinguish statistical significance from clinical utility.

The manuscript correctly notes that all PCMs were significantly associated with GPR, but marginal R² remained low. This point deserves even stronger emphasis because statistical significance alone may be overinterpreted by readers as evidence of strong predictive value. The real contribution of the PCMs here appears to be surveillance and risk flagging rather than robust prediction of delivery accuracy.

Subgroup-level interpretations should be presented more cautiously.

The authors already mention that the study includes a modest sample size and that the smallest subgroup was limited. This limitation should be more clearly linked to the uncertainty of subgroup ICC interpretations. It would help to explicitly state that subgroup-specific variance decomposition results are exploratory and should not be overgeneralized.

Minor Edits

1. There are several minor language and typographical issues that should be corrected, such as “tretament delivery” instead of “treatment delivery.”
2. The abstract sentence stating that the AI optimizer produced “systematically high-modulation plans” may be slightly interpretive. A more descriptive phrasing based on the observed metric values would be preferable.
3. Reference 13 appears incomplete in the bibliography and should be fully formatted if available.
4. The ethical and consent statements should be polished for grammar and journal style. The wording “only technical nature of the study” is awkward and should be revised.
5. Table 1 could be presented more clearly, as the mixture of fractionation schemes within target groups is somewhat difficult to follow.

Recommendation

Minor Revision

The manuscript is relevant, methodologically sound overall, and likely suitable for publication after revision. The requested changes are mainly related to interpretive caution, reproducibility of the SPC description, and minor presentation issues rather than fundamental scientific flaws. The study makes a useful contribution to the QA literature in online adaptive radiotherapy and will be of interest to medical physicists working with AI-assisted adaptive platforms.

 

Author Response

 

[COM] - 1) Interpretation of the main conclusion should be softened. The manuscript concludes that plan complexity is governed mainly by the optimization paradigm and patient anatomy rather than daily adaptive changes. While this interpretation is plausible, it is somewhat stronger than the presented evidence fully supports. The absence of statistically significant differences between offline and adapted plans does not necessarily prove that daily adaptation has little effect. A more cautious wording would be preferable, for example stating that no systematic shift in the selected PCMs was observed between offline and adapted plans.

[ANS] We agree and have adopted the Reviewer’s suggested wording to ensure a more cautious interpretation.

 

[COM] - 2) Clarify the PSQA sampling strategy and its implications. The study acknowledges that the PSQA subset was not randomized and that only five fractions per patient were selected. This should be emphasized more clearly in the Methods and Discussion because this design may affect the generalizability of the PCM–GPR association analysis. The current wording may lead readers to interpret these associations as representative of the entire adaptive treatment course.

[ANS] We included in the Supplementary Material the clear explanation of how the subset has been selected, and it has been built so that the subset is closely representative (in sa statistical sense) of the entire treatment course and of the entire plans cohort.

 

[COM] - 3) Provide more reproducible detail on SPC calculations. Because SPC is one of the practical outputs of the paper, the description of tolerance and action level derivation should be more explicit. The exact formula used for the action level should be reported, and the authors should clarify whether the same derivation was applied separately for global and local GPR and within each LOPO-CV iteration.

[ANS] SPC has been computed following word-by-word the method proposed in AAPM TG-218, a few details has been included in section 2.6.2

 

[COM] - 4) The phrase “high process stability” should be moderated. The LOPO-CV results are encouraging, but this remains a single-institution cohort of 21 patients, subdivided into three target groups, with very small numbers in the prostatic bed subgroup. Therefore, describing the process as “highly stable” may be too strong. It would be more appropriate to describe the thresholds as showing promising or reasonable stability within this dataset.

[ANS] We thank the Reviewer for this constructive observation. We have revised the Results to describe the stability of the SPC thresholds as 'reasonable' within the context of our dataset.

 

[COM] - 5) The Discussion should further distinguish statistical significance from clinical utility. The manuscript correctly notes that all PCMs were significantly associated with GPR, but marginal R² remained low. This point deserves even stronger emphasis because statistical significance alone may be overinterpreted by readers as evidence of strong predictive value. The real contribution of the PCMs here appears to be surveillance and risk flagging rather than robust prediction of delivery accuracy.

[ANS] We agree with the Reviewer and have addressed this point by clarifying the distinction between statistical significance and clinical utility. We fully revised the Discussion section to take this into account.

 

[COM] - 6) Subgroup-level interpretations should be presented more cautiously. The authors already mention that the study includes a modest sample size and that the smallest subgroup was limited. This limitation should be more clearly linked to the uncertainty of subgroup ICC interpretations. It would help to explicitly state that subgroup-specific variance decomposition results are exploratory and should not be overgeneralized.

[ANS] We amended the sentence to explicitly state that the ICC estimates for anatomical subgroups are exploratory due to the modest sample size.

 

[COM] - 7) Minor Edits

1. There are several minor language and typographical issues that should be corrected, such as “tretament delivery” instead of “treatment delivery.”
   [ANS] Done.
2. The abstract sentence stating that the AI optimizer produced “systematically high-modulation plans” may be slightly interpretive. A more descriptive phrasing based on the observed metric values would be preferable.

[ANS] We acknowledge the Reviewer’s point regarding the use of interpretive language. We have revised the.

 

1. Reference 13 appears incomplete in the bibliography and should be fully formatted if available.

[ANS] Thank you for pointing this out. Reference 13 corresponds to a manuscript that been recentrly accepted for publication in Medical Physics. We are uploading along the manuscript.

 

1. The ethical and consent statements should be polished for grammar and journal style. The wording “only technical nature of the study” is awkward and should be revised.

[ANS] We appreciate this feedback and have revised the Ethical and Informed Consent statements.

 

1. Table 1 could be presented more clearly, as the mixture of fractionation schemes within target groups is somewhat difficult to follow.

[ANS] We agree with the Reviewer. We have restructured Table 1 to improve clarity.

Reviewer 3 Report

Comments and Suggestions for Authors

Antonio Giuseppe Amico et al. presents an investigation into the role of plan complexity metrics (PCMs) in the quality assurance workflow for AI-assisted online adaptive radiotherapy (oART) for prostate cancer. The study address a critical gap in how to ensure delivery quality when per-fraction measurement-based QA is impractical. The methodology employs linear mixed-effects models, variance decomposition (ICC), and a robust leave-one-patient-out cross-validation (LOPO-CV) framework. The findings suggest that PCMs can serve as surveillance tools within an SPC framework. However, several issues should be addressed before final acceptance.

1.The specific PCM values (e.g., MU/cGy of 7.26) and the observed variability patterns are likely technique-specific. A more explicit statement that the results are primarily valid for this 9-field IMRT approach would improve clarity.

2.To enhance the clinical relevance of the findings, the authors should consider expanding the analysis to include DVH-based metrics for the fractions that underwent PSQA. For example, did the plans with lower GPR also show clinically meaningful deviations in target coverage (e.g., D98%) or OAR sparing (e.g., rectum V40Gy)? This would provide a stronger link between the complexity signals and potential patient impact.

3.The authors state that "no corresponding anomalies in anatomical metrics were observed." However, the data in Table 2 shows this patient was in the "high CTV volume region." To make this case stronger, the authors should provide a more detailed, quantitative profile of this patient compared to the rest of the cohort.

4.The discussion mentions the new complexity control feature in Ethos 2.0. This is purely speculative. Since the study data is from Ethos 1.0, the authors should avoid over-interpreting what a future version might do. Instead, they could frame this as a critical future research question: This reframes it from speculation to a testable hypothesis.

5.The authors use both "online adaptive radiotherapy (oART)" and "CBCT-guided oART." While not incorrect, standardizing to "oART" after the first mention would improve readability.

6.The term "IBRT" is used to describe the 9-field delivery. It would be clearer to state "9-field sliding-window IMRT" explicitly, as IBRT is a less common acronym.

7.Figure 3 is dense and makes it difficult to distinguish trends. Consider using different markers for the three anatomical targets (as done) and perhaps adding a simple regression line (with confidence bands) for the entire cohort to help the reader visualize the negative slope described in the text.

8.The p-values are marked with asterisks in the Table. Please define what * represents in the table caption (e.g., * p < 0.05 after FDR correction). The same applies to any other tables with significance markings.

9.Line 267: "exibited" should be "exhibited".

10.Line 282: In Table 3 (the first one), the p-value column header is slightly misaligned.

11.The sentence "attributed to the degeneracy of the optimization problem and a lack of proper limitation strategy." This phrase is a bit vague.

Author Response

[COM] - 1) The specific PCM values (e.g., MU/cGy of 7.26) and the observed variability patterns are likely technique-specific. A more explicit statement that the results are primarily valid for this 9-field IMRT approach would improve clarity.

[ANS]  We have updated the Discussion to explicitly state that our results are primarily representative of the 9-field static IMRT approach on the Ethos platform.

 

[COM] - 2) To enhance the clinical relevance of the findings, the authors should consider expanding the analysis to include DVH-based metrics for the fractions that underwent PSQA. For example, did the plans with lower GPR also show clinically meaningful deviations in target coverage (e.g., D98%) or OAR sparing (e.g., rectum V40Gy)? This would provide a stronger link between the complexity signals and potential patient impact.

[ANS]  We thank the Reviewer for this suggestion. We believe, however, that DVH-based analysis falls outside the scope of the present study, which was specifically designed to evaluate plan complexity as a process-level surveillance tool for delivery accuracy monitoring.

The GPR was chosen as the endpoint precisely because it reflects the fidelity of the delivered fluence to the planned one — a quantity that complexity metrics are mechanistically expected to influence. DVH-based endpoints conflate delivery accuracy with plan quality and daily anatomical variation, requiring a different study design.

That said, there are structural reasons to expect limited DVH variability in this context. The Ethos IOE operates as a satisficing optimizer: it enforces a predefined directive hierarchy and accepts the first compliant solution, which inherently constrains the DVH envelope across fractions. Moreover, our anatomical analysis showed high volumetric stability (PTV/CTV ICC: 0.88–0.96) and limited intra-patient variation in PTV–OAR overlap, which are the primary geometric drivers of DVH differences in the high-dose region.

[COM] - 3) The authors state that "no corresponding anomalies in anatomical metrics were observed." However, the data in Table 2 shows this patient was in the "high CTV volume region." To make this case stronger, the authors should provide a more detailed, quantitative profile of this patient compared to the rest of the cohort.

[ANS]  The reviewer is right, the patient lies in the “high CTV volume region” but with the revised analysis LMEM shows that even in case of large anatomic anomalies the PCM is loosely affected.

 

[COM] - 4) The discussion mentions the new complexity control feature in Ethos 2.0. This is purely speculative. Since the study data is from Ethos 1.0, the authors should avoid over-interpreting what a future version might do. Instead, they could frame this as a critical future research question: This reframes it from speculation to a testable hypothesis.

[ANS]  We agree with the Reviewer that our original phrasing overinterpreted the implications of the Ethos 2.0 complexity control feature, so it has been removed.

 

[COM] - 5) The authors use both "online adaptive radiotherapy (oART)" and "CBCT-guided oART." While not incorrect, standardizing to "oART" after the first mention would improve readability.

[ANS]  Done

 

[COM] - 6) The term "IBRT" is used to describe the 9-field delivery. It would be clearer to state "9-field sliding-window IMRT" explicitly, as IBRT is a less common acronym.

[ANS]  Amended the typo.

 

[COM] - 7)  Figure 3 is dense and makes it difficult to distinguish trends. Consider using different markers for the three anatomical targets (as done) and perhaps adding a simple regression line (with confidence bands) for the entire cohort to help the reader visualize the negative slope described in the text.

[ANS]  Figure 3 has been completely revised to clarify the information expressed therein.

 

[COM] - 8) The p-values are marked with asterisks in the Table. Please define what * represents in the table caption (e.g., * p < 0.05 after FDR correction). The same applies to any other tables with significance markings.
[ANS]  Done. We have updated the captions Table 3 to explicitly define the significance markings.

 

[COM] - 9) Line 267: "exibited" should be "exhibited".

[ANS]  Amended.

 

[COM] - 10) Line 282: In Table 3 (the first one), the p-value column header is slightly misaligned.

[ANS]  We noticed, we hope that the editorial office would revise and adjust the typesetting.

 

[COM] - 11) The sentence "attributed to the degeneracy of the optimization problem and a lack of proper limitation strategy." This phrase is a bit vague.

[ANS]  We have revised the sentence to be more specific and less vague.

Round 2

Reviewer 3 Report

Comments and Suggestions for Authors

No further comments