Review Reports

Round 1

Reviewer 1 Report

The authors performed a meta-analysis of RCTs and observational studies that have compared SCRT with or without consolidation chemotherapy versus standard CRT. Overall it is an interesting topic, and a hot topic at present.

However, there are several aspects that I find questionable.

-The authors do not sufficiently emphasise the limitations of introducing non-randomised studies into the systematic review/meta-analysis, nor are they aware of the risk of bias in some RCTs. For example, Bahadoer's paper (RCT RAPIDO) is described as "low risk of bias". This is questionable. In the RCT RAPIDO only half of the subjects in the control group received adjuvant chemotherapy. The reason for this was that the authors approached a pragmatic design due to the fact that in many European centres adjuvant chemotherapy is not standard. The design of the RCT RAPIDO involves the assumption that adjuvant chemotherapy has no effect in localised rectal cancer. However, the null effect is only a concrete value within an uncertain parameter. Any deviation from this null effect assumption may bias the results, as all patients in the experimental arm benefited from chemotherapy in the form of TNT, while only half of the patients in the control arm benefited from adjuvant chemotherapy. Although the evidence is limited, a meta-analysis suggests that the most plausible hypothesis is that adjuvant chemotherapy is effective in rectal cancer.  

Zhao, L., et al. "Oxaliplatin/fluorouracil‐based adjuvant chemotherapy for locally advanced rectal cancer after neoadjuvant chemoradiotherapy and surgery: a systematic review and meta‐analysis of randomized controlled trials." Colorectal Disease 18.8 (2016): 763-772.

The authors of the RAPIDO RCT perform a sensitivity analysis suggesting that there is no heterogeneity within the control arm according to hospital policy on adjuvant. However, the update of these data presented at ESSO#40 shows that patients who did not receive chemotherapy had a statistically significant better baseline prognosis. This shows that the decision not to give adjuvant chemotherapy is a non-randomised component affected by confounders acting on hospital policy, or decision making based on the results of radiotherapy/surgery.

See: Bahadoer RR, Dijkstra EA. Patterns of locoregional failure and distant metastases after five years in patients treated for locally advanced rectal cancer in the RAPIDO trial. In: 40th Congress of the European Society of Surgical Oncology. Lisboa; 2020

Therefore, the decision not to make the use of adjuvant chemotherapy mandatory in the control arm may have biased the results in favour of the experimental arm. This aspect should be commented on in the bias assessment, and counted as a limitation. The estimation of DFS is at high risk of being biased. In fact, the 5-year update in ESSO #40 shows less impressive results, with a significant increase in the risk of local recurrence.

-Of the articles included in the DFS estimate, the RCT RAPIDO is possibly biased. The other articles by Beppu, Chung and Thakur are observational studies; Markovina is a sub-analysis comparing the cohort of a phase II trial with a matched group of patients who received CRT. It is therefore another non-randomised study. Cisel is the only true RCT in the meta-analysis without risk of bias for the DFS endpoint, and it shows no difference. Interestingly, the authors give preoperative chemotherapy in the standard arm, but not adjuvant chemotherapy.

- In my opinion, all these limitations, including the limitations of existing RCTs, and non-randomised studies, need to be clearly and extensively emphasised.

-The recently published RCT Stellar, an interesting contribution, is not included.

-Equating hazard ratios with risk ratios here is questionable. For example, figure 3 in Kairevičė 2017 shows heavy censoring for the Kaplan-Meier estimator. This is the only RCT comparing the effect of administering SCRT without chemotherapy vs CRT, therefore, the conclusions derive from this study. The authors of this meta-analysis calculate a risk ratio of 0.86 (0.68-1.08) from the event-to-patient ratio, ignoring the time-to-event analysis. According to this analysis, the signal is weak and does not allow ruling out the null effect. However, the correct analysis in the original article shows a very different story: "Overall survival (OS) at 5 years was 79% in CRT (n = 72) and 62% in SCRT (n = 68) group, and this difference was also statistically significant (P = 0.015; HR = 2.05; 95% CI, 1.13-3.70; P = 0.017)". The discrepancy shows very clearly why risk ratios are not hazard ratios, and event/patient ratios are not fixed-point survival estimates. Survival analysis cannot be replaced by raw fractions without distorting the result. From a clinical point of view this issue is not trivial because it leads to reinterpreting the message of the original paper, which concluded exactly the opposite, on a clinically relevant issue.

-The same can be said for the estimation of local recurrences. The meta-analysis concludes that there is no difference in local recurrences. The Polish study carries the most weight and shows no difference. However, the 5-year update of the RAPIDO trial revealed an increase in the risk of local relapse with a HR 1.95 (95% CI, 1.18-3.23); p-value=0.009. The discrepancy might have been even more striking if a higher proportion of patients in the control arm of RAPIDO had received adjuvant chemotherapy.

Bahadoer RR, Dijkstra EA. Patterns of locoregional failure and distant metastases after five years in patients treated for locally advanced rectal cancer in the RAPIDO trial. In: 40th Congress of the European Society of Surgical Oncology. Lisboa; 2020

The discrepancy between the RCT RAPIDO and the Polish study may be the more advanced stage, with a higher hazard rate in the former. The rest are observational studies. With these data, I am not comfortable concluding that there is no difference in local recurrence, when the updated results of RCT RAPIDO are compatible with a threefold risk (according to the upper limit of the confidence interval), with a different population, possibly with a higher risk than Cisel et al, and with the rest of the observational studies. This is the typical "absence of evidence is no evidence of absence" fallacy. That could be important for patients with high risk, bulky tumors, as shown in the RAPIDO trial.

- These problems prevent proper calibration of the evidence. For example, in the discussion, the authors argue that the better response with SCRT followed by chemotherapy might allow for a higher rate of sphincter-sparing surgery, including complete organ preservation (line 298). However, if one looks at the results of the Rapido RCT, there does not seem to be a correlation between pathological complete response, and local recurrences (see above). It is difficult to propose organ preservation with a modality where you cannot statistically rule out that the risk of local recurrence is tripled. 

- The authors suggest that the addition of oxaliplatin to fluoropyrimidine is useful in the perioperative setting. Well, the data with CRT do not say exactly this: Hüttner FJ, Probst P, Kalkum E, Hackbusch M, Jensen K, Ulrich A, et al. Addition of platinum derivatives to fluoropyrimidine-based neoadjuvant chemoradiotherapy for stage II/III rectal cancer: systematic review and meta-analysis. JNCI J Natl Cancer Inst. 2019;111(9):887-902. 

- This is a problem because the two top-ranked RCTs (Cisel and RAPIDO) administer oxaliplatin-based chemotherapy as TNT, but then do not administer adjuvant chemotherapy (47% in RAPIDO), favouring the experimental arm in both cases.

-The authors do not devote much space to the generalisability of the findings in view of the heterogeneity of rectal cancer. In particular, I do not believe that the results can be translated equally to bulky or non-bulky tumours, and I have doubts also for low rectal tumours.

- Discussion. Overemphasis on the advantages of SCRT during the pandemic. This seems relative to me. The most important thing for the patient is disease control, and reduction of recurrences. Covid-19 is likely to become less and less relevant in the future.

- Conclusion: I think it is unbalanced, not taking into account the limited evidence and risks of biases. I agree that survival does not change, and possibly metastases are reduced by the introduction of TNT. However, the results are basically derived from the RAPIDO RCT which had a crucial methodological problem that is not mentioned in the manuscript (see above). The meta-analysis incorporates the 3-year RAPIDO local recurrence data, but not the 5-year update. In such a case the results would perhaps be consistent with increased risk of local recurrence in more locally advanced tumours. I am concerned that this modality is selected in patients with larger tumours at higher risk of persistence/recurrence. In a scenario of similar survival, in the end the net result could be fewer metastases and more local relapses, without being very clear what patients really gain. I may agree more or less with the authors' opinion, but it seems to me that the current data are not robust for such strong conclusions.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

This paper is a review/meta-analysis of 15 included studies on neoadjuvant treatment for locally advanced rectal cancer comparing short course radiotherapy (SCRT) with (long course) chemoradiotherapy (CRT).

The aim was to demonstrate whether short course followed by consolidation chemotherapy is superior or at least equal to long course which includes always a chemotherapy regimen. Looking on the included studies only nine studies were randomized controlled trials of which three had no consolidation chemotherapy after short course radiation. Another two included retrospective trials also had no chemotherapy regimen in the short course treatment arm. As it is written at the end of the discussion “… this meta-analysis provides an overview of the current evidence regarding the use of SCRT with delayed surgery and consolidation chemotherapy”.

The sophisticated question in the title  whether “short course radiotherapy followed by consolidation chemotherapy before surgery (can) be a new standard for treating locally advanced cancer” remains unanswered, but it is unquestionable an alternative treatment regimen.

Specific comments:

p4 table1: in the text “CRT” is used as abbreviation, in all tables “CCRT”

p7 line178: “tumor downstaging” is not defined

p9 line206: “LR” for local recurrence

p10 fig.6/7: in the forest plot “favors SCRT” is on the left side, in fig. 3/4/5/8/9/10/11 on the right side

suppl.: in fig. S2/S3/S4/S5/S7/S8/S9/S10 just the same

p14 line330: concerning TNT the ongoing ACO/ARO/AIO-18.1 phase III trial could be included for discussion as it compares SCRT with consolidation chemotherapy and CRT followed by consolidation chemotherapy.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

The authors have made substantial modifications to the manuscript in line with the previous revision. In my opinion the text is overall interesting, and condenses the available information on a relevant, topical issue. However, my impression is that the text could be further clarified at some points. I therefore suggest the following minor changes:

1. The authors present their results both as hazard ratios and risk ratios. The reason is that some studies do not report hazard ratios, and therefore the authors try to calculate risk ratios from fixed-point survival rates. The problem is that in some passages this makes readability very difficult. For example, the abstract states that the pCR rate was higher with SCRT + CCT than that with CRT (RR: 1.60; 95% CI: 1.35-1.91; p<0.01). However, the text goes on to state that SCRT + CCT provided a higher ypN0 response (RR: 0.85; 95% CI: 0.75-0.98; p=0.02). It is unclear why at one site the increased effect translates as an RR >1 while the opposite (RR<1) is also true later on. The problem is further exacerbated because RR and HR are interpreted in opposite ways (RR>1 indicates an increase in survival rates, while the opposite is true for HR). This is sometimes difficult to understand. For example, Kairevičė 2017 report a hazard ratio of around HR = 2.05; 95% CI, 1.13-3.70 while the authors convert this to an RR 0.86 (0.68-1.08). The authors have simply replied that RRs are an acceptable measure in meta-analyses. Well, maybe, but in this case it is not always well understood. Moreover, in cases where both metrics (e.g., for overall survival analysis) are used, the interpretation changes drastically. For example, figure 4 shows that SC-RT followed by chemotherapy improves OS (RR 1.05; 95% CI: 1.00-1.11; p=0.04). However, estimation based on hazard ratios shows a slightly opposite conclusion (HR: 0.85; 95% CI: 214 0.71-1.01; p=0.07). In the Stellar trial there is a benefit in OS: hazard ratio, 0.883; one-sided 95% CI, not applicable to 1.11; P < .001. However, the authors use a risk ratio of 1.15. The statistics section is sparse about all this. The authors could improve readability if they clarify (briefly and without excessive complexity) the interpretation of both measures (hazard ratios vs. risk ratios): How are values greater or less than 1 interpreted in both cases? What different information do the two measures convey? etc.
2. In the discussion I think it is worth clarifying that the estimation using HRs is based on results from RCTs, whereas the meta-analysis based on RRs contains data from observational studies, with a higher risk of bias.
3. It is not clear to me why the authors refer to the meta-analysis based on hazard ratios as "Forest plot of the cumulative overall survival (OS)". (line 211 and figure at bottom of page). By cumulative, most readers will probably mean a survival rate or a probability of survival over time, as estimated by Kaplan-Meier on the "cumulative scale". While hazard ratios only relate to survival probabilities via the exponential survival function, I find the name cumulative overall survival confusing. Meta-analysis based on RRs or HRs might suffice.
4. This article supports the method commentary (line 176), about the concerns in the RCT RAPIDO and can be referenced: https://pubmed.ncbi.nlm.nih.gov/35462008/
5. On line 334 they say that in the Polish-2 study patients in the control arm did not receive adjuvant chemotherapy. This is not entirely true. In fact, postoperative chemotherapy was at the discretion of the investigators as in the RAPIDO trial and was received by 15-11% (Bujko K, et al: Ann Oncol 27:834-842).
6. The authors continually equate the absence of statistically significant differences with the absence of association. This is an error that has not been corrected. For example, the authors state that there is no difference in toxicity between the arms with RR: 1.30; 95% CI: 0.91-1.85; p=0.15. These passages should be qualified by clarifying that there is insufficient evidence to contradict the null hypothesis (of equality), but not that the results are truly similar. This is made particularly difficult to understand by contradictions within the study. For example, the discussion begins: "This meta-analysis demonstrated that treating LARC with SCRT followed by consolidation chemotherapy provides better pCR rates, results in more ypN0 status with positive lymph nodes, increases sphincter preservation surgery, and improves the OS and DFS compared to conventional CRT treatment". However, the discussion concludes that: "Moreover, SCRT followed by consolidation chemotherapy can improve tumour downstaging, induce more pCR, and help in sphincter-preservation surgery. However, DFS, OS, toxicity, postoperative complications, and treatment compliance of the individuals were similar to those who underwent conventional CRT." Where do we stand? Does it or does it not improve survival?
7. From line 320 onwards the discussion becomes dense and difficult to follow.
8. Moreover, the completion rate of scheduled dose of adjuvant chemotherapy was reported to be only 47-48%. (line 336) Which trial are you referring to? This deviation from the intended interventions caused a bias in the study in the control arm. I'm not so sure that these are true deviations, because the design left the decision up to the researcher to decide.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

The paper has been well revised and extended. The new title „Neoadjuvant short-course radiotherapy followed by consolidation chemotherapy before surgery for treating locally advanced rectal cancer: A systematic review and meta-analysis“ has been well choosen. Nevertheless, several issues should be further revised.

The statements in the abstract, at the beginning of the discussions (lines 280-289) and in the conclusions should be adjusted and equalized. They should focus only on the statistical significant findings of this paper. „Trends“ are non-significant findings and should only be illuminated in the discussion as their interpretation is always subjective.

Specific comments

Figures 3 – 11

„CCRT“ is still used in the figures instead of  „CRT“. An adjustment would be desirable.

Lines 281

„… results in more ypN0 status with positive lymph nodes“: 

This statement should be modified for better understanding.

Line282

„and improves the OS and DFS compared to conventional CRT treatment“:

This statement should be modified as the authors write in lines 27/28 „There was no difference in the OS and DFS between the SCRT + CCT and CRT arms; however, a trend favoring SCRT + CCT was observed (OS: HR: 0.85, p=0.07; DFS: 28 HR: 0.88, p=0.08)“ and in lines 465/466 „DFS, OS, toxicity, postoperative complications, and treatment compliance of the individuals were similar to those who underwent conventional CRT.“

Line284

„a trend of reduced distant metastasis were observed“

The question is whether we can talk about a „trend“ at all in this context. The authors write in lines 221/223 „Regarding the incidences of DM, there was no statistical difference; however, a trend favors SCRT + CCT over conventional CRT was found (RR: 0.86; 95% CI: 222 0.68–1.07; p=0.18) (Supplementary Figure 5).“ Like written above it would be desirable to focus only on the statistical significant findings. „Trends“ can be detected in most forest plots in some form.

Line463

„SCRT followed by consolidation chemotherapy can improve tumor downstaging, induce more pCR, and help in sphincter-preservation surgery.“

As the autors state in line 451 „clinical heterogeneity remained, as tumor location varied across each study“ it might not be justified to include this statement about sphincter preservation in the conclusions. The phrasing  „SCRT … help in sphincter-preservation surgery“ is choosen unfavourable,  surgical skills are a more important factor concerning  this.

Author Response

Please see the attachment.

Author Response File: Author Response.docx