Evaluation of Patient-Reported Outcome Differences by Radiotherapy Techniques for Bone Metastases in A Population-Based Healthcare System

Round 1

Reviewer 1 Report

The authors report pain-related PROs before and after radiation therapy for bone metastases. The patient population data is centralized in a provincial database. The PROs were prospectively collected. The authors report that the use of advanced RT did not significantly improve pain response compared to simple RT; however, the use of advanced RT ‘increased significantly’.

This is an important area of research and the prospective nature of data collection, as well as the centralized data, are strengths. However, a tremendous amount of information is missing from the methodology section. Therefore the study itself, and the validity of the results, cannot be assessed.

Please see specific comments below:

Title

• ‘Non-incentivized’ is distracting. I suggest removing.

Abstract

• Does the BPI include pain interference? It is not clear in the abstract
• Please provide the statistical approach to the many p-values
• What is meant by ‘response’? Do you mean ‘pain response’?

Introduction

• Please expand on the findings of the published RCTs on advanced RT techniques. What were the conclusions and how did they lead to this study?
•  
• Is it possible that advanced RT is a better oncologic choice? Where does that issue fall in the scope of the current study?

Methods

The methods are much too sparse to assess. There is no description of the patient population: how many were included and excluded due to lack of PRO data?

There is no description of the PROs and the specific information they capture, how and when they are completed, if they are specific to the bone metastasis site.

There are also issues with selection bias as to why some patients received standard and others received advanced RT.

How was ‘partial’ and ‘complete’ response defined?

Is there a minimally important difference (MID) for the PROs used?

Was a sample size calculation/power analysis done to confirm study power?

This section needs significant expansion so that the methodology can be adequately assessed.

Results

Table 1: please annotate the statistical test used to calculate the p-values

The number of patients that underwent advanced RT is very small compared to simple RT. The groups also appear to be quite different. There is significant risk for selection bias affecting the outcomes data.

Increased use of advanced RT: The abstract indicates it was ‘significant’, but described as ‘mild’ in the Results section. Please be consistent.

Overall, the Results section cannot be adequately assessed given the lack of information on the methodology to obtain the results.

Discussion

What does the current study add given that  there has already been RCTs? Please elaborate

Author Response

Response to reviewer is in tracked changes and italicized

The authors report pain-related PROs before and after radiation therapy for bone metastases. The patient population data is centralized in a provincial database. The PROs were prospectively collected. The authors report that the use of advanced RT did not significantly improve pain response compared to simple RT; however, the use of advanced RT ‘increased significantly’.

This is an important area of research and the prospective nature of data collection, as well as the centralized data, are strengths. However, a tremendous amount of information is missing from the methodology section. Therefore the study itself, and the validity of the results, cannot be assessed.

Thank you for the comments. We are presenting this as a short communication, which limits the ability to present the methodology in detail, but we have attempted to improve within space constraints

Please see specific comments below:

Title 

• ‘Non-incentivized’ is distracting. I suggest removing. Removed

Abstract

• Does the BPI include pain interference? It is not clear in the abstract: it does. The abstract does not have space to include all details, but the BPI is a well supported tool for research on bone metastases, and most readers will be aware of its contents. Those who are not, can easily look up the BPI and we attempted to make finding these details easier.
•  
• Please provide the statistical approach to the many p-values: It is hypothesis generating. We believe that only if you are trying to state there is a significant response, would you need to qualify that p value must be much lower than 0.05 because of multiple hypothesis testing. Given we did not find a difference, the non-significance still applies.
•  
• What is meant by ‘response’? Do you mean ‘pain response’? fixed.

Introduction

• Please expand on the findings of the published RCTs on advanced RT techniques. What were the conclusions and how did they lead to this study? This short communication limits the ability to expand on those well known trials, but we have added 3 sentences
• Is it possible that advanced RT is a better oncologic choice? Where does that issue fall in the scope of the current study? This may be true (not yet proven in phase III trials which are underway, two of which we are leading), but this issue looks at a much more broad population with more generalizable results, as now added to intro.

Methods

The methods are much too sparse to assess. There is no description of the patient population: how many were included and excluded due to lack of PRO data? This data is not readily available unfortunately.

There is no description of the PROs and the specific information they capture, how and when they are completed, if they are specific to the bone metastasis site. The BPI is well known to bone metastases researchers and we have tried to expand on this, but are limited by this short communication.

There are also issues with selection bias as to why some patients received standard and others received advanced RT. Agreed, and now added to discussion.

How was ‘partial’ and ‘complete’ response defined? International consensus definitions that are now referenced.

Is there a minimally important difference (MID) for the PROs used? Intenational consensus

Was a sample size calculation/power analysis done to confirm study power? Unfortunately not.

This section needs significant expansion so that the methodology can be adequately assessed.

Results

Table 1: please annotate the statistical test used to calculate the p-values. Added to methodology

The number of patients that underwent advanced RT is very small compared to simple RT. The groups also appear to be quite different. There is significant risk for selection bias affecting the outcomes data. Agree. Added to discussion / limitations

Increased use of advanced RT: The abstract indicates it was ‘significant’, but described as ‘mild’ in the Results section. Please be consistent. corrected

Overall, the Results section cannot be adequately assessed given the lack of information on the methodology to obtain the results. Attempted to expand within limits of a short communication.

Discussion

What does the current study add given that  there has already been RCTs? Please elaborate. This is a population-based analysis that complements the RCTs, and includes a much broader group. SC.24 is only spine mets, for example, and SABR-COMET is limited to the rare group of oligomets. Added to discussion

Reviewer 2 Report

In this retrospective analysis conducted by Olson and colleagues, the authors compared simple vs. more advanced RT techniques such as IMRT and 3D-CRT in terms of pain response and pain-related QOL in patients with metastatic bone lesions treated with palliative RT. The authors did not find a significant difference in pain response or pain-related QOL between the two treatment groups. The authors are uniquely poised to conduct such a study in a healthcare system with little financial incentive to choose more advanced RT techniques. The authors should be commended for conducting such a study which is of great clinical interest especially in the setting of increased healthcare cost. Sometimes less is more. I do have a few comments and suggestions that I believe the manuscript may benefit from.

Major comments:

1. Radiation dose was not documented or analyzed in this paper. The authors should report BED or EQD2 for the two groups and incorporate dose as a variable in the MVA.

2. The treatment region in the two groups are not well balanced (table 1, p<0.001). For example, the advanced group had more skull lesions, which are associated with lower PR and CR compared to lesions in other regions. Conceivably, there is still considerable selection bias in terms of which RT modality to use based on clinical and other factors (although I understand it is conducted in a non-incentivized health care system). This should be mitigated aggressively with more sophisticated statistical approaches. For example, 1:1 propensity score matching or IPTW should be used.

3. There was no information regarding the timing of when the BPI PRO questionnaires were administered. The authors should report the mean, median, range and IQR of the days post RT that the BPI was administered, in each group.

4. Related to point 2, the timing of the BPI is important as pain flare is a known phenomenon. If the timing is significantly different between the groups, it may introduce additional bias. The authors should comment on the rate of pain flare between the two groups if such data is available.

5. Besides pain control per se, more conformal techniques can decrease the radiation dose to surrounding OARs. For example, pts with lumbar spine mets of mets in the pelvis may experience more diarrhea and bladder irritation when treated with parallel-opposed field than IMRT. This is an important factor affecting pt's QOL but is not captured in BPI. In fact one may argue that we do not expect an increased pain control with more advanced RT techniques. The advantage lies in the smaller margin and lower dose to surrounding OARs. The authors only briefly touched on this in the discussion and should expand on this.

Minor points:

1. Indeed the SBRT was under-represented in the advanced RT technique group. The author should make it very explicit in the abstract that the conclusion does not apply to SBRT.
2. Table 5: "adavnced vs simplae RT" typo. Please correct.
3. Figure 1. Please break down the trend into 3D-CRT, IMRT and SBRT categories.

Author Response

Major comments:

1. Radiation dose was not documented or analyzed in this paper. The authors should report BED or EQD2 for the two groups and incorporate dose as a variable in the MVA.

In BC, the vast majority (>98%) of simple RT received 20/5 or 8/1, and therefore dose and technique are highly co-linear and couldn’t be put into the same multivariable model

1. The treatment region in the two groups are not well balanced (table 1, p<0.001). For example, the advanced group had more skull lesions, which are associated with lower PR and CR compared to lesions in other regions. Conceivably, there is still considerable selection bias in terms of which RT modality to use based on clinical and other factors (although I understand it is conducted in a non-incentivized health care system). This should be mitigated aggressively with more sophisticated statistical approaches. For example, 1:1 propensity score matching or IPTW should be used.

Thank you for the suggestion. Our numbers are too small for this technique but agree it would be worthwhile when we reanalyze in a few years when we anticipate advanced techniques will be much more common. Limitation of selection bias added to discussion.

1. There was no information regarding the timing of when the BPI PRO questionnaires were administered. The authors should report the mean, median, range and IQR of the days post RT that the BPI was administered, in each group. This data is not actually available in our prospective database. They are most commonly performed pre-RT and 6 weeks post RT. Added to methods.

1. Related to point 2, the timing of the BPI is important as pain flare is a known phenomenon. If the timing is significantly different between the groups, it may introduce additional bias. The authors should comment on the rate of pain flare between the two groups if such data is available. Data not available. Timing of BPI added to methods.

1. Besides pain control per se, more conformal techniques can decrease the radiation dose to surrounding OARs. For example, pts with lumbar spine mets of mets in the pelvis may experience more diarrhea and bladder irritation when treated with parallel-opposed field than IMRT. This is an important factor affecting pt's QOL but is not captured in BPI. In fact one may argue that we do not expect an increased pain control with more advanced RT techniques. The advantage lies in the smaller margin and lower dose to surrounding OARs. The authors only briefly touched on this in the discussion and should expand on this. expanded.

Minor points:

1. Indeed the SBRT was under-represented in the advanced RT technique group. The author should make it very explicit in the abstract that the conclusion does not apply to SBRT. added
2. Table 5: "adavnced vs simplae RT" typo. Please correct. corrected
3. Figure 1. Please break down the trend into 3D-CRT, IMRT and SBRT categories. added

Round 2

Reviewer 1 Report

Thank-you for addressing my comments. The methods are still very sparse. However, this apparently fits with the format of a 'communication.'. Is there a reason why a full study was not submitted? Finally, the deluge of p-values is inappropriate. The authors acknowledge that the numbers are small in most groups, and make the argument that since the p-values are not significant, study power is not necessary. I suggest removing the p-values (reduce the p-hacking) and simply present the findings descriptively. 

Author Response

p values removed from Table 5, and presented as descriptive only. corresponding p values removed from abstract.