Assessment of Risk of Bias in Osteosarcoma and Ewing’s Sarcoma Randomized Controlled Trials: A Systematic Review

Aim: The aim of this study was to systematically assess the risk of bias in osteosarcoma and Ewing’s sarcoma (ES) randomized controlled trials (RCT) and to examine the relationships between bias and conflict of interest/industry sponsorship. Methods: An OVID-MEDLINE search was performed (1976–2019). Using the Cochrane Collaboration guidelines, two reviewers independently assessed the prevalence of risk of bias in different RCT design domains. The relationship between conflicts of interest and industry funding with the frequency of bias was examined. Results: 73 RCTs met inclusion criteria. Prevalence of low-risk bias domains was 47.3%, unclear-risk domains 47.8%, and 4.9% of the domains had a high-risk of bias. Domains with the highest risk of bias were blinding of participants/personnel and outcome assessors, followed by randomization and allocation concealment. Overtime, frequency of unclear-risk of bias domains decreased (χ2 = 5.32, p = 0.02), whilst low and high-risk domains increased (χ2 = 8.13, p = 0.004). Studies with conflicts of interest and industry sponsorships were 4.2 and 3.1 times more likely to have design domains with a high-risk of bias (p < 0.05). Conclusion: This study demonstrates that sources of potential bias are prevalent in both osteosarcoma and ES RCTs. Studies with financial conflicts of interest and industry sponsors were significantly more likely to have domains with a high-risk of bias. Improvements in reporting and adherence to proper methodology will reduce the risk of bias and improve the validity of the results of RCTs in osteosarcoma and ES.


Introduction
Randomized controlled trials (RCTs) are the cornerstone of modern evidence-based medicine. In the 1970s, the medical scientific community entered a new era, when the Food and Drug Administration (FDA) required pharmaceutical companies to submit RCTs for new drug applications [1]. Since that time, RCTs have become the gold-standard for assessing the safety and efficacy of experimental therapies and interventions [1]. Bias refers to systematic errors leading to deviation of results that can cause over-or under-estimation of the true effect of an intervention. Given this, bias has the potential to undermine RCT findings and may limit the utility of the trial in clinical practice.
The Cochrane Collaborations tool for assessing risk of bias in RCTs is the most widely utilized and recognized tool used to critically appraise RCTs (Table 1) [2]. This tool assesses the various methodological "domains" that are potential sources of bias in RCTs. Each domain is then rated based on its methodological quality as low risk, high risk or unclear risk of bias as per the guidelines set out by the Cochrane tool (Table 2). Unclear risk indicates "either a lack of information or uncertainty over the potential for bias" in a specific domain. According to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, a risk of bias assessment of included trials must be performed as part of systematic reviews [3]. This tool has been utilized widely across the medical literature and can provide an understanding of the quality of RCTs at both the individual level and an appraisal of the quality of the literature within a particular field as a whole. Risk of bias assessments have been undertaken in both surgical and oncology trials [4,5]. However, to our knowledge, no studies have evaluated the presence of bias in RCTs focused on osteosarcoma and Ewing's Sarcoma (ES).   [6,7].

Low Risk
Interpreted as potential bias unlikely to affect the results.

Unclear Risk
Interpreted as potential bias that raises some concerns about the results.

High Risk
Interpreted as potential bias that seriously reduces confidence in the results.
A source of bias not considered in the Cochrane Handbook is impact of industry funding and author conflicts of interest on methodologic quality and reported outcomes. RCTs are frequently supported by industry sponsors leading to potential conflicts of interest in these clinical studies [8]. It has been previously demonstrated that studies on the efficacy of drugs and devices, which are supported by manufacturing companies, report higher efficacy [9]. To our knowledge, the association between conflicts of interest and industry sponsorship with risk of bias in RCTs has not been previously assessed in orthopedic oncology studies.
RCTs for patients with primary bone tumors have unique challenges and a focused critical appraisal of this literature is warranted. Osteosarcoma and ES are rare entities, which has been shown to increase the bias, particularly in blinding and sample sizes [10,11]. The mainstay of definitive management of localized primary bone tumors includes surgical excision. Surgical trials pose many methodological challenges that are not always present in medical trials, which has the potential to introduce biases and reduce the validity of the results [12].
The primary objective of this study is to determine the prevalence of risk of bias in primary bone cancer RCTs. Secondarily, the impact of conflict of interest and industry sponsorship on risk of bias will be investigated. Finally, an evaluation of the change of the quality of RCTs over time will be undertaken.

Methods
A systematic review was performed on RCTs evaluating osteosarcoma and ES. This review adhered to the recommendations outlined in the PRIMSA and Cochrane Collaboration guidelines for the reporting of systematic reviews [3,6].

Eligibility Criteria
The inclusion and exclusion criteria were defined a priori. Inclusion criteria were as follows: (1) prospective study that had a parallel or cross-over longitudinal design, (2) studies examining a causal relationship between interventions and outcomes, (3) control or comparative group, and (4) presence of any number of included any cases of osteosarcoma and/or ES. Exclusion criteria included: (1) nonrandomized trials, (2) studies involving other diseases that could not be stratified by disease, (3) studies without online access (4) non-human trials, and (5) non-English studies.

Search Strategy
To systematically assess the risk of bias in published RCTs in osteosarcoma or ES, an Ovid MEDLINE (1946-2019) search was performed on 13 April 2020. Keywords included "Osteosarcoma" or "Ewing's Sarcoma". The search was limited by publications designated as "Randomized Controlled Trials".

Study Selection
The title and abstract of each article were screened and studies that did not meet the inclusion criteria were excluded. Full-text review was then performed for final assessment of study eligibility.

Data Extraction
Data extracted from the included studies was entered into a collaborative spreadsheet (Microsoft Excel V16.40). Study characteristics were recorded, including the publishing journal and impact factor, year of publication, conflicts of interests, and reported industry sponsorships. Trials were stratified by type: surgical vs. medical. Conflicts of interest were given one of three possible ratings: (1) No conflict of interest, when the study clearly stated that no conflict of interest was present, (2) Unclear conflict of interest, when there was no mention of the presence or absence of conflicting interests, (3) Conflict of interest, when mentioned. The same methodology was used for rating industry sponsorships.
Next, one reviewer (RK) assessed and appraised each study for its risk of bias using the Cochrane Handbook for Systematic Reviews of Interventions version 5.2 (2017)'s seven design domains. A senior author (JP) appraised the results to ensure accuracy and completeness. The domains are as follows: (1) Random sequence generation, (2) Allocation concealment, (3) Blinding of participants and personnel, (4) Blinding of outcome assessment, (5) Incomplete outcome data, (6) Selective outcome reporting, (7) Other potential threats to validity (Table 1). Each domain was designated with a low, unclear, or high risk of bias rating as per the Cochrane Handbook (Table 2) [7]. Disagreements were resolved by consensus.

Statistical Analysis
The presence, type, and rate of bias were summarized descriptively. The rate and type of bias present in each of the domains were also presented. Association between presence or absence of conflict of interests and industry sponsorships with the frequency of high risk domains was evaluated. The Fisher's exact test was used to measure the statistical significance in the analysis of contingency tables. An analysis of the risk of bias over time was performed to assess if bias has decreased over time. The frequency of low risk, unclear, and high risk of bias ratings were calculated for studies "prior to 1996", "1996-2000", "2001-2009", "2010-present" in correspondence with the Consolidated Standards of Reporting Trials (CONSORT) statement and its subsequent revisions published in 2001 and 2010 [13]. Pearson Chi-squared analyses were used to evaluate non-random statistical changes in the distribution of bias over time. Statistical significance was set at p-value < 0.05. GraphPad Prism V8.4.3 and Microsoft Excel V16.46 was used to prepare figures and perform the statistical analysis.

Results
The results of the search strategy are outlined in Figure 1. Overall, 164 studies were returned by the OVID MEDLINE search. After title and abstract screening, 91 studies did not meet inclusion criteria and were excluded. 73 studies met all inclusion criteria and moved forward for full text analysis and appraisal (Appendix A Table A1  Among the included studies, 94.5% (69/73) of trials were medical and 5.5% (4/73) were surgical trials. (Figure 2A). With 13 RCTs, the "Journal of Clinical Oncology" had the most included trials, followed by "International Journal of Radiation Oncology, Biology, Physics" with 5 RCTs, and "Cancer" with 4 RCTs. Other journals with frequent osteosarcoma and ES trials are displayed in Table 3. Stratification of overall risk of bias stratified by type of sarcoma (osteosarcoma vs. Ewing sarcoma), type of intervention (medical vs. surgical), and presence or absence of metastasis is displayed in Appendix A Table A2.   Table A3). The prevalence of risks of bias in different design domains are depicted in Figure 3. Across all of the appraised RCTs, the domain "selective outcome reporting" was found to have the lowest risk of bias. In contrast, for the blinding of participants and personnel domain no study adequately met low risk criteria and were either unclear risk (89.04%) or high risk (10.96%). Similarly, blinding of the outcome assessment had high rates of bias. For medical RCTs, 47.4% of the domains had an unclear risk of bias, 47.6% had a low risk of bias, and 5.0% of domains had a high risk of bias. For surgical RCTs, 53.6% of domains had an unclear risk of bias, 42.9% of domains had a low risk of bias, and 3.6% had a high risk of bias. Comparison of prevalence of risk of bias in surgical and medical RCTs are shown Figure 2B,C. Compared to medical RCTs, no significant difference in the risk of bias was found in surgical RCTs.
Conflicts of interest were present in 8.22% of studies, while 64.38% of studies did not have any reported conflict of interest. In the remainder of studies, it was not clearly stated whether a conflict of interest was present or not ( Figure 4). Industry sponsorship was present in 8.22% of RCTs, while 38.36% of studies had no sponsorship or sponsors were not from the industry. Compared to RCTs with no conflicts of interest, studies that had a conflict of interest were 4.16 times more likely to have a high risk domain (Range: [1.56 to 11.47], p = 0.01). Additionally, compared to studies without an industry sponsor, studies with a sponsor were 3.06 times more likely to have a high risk domain ([1.15 to 8.34], p = 0.03).
The frequency of the risk of bias over time is displayed in Figure 5. Frequency of unclear risk domains decreased, while both low risk and high risk domains increased over time. Chi-square test for trends showed a decrease in unclear risk domain (χ 2 = 5.32, p = 0.02) and an increase in high risk domains (χ 2 = 8.13, p = 0.004).

Discussion
In this systematic review of 73 RCTs, 47.8% of bias domains had an unclear risk of bias and 4.9% of domains had a high risk of bias. The domains with the most risk of bias were blinding of participants and personnel and blinding of outcome assessors, followed by randomization and allocation concealment. Studies were significantly more likely to have a high risk of bias domain if author conflicts of interest or industry funding were present. These findings are in line with other areas of the literature including orthopedic surgery, ophthalmology, and plastic surgery [14][15][16].
Randomization is the cornerstone of RCTs and ensures comparative groups have similar known and unknown prognostic factors so causality can be established [17]. It is critical for study designers to properly randomize studies and to conceal the sequence allocation to maintain the internal validity of the study results. In 71.2% of the appraised studies an unclear risk of bias was present for randomization and sequence generation. Another frequent source of risk of bias was from allocation sequence concealment. Allocation is related to, but different than randomization and refers to the practice of keeping researchers unaware of the sequence of randomization until the moment of assignment. When the allocation process is not concealed, investigators may (knowingly or unknowingly) choose to enroll only certain eligible patients in a trial [18]. This can lead to biased estimates of the treatment/intervention effect [19,20]. Allocation concealment is always possible in RCTs and should be explicitly stated to reduce selection bias [21].
Blinding is an important methodological feature in the design of RCTs and attempts to reduce performance and detection bias. In the current review, 89% of studies were found to have unclear risk of bias and the remaining 11% were found to be of high risk of bias with respect to blinding of both participants and personnel. This high rate of bias is concerning and likely multifactorial. Firstly, trialists often fail to adequately describe which individuals in the study are blinded [12]. Using the term "blinded" or "doubleblinded" is vague and fails to tell the reader which individuals involved in the study were blinded [22]. Secondly, 5.5% of the trials included surgical interventions, making blinding inherently more difficult for a number of reasons. Surgical interventions are often more difficult to blind than drugs trials, as placebos would require sham surgery [12]. Similarly, if the trial is comparing intraoperative methods, blinding of the surgeon becomes nearly impossible. However, despite these challenges, researchers should aim to blind as many of the involved individuals as possible and explicitly state how blinding is achieved in the methodology [23]. Although blinding of the treating surgeon and patient may be challenging or impossible, the blinding of outcome assessors and data analysts is usually possible and has been demonstrated successful in recent trials in the field [24,25]. Given that a lack of blinding is associated with more significant trial outcomes, it remains an important area to assess and scrutinize [23].
Financial conflicts of interest and industry sponsors are frequently found in clinical research, and are more common in oncology trials [8]. A review of both medical and surgical RCTs demonstrated that industry funded trials had significantly higher rates of positive trials in favor of the new industry product [26]. However, it is important to note that in the setting of rare tumors, alternative statistical methods are often utilized and these results may not be generalizable to this population. Adding further uncertainty, RCTs inconsistently report the presence or absence of conflicting interests and industry sponsors [27]. Our review is in line with these findings as studies with conflicts of interests were over four times more likely to have a high risk domain and if an industry sponsor was present the studies were over three times more likely to have a high risk domain (p < 0.05). Additionally, a limitation to the quality assessment of the trials was that 27.4% of studies did not discuss the presence or absence of conflicts of interests and over half of the trials did not report the source of their funding. Improved approaches are required for the identification and verification of conflicts of interest and sources of sponsorship [8].
The frequency of domains with unclear bias decreased over time. This is likely partially attributed to the widespread adoption of the CONSORT statement by both journals and researchers [28]. The CONSORT statement was first introduced in 1996 and trials published prior to this should be viewed through a different lens given the lack of reporting standards prior to this. This is encouraging and indicates improved reporting given that a large proportion of the unclear bias category is due to lack of reporting and not frank concerns with study methodology. However, given that there was an increased proportion in high risk domains, this review demonstrates the need for improvements in the design and implementation of bone tumor RCTs of Ewing's and osteosarcoma.
It is also important to note the significant methodologic challenges associated with the design and implementation of trials in rare diseases such as OS and ES. Trials involving patients with rare tumors face recruitment challenges and long follow-up periods. While methodological bias is present in the literature, it should be noted that trials have led to significant advances in the treatment of bone and soft tissue sarcomas. Prior to the employment of adjuvant and neoadjuvant therapies and novel limb salvage surgeries, the five-year survival of ES and OS used to be 20-30% [29][30][31][32]. After the introduction of these medical and surgical advances, the current five-year survival of ES and OS are 60-70%. Therefore, while we identified areas of potential design improvement, the great strides that have been made in this field are commendable [29][30][31][32].
This review is strengthened by the novelty of its findings. To our knowledge, this study is the first to assess and the risk of bias of RCTs in osteosarcoma and ES. Secondly, this review was comprehensive in nature, including a large number of RCTs, increasing the robustness of the findings. There are several limitations that must be considered. A limitation of the risk of bias assessment is that it is unable to distinguish between how the trial was conducted and how it was reported. A trial can be methodologically sound but poorly reported and vice-versa. Although the most widely adopted, the Cochrane risk of bias tool is one of many ways clinicians can critically appraise randomized controlled trials [33]. Finally, a potential limitation of the current review is the utility of the Cochrane risk of bias tool. Although this tool has been utilized widely and validated in common diseases, it may be less applicable in rare diseases.

Conclusions
Sources of potential bias are prevalent among osteosarcoma and ES RCTs, particular in the domains of randomization, allocation concealment, and blinding. Conflicts of interest/industry sponsors were shown to increase the likelihood of having high risk domains. Improvements in reporting and adherence to proper methodology, will reduce the risk of bias and improve the validity of the results of RCTs in osteosarcoma and ES.

Conflicts of Interest:
The authors declare no conflict of interest.