Rapid Review of Real-World Cost-Effectiveness Analyses of Cancer Interventions in Canada

Cost-effectiveness analysis (CE Analysis) provides evidence about the incremental gains in patient outcomes costs from new treatments and interventions in cancer care. The utilization of “real-world” data allows these analyses to better reflect differences in costs and effects for actual patient populations with comorbidities and a range of ages as opposed to randomized controlled trials, which use a restricted population. This rapid review was done through PubMed and Google Scholar in July 2022. Relevant articles were summarized and data extracted to summarize changes in costs (in 2022 CAD) and effectiveness in cancer care once funded by the Canadian government payer system. We conducted statistical analyses to examine the differences between means and medians of costs, effects, and incremental cost effectiveness ratios (ICERs). Twenty-two studies were selected for review. Of those, the majority performed a CE Analysis on cancer drugs. Real-world cancer drug studies had significantly higher costs and effects than non-drug therapies. Studies that utilized a model to project longer time-horizons saw significantly smaller ICER values for the treatments they examined. Further, differences in drug costs increased over time. This review highlights the importance of performing real-world CE Analysis on cancer treatments to better understand their costs and impacts on a general patient population.


Introduction
Economic evaluation produces important results that inform healthcare funding decisions in Canada. Not only do economic evaluations like cost-effectiveness analysis (CE Analysis) provide evidence about the costs and effects of new treatments and interventions for healthcare payers, they also provide patients and providers with comprehensive evidence in order to inform their decisions. This is especially important when considering expensive, new cancer treatments and interventions in Canada. For example, in Canada's public health care system, cancer drugs are reviewed by Heath Canada for safety and efficacy. The pan-Canadian Oncology Drug Review (pCODR), through its expert review committee, makes reimbursement recommendations for oncology pharmaceuticals to the participating federal, provincial, and territorial publicly funded drug programs. The pCODR also makes recommendations related to the identification, evaluation, and promotion of responsible drug prescribing and use in Canada [1]. The expert review committee uses a deliberative framework to ensure the consistency and transparency of its cancer drug review process, and this framework includes cost-effectiveness as key component [1]. Curr. Oncol. 2022, 29 7286 The trial-based evidence that informs the economic models used to estimate costeffectiveness often comes from brief "controlled" trials with rigid study protocols, involving special oncologists and some of their unique patients. Use of these data is the cornerstone of evidence-based recommendations before a cancer drug is funded. After a drug is funded, however, data exist about the real-world costs and outcomes, since healthcare payers generally keep track of their beneficiaries' costs and mortality status. Real-world CE Analysis provides information about a new drugs' extra cost (∆C) and extra effect (∆E) after it has been funded and used by real clinicians and their potentially less healthy, less young, and less adherent patients [2]. In this paper, we review the Canadian literature on the real-world CE Analysis of recent cancer treatments and interventions. We summarize our findings and provide suggestions for next steps.

Background
Cost-effectiveness analysis (CE Analysis) studying a new treatment or intervention's extra cost (∆C) and extra effect (∆E) can be conducted by analyzing a dataset or by synthesizing a body of evidence using a decision analytic model. Before a drug is funded, often phase III randomized controlled trials (RCTs) provide this type of data. Besides the concerns related to generalizability of the clinicians and patients, RCTs may have challenges related to timing. For example, Haslam et al. (2022) found that the median duration of treatment in studies initially testing a drug was 6.0 months (range: 2.2-12.7 months), whereas the median duration of treatment when the same drug was used as a comparator was 4.9 months (range: 1.7-12.0 months) [3]. Of course, treatment duration outside of an RCT may differ. In addition, Del Paggio et al. (2021) found median follow-up has decreased over three time periods from 47 months (1995)(1996)(1997)(1998)(1999)(2000)(2001)(2002)(2003)(2004) to 37 months (2005-2009) to 25 months (2010-2020) [4]. They conclude that contemporary oncology RCTs now largely measure surrogate end points, such as disease-free and progression-free survival (PFS) and are almost exclusively funded by the pharmaceutical industry. Del Paggio et al. (2021) [4] also note that, "while there are some contexts in which PFS is an appropriate end point, this is the exception and not the rule; in most contexts, PFS is not a valid surrogate" for quality of life or length of life [5][6][7][8].
Using data from studies that could have benefited from a longer duration means either having to use surrogate outcomes or having missing survival time data. In these situations, researchers resort to economic modeling to connect secondary outcomes (e.g., life years) to primary outcomes (e.g., quality adjusted life years or QALYs), or to extend the analysis time horizon from something convenient to something useful. The first part of any debate about the cost-effectiveness of a cancer treatment involves the estimates of ∆C and/or ∆E. A CE Analysis model synthesizing the available evidence is all there is to go on before a cancer drug or intervention is funded. However, after a drug is funded, data exists on how much more expensive (or cheaper) a new treatment is and how much more (or less) effective it is. In general, payers know what they are paying and whether patients are alive. Therefore, it is possible with data routinely collected by a Provincial Cancer Agency or a Ministry of Health (MOH) to estimate ∆C and ∆E. There are both familiar (e.g., missing outcome data) and new challenges (e.g., no randomization) that accompany the use of real-world data, however. Methods for statistical cost-effectiveness analysis as well as economic models provide ways to address many of these challenges.
In the past, there has not been a formal process introducing to decision makers and healthcare funders the results of real-world CE Analysis; nevertheless, researchers have contributed examples of real-world CE Analysis to the scientific literature. However, recently, Canada has created a real-world evidence Working Group to develop guidance on real-world studies for the purpose of health technology assessment in Canada [9]. Therefore, the insights that come from a review of real-world CE Analysis in oncology are useful for informing new plans for optimizing access (via public funding) to both current and future treatments and interventions. While there are many reasons to believe that a single cancer treatment's ∆C, ∆E, and their ratio called the incremental cost-effectiveness ratio (ICER) may differ based on evidence available before versus after funding, it is important to learn from a summary of the published findings from Canadian real-world CE Analyses that have the potential to inform the role of future of real-world evidence initiatives in Canadian oncology.

Methods
Our team conducted a rapid review of existing literature following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [10]. Ethics review was not needed, as this research was conducted using solely publicly available information sources and published articles.

Search Strategy
This review focused on the utilization of "real-world" data for CE Analysis of cancer treatments in Canada. "Real-world" in the context of this review was defined as using data from a cohort, observational study or using individual-level person data from administrative or clinical databases. If evaluating pharmaceuticals, data collection had to occur after public funding. After initial widespread searches, a narrower search strategy was developed with the goal of finding studies fulfilling all of the search criteria. Thus, terms of "Canada," "Cancer," "Cost-effectiveness," and "Real-world" (and variations) were used in the search strategy.
Example Search: "Canada AND (cost-effectiveness OR costs) AND cancer AND (realworld OR observational)."

Information Sources
The research team conducted a primary review in July of 2022 using PubMed, and Google Scholar to potentially identify new sources. Google Scholar yielded no new results. Given the nature of the review and the focus on Canada, all studies selected were based in Canada with a focus on cancer treatments or interventions (e.g., treatment, screening, etc.). Studies needed to include CE Analysis information, and contain information on ∆C and ∆E. ICERs were either obtained directly from the articles, or hand calculated based upon ∆C and ∆E if possible (e.g., ICER = ∆C/∆E). All studies needed to include analysis on a "real-world" population, as per the definition above. There was no date restriction on the search.

Screening Process
Each article included in this review was identified through a systematic review process including (1) title screening, (2) abstract screening, (3) full text review, and (4) full text extraction. At each phase, two reviewers (AMG and HKB) independently screened each article, with JSH serving as arbiter. Despite having an arbiter to make the final decision, disagreements were discussed among all reviewers for a consensus. Covidence software [11] was used to ensure anonymous reviewing, ease of conflict resolution, and to calculate interrater reliability. Study inclusion and exclusion criteria were applied to each phase of review and can be found in Table 1. We calculated a Kappa statistic to measure agreement between the reviewers using the equation: [(% agreement) − (% agreement based on chance)]/(1 − % agreement based on chance).

1.
Study takes place in Canada (province or territory).

2.
Study must focus on cancer and cancer interventions (treatment or prevention).

3.
Study or article conducts or models a cost-effective analysis on cancer treatments/prevention.

4.
Study or article must focus on a "real-world" population.

5.
Articles must be in English. Study or article does not focus on a "real-world" population.

5.
Study or article discusses only an RCT design with restricted populations. 6.
Article is not available in English. 7.
Article is a review (systematic or otherwise).

Conversions and Calculations
If a paper presented a range for any ∆C, ∆E, or ICER, the values were averaged to create a single number for data visualization. To summarize results for comparison, we adjusted all ∆Cs reported in Canadian dollars (CAD) to 2022 Canadian dollars using the Bank of Canada inflation calculator [12]. One study, reported in United States Dollars (USD), was first converted to 2022 value [13], and then to CAD [14]. Once we standardized ∆Cs to 2022 CAD, we calculated the ICER as ∆C/∆E.

Estimates by Study Type
As an exploratory analysis of whether estimates differed by study type, we choose four study type classifications as a matter of sample size and convenience. These study types were defined as binary variables indicating if the research was (i) studying a drug; (ii) using a model; (iii) using the QALY as an outcome; or iv) was recent (i.e., after 2017). We compared means and medians for the estimates of ∆C, ∆E and the ICER. Monetary values were converted to 2022 Canadian dollars as described above. To test differences in the means, we conducted t-test assuming unequal variances between groups. To test differences in the medians we used a non-parametric test on the equality of medians with chi-squared test statistics computed with a continuity correction. The t-test and median testing were done in Stata using the commands t-test and median, respectively [15].

Estimates of Extra Cost and Extra Effect in Real World Studies of Drugs
To illustrate estimates for studies of drugs, we plotted ∆C and ∆E on a cost-effectiveness plane. We picked an arbitrary willingness-to-pay (WTP) value of λ = CAD 100,000 to provide context about the cost-effectiveness.

Article Inclusion and Exclusion Results
The search of the literature in the PubMed database resulted in the identification of 206 unique articles. After conducting the title review, 149 articles were excluded, leaving 57 articles for abstract review. After conducting the abstract review, 23 articles were excluded, leaving 34 to be included in the full-text review. Reasons for exclusion in the full text review portion of the process were: (1) Did not include cost-effectiveness analysis (e.g., only focused on cost or effect), (2) Incorrect setting (e.g., not conducted in Canada), (3) Used a restricted population (e.g., utilized data from a randomized controlled trial), or (4) Did not focus on a cancer intervention. After the full text review, 22 articles remained for data extraction (Figure 1). Overall interrater reliability based on title review, abstract review, and full text review was calculated as 0.87, strong agreement. Curr. Oncol. 2022, 29, FOR PEER REVIEW 5

Overview of the Studies.
The majority of Canadian studies (59%) examining real-world cost effectiveness focused on drugs (Table 2). A smaller portion (41%) of studies focused on non-drug interventions such as screening, surgical interventions, and genetic sequencing. From the 22 studies, we were able to extract 29 (ΔC, ΔE) pairs and compute 27 ICERs. ICERs were not computed for new drugs that were less effective than usual care, as best practice is not to report negative ICERs [16].

Overview of the Studies
The majority of Canadian studies (59%) examining real-world cost effectiveness focused on drugs (Table 2). A smaller portion (41%) of studies focused on non-drug interventions such as screening, surgical interventions, and genetic sequencing. From the 22 studies, we were able to extract 29 (∆C, ∆E) pairs and compute 27 ICERs. ICERs were not computed for new drugs that were less effective than usual care, as best practice is not to report negative ICERs [16].    [18]. Given the real-world nature of these studies, the study populations more accurately reflect actual patient populations than clinical trial populations. For example, Khor et al. [29] included patients over the age of 80 with diffuse-large-B-cell-lymphoma. Not only is the age of these patients remarkable, but this patient population is also more likely to have confounding comorbidities; something not common in RCTs with "ideal" study populations. Similarly, other studies included in the review often cited a mean age of over 60 years of age, with common median and mean ages between 60 and 64 [17,20,21,[23][24][25]29,[31][32][33][34]36].

Study Designs
A small majority (59%) of studies utilized purely administrative or retrospective data to inform their calculations and studies. The rest (41%) used these administrative and retrospective data to inform models for cancer interventions. Commonly used datasets came from Cancer Care Ontario's New Drug Funding Program (NDFP) database [39], the BC Cancer Agency Information System (CAIS) database [40], the pan-Canadian Early Detection Lung Cancer Study (PanCan) [41], Ontario Cancer Registry [42], and the Canadian Kidney Cancer Information System (CKCis) database [43].

Cost Perspectives
All studies cited governmental healthcare payers as the cost perspective. Given the nature the Canada Health Act (CHA) which aims to ensure that all eligible Canadians have access to prepaid insured health services, it is reasonable for studies to utilize a governmental payer perspective, especially if researchers seek to inform government funding decisions [44].

Difference in Effectiveness
The range of LYs gained ranged between -0.66 [23] all the way up to 1.7 [28] among the studies that focused on drug treatments, and 0.06 [25] through 1.4 [28] for QALYs. Gilbert et al. [23] and Arciero et al. [17], reported the only negative values for ∆E, indicating that the intervention was less effective than the standard treatment. The range of ∆Es was smaller for studies that did not focus on drug therapies. ∆Es involving QALYs for non-drug therapies ranged from 0.027 [18] to 0.0662 [33]. Only one study investigated LYs for non-drug therapies which resulted in a ∆E of 0.0025 [38] (Table 3).

Costs
Once all costs were converted to 2022 CAD, studies with the outcomes of LYs and QALYs were broken into the same categories as above: drug therapies and other cancer investments. Of these, drug therapies had, on average, higher ∆Cs compared to the other cancer care investments (CAD 51,489 vs. 2,872, respectively). From 2010, the incremental cost effectiveness ratio (ICER) for drug therapies has increased. The average ICER before 2018 was CAD 126,449.90 and 338,009.02. While the difference was not statistically significant (p = 0.36) the estimated change in ICER value for the two time periods was CAD 211,559.12 (Table 4). For drug therapies, the average ICER values were CAD 134,841/QALY and 187,549/LY. Non-drug approaches to cancer care had a lower average ICER at CAD 37,993/QALY. Only one study examined LYs gained for non-drug cancer care. Weymann et al. [38] reported low observed survival gain (0.0025 LYs or a little less than one more day) with an ICER of CAD 2,501,696/LY. Note: All tests of means were conducted using t-tests assuming unequal means. All tests of medians were conducted using a non-parametric K-sample test on the equality of medians with the chi-squared test statistic computed with a continuity correction. * p < 0.05, ** p < 0.01, *** p < 0.001. Table 4 shows that estimates of ∆C, ∆E, and the ICER did not differ statistically by study type, for the most part. The estimates of the means and the medians do appear to the eye to differ in certain cases; however, because of small sample size, it is likely there is not enough power to detect a statistically significant difference. For example, as mentioned above, the means for ICERs before and after 2018 seem differ by over CAD 200,000; however, the p-value for their difference is greater than 5%. This is seen as well with the ICER using QALYs versus using LYs; there is a difference of over CAD 200,000 but an insignificant p-value (p = 0.29). Studies of drugs had a significantly higher ∆Cs (p < 0.05) and ∆Es p < 0.001) compared to non-drug studies pharmaceuticals. While the ICERs for drug studies vs. non-drug studies differed by over CAD 300,000, this was not statistically significant (p = 0.49). The median ICERs produced by modeling studies was significantly smaller than that produced by person-level data; however, there was no difference in the mean, likely as a result of skewed data and small sample size. Lastly, average ∆C after 2017 was significantly higher (p < 0.05) than in the earlier period. Figure 2 shows estimates of ∆C and ∆E plotted with "Q" or "L" for models using QALYs or Life Years as the outcome. Studies that analyzed a dataset (non-modeling studies), have "q" or "l" to indicate the use of qalys or life years as the outcome. Most of the estimates appear below the dashed willingness-to-pay line with slope CAD 100,000. Table 4 shows that estimates of ΔC, ΔE, and the ICER did not differ statistically study type, for the most part. The estimates of the means and the medians do appear the eye to differ in certain cases; however, because of small sample size, it is likely th is not enough power to detect a statistically significant difference. For example, as me tioned above, the means for ICERs before and after 2018 seem differ by over CAD 200,0 however, the p-value for their difference is greater than 5%. This is seen as well with t ICER using QALYs versus using LYs; there is a difference of over CAD 200,000 but insignificant p-value (p = 0.29). Studies of drugs had a significantly higher ΔCs (p < 0. and ΔEs p < 0.001) compared to non-drug studies pharmaceuticals. While the ICERs drug studies vs. non-drug studies differed by over CAD 300,000, this was not statistica significant (p = 0.49). The median ICERs produced by modeling studies was significan smaller than that produced by person-level data; however, there was no difference in t mean, likely as a result of skewed data and small sample size. Lastly, average ΔC af 2017 was significantly higher (p < 0.05) than in the earlier period. Figure 2 shows estimates of ΔC and ΔE plotted with "Q" or "L" for models usi QALYs or Life Years as the outcome. Studies that analyzed a dataset (non-modeling stu ies), have "q" or "l" to indicate the use of qalys or life years as the outcome. Most of t estimates appear below the dashed willingness-to-pay line with slope CAD 100,000.

Estimates of Extra Cost and Extra Effect in Real World Studies of Drugs
Studies with results to the left of the vertical line at 0, indicate "lose-lose" situatio where an option may actually be less effective but more costly. There are three estima like this in Figure 2    Studies with results to the left of the vertical line at 0, indicate "lose-lose" situations where an option may actually be less effective but more costly. There are three estimates like this in Figure 2 (two using life years and one using qalys). Toward the top of Figure 2, there are three studies in a "poor value" neighborhood. These studies provide evidence of drugs with little extra effect with extra costs in excess of CAD 200,000. Six of the 24 points in Figure 2 have extra effect estimates greater than one year. A majority of these (5 of 6) are estimates from models.

Discussion
Our study reviewed Canadian research exploring 'value' in the real-world, and summarized CE Analyses from Canadian cancer studies to identify post-funding ∆Cs, ∆Es, ICERs and outcome measures. Systematically reviewing these CE Analysis studies provides additional insight; it allows us to identify trends and overall findings at an aggregate level. Our findings show it is possible for researchers to utilize "real-world" databases (e.g., administrative data) to compare incremental cost, incremental effect, and incremental cost effectiveness ratios for different cancers. However, our results also indicate that there is a dearth of CE Analyses for select cancers. This suggests that real-world CE Analysis may not be a feasible option for all cancer types and treatments.
While funding decisions are often informed by CE Analysis results and RCT data, this evidence is from a restricted population that is selected based on strict criteria that make RCT participants different from common cancer patients. Not only does this mean that those who would not quality for trials may not have access to treatments that might have been effective for them, but it also leaves physicians and patients with little concrete information about how to proceed with treatment decisions. The studies in our review conduct CE Analysis using a "real-world" population using administrative databases and medical records. This allows readers to understand what the extra gains and costs are when treating common real-world patients with novel treatments or interventions. Oftentimes, these studies acknowledge the difference in ∆Cs (generally higher) [20,21], ∆Es (generally lower) [22,29] leading to higher ICERs when comparing their real-world results to those from RCTs [21].
Many of the papers we reviewed provide evidence of cost-effectiveness of cancer drugs in the real world as demonstrated by the marker position under the WTP dashed line in Figure 2. However, there are some instances where treatments show poor cost-effectiveness in the real world. For example, Arciero et al. [17] found that first-line gemcitabine plus nabpaclitaxel (Gem-Nab) was dominated by fluorouracil, folinic acid, irinotecan, oxaliplatin (FOLFIRINOX) in patients with advanced pancreatic cancer, as it was less effective and more costly. There are no RCTs comparing Gem-Nab to FOLFIRINOX. Likewise, Gilbert et al. [23] studying repeated cytotoxic chemotherapy treatments for recurrent high-grade serous cancer (HGSC) of the ovaries found that after the third relapse of HGSC, cytotoxic chemotherapy did not prolong survival but was associated with substantially increased healthcare costs. These findings from Arciero et al. [17] and Gilbert et al. [23] provide valuable feedback about real world use of treatments that are both more costly for payers and less effective for patients. They also help demonstrate why both pre-and post-funding CE Analyses are valuable-such studies can help calibrate decision making processes with real-world evidence.
When real world data are combined in a decision analytic model, it is possible to extend past conventional study time horizons. Figure 2 shows this benefit commonly associated with models as a majority of the largest extra effect estimates (i.e., ∆Es) come from model-based analyses. The largest gain from treatment was 1.7 additional life years (indicated with an "L" on the far right of Figure 2). Johnston et al. [28] created this estimate over a 15-year time horizon using a patient-level simulation model for diffuse large B-cell lymphoma (DLBCL) patients initiating treatment with cyclophosphamide, doxorubicin, vincristine, and predisone (CHOP) chemotherapy versus the addition of rituximab to CHOP (CHOP-R). Khor et al. [29] also studied CHOP-R vs. CHOP in DLBCL patients using a cost-effectiveness dataset. They estimated a life expectancy increase of 3.2 months over five years; this corresponds to 0.27 life years (3.2 months/12 months). Thus, models provide more time to see potential gains from treatment. In fact, even within their own data, Khor et al. [29] were able to show how a longer analysis time horizon could boost extra gains; RCHOP was associated with a mean absolute survival gain of approximately 1.3 months (95% CI 0.7-2.3) at three years but it increased to 3.2 months (95% CI 1.6-4.7) at five years. Adding two more years of analysis to the study time horizon increased the ∆E estimate by 1.9 months, more than 146%. The cost of a longer time horizon is often more uncertainty. One can see this in the larger 95% CI for Khor et al.'s five-year ∆E estimate compared to the three-year 95% CI [29].
After adjusting for inflation, studies focusing on cancer drugs had a significantly higher ∆C after 2017. This larger ∆C may reflect rising drug costs, as all costs in Table 4 are presented in 2022 Canadian dollars. While the difference in the ICER was not statistically significant, the estimates were drastically higher post-2017 whereas there was no difference in the incremental effects of the drugs. One approach to improving ICER values in realworld settings is to lower drug costs. The lower numerator (∆C) in the ICER calculation results in a more favorable ICER when considering patients treatment. In real-world analysis, we observed increasing costs and the use of modeling to have a significant impact on the overall value of cancer care. These findings must be interpreted with caution as hypothesis generating. However, utilizing the results of this review as a baseline, it is possible to, with future research, continue to refine clinical trial results to reflect "realworld" evidence. As our review has shown that real-world CE Analysis is possible, future post-funding analyses are indicated in order to validate findings, gain experience, and build community [45].

Strengths and Limitations
There are several potential limitations to this review. First, given the large number of publication databases, it is possible that our search did not include every relevant article. However, given that we found 22 articles to be included after full-text review, it is very likely that we gathered enough source material to convey accurate impressions about the state of the field. Second, currently published real-world CE Analyses may not be representative of results from unpublished real-world studies. It is possible that published research represents the results from studies that are feasible to do (e.g., fast acting cancers). This bias is akin to survivorship bias, as only the studies that can survive the research and publication processes survive to make it into print. Lastly, due to the nature of our review, our findings are intended to inform and promote future research and research directions, rather than directly inform policy or treatment decisions about any particular drug.
Major strengths of our review include a systematic search of PubMed for Canadian economic evaluations of cancer treatments or interventions using real-world data. For our research, two reviewers were used and a third one served as arbiter. This study summarizing Canadian results in this way is one of the first of its kind. We found a variety of published examples from real-world analyses of current cancer treatments that were (1) cost-effective (below the WTP line); (2) not cost-effective (above the WTP line); substantially more effective (with ∆E > 1), and even less effective (with ∆E < 0).

Conclusions
Economic evaluations like cost-effectiveness analysis (CE Analysis) are key to describing value and efficiency. Our findings illustrate the importance of analyzing value pre-and post-funding, considering study design, use of modeling, and time horizon. The results of randomized controlled trials (RCTs) have long been used in funding decisions for cancer care and interventions, but RCTs do not generally use a representative patient population. By looking instead at using real-world observational data on the same treatments through a CE Analysis lens, we see how these interventions will impact the average patient and the public healthcare payer. This type of research allows for the examination of real-world costs and effectiveness of new cancer treatments and interventions, accounting for patient diversity, long-term effects, and generalizability to Canada's cancer patient population. Utilizing real-world data allows for true, large-scale CE Analysis on patient populations who are the actual consumers of cancer interventions and who have not been screened out for the sake of drug approval and funding decisions.