Comment on Bar-Sela et al. Cannabis Consumption Used by Cancer Patients During Immunotherapy Correlates with Poor Clinical Outcome. Cancers 2020, 12, 2447

Simple Summary

Medical cannabis is an evidence-based treatment for chronic pain and chemotherapy-induced nausea and vomiting. Immunotherapies like pembrolizumab work by blocking proteins that stop the immune system from working properly. This commentary critically analyzes a small study by Bar-Sela and coworkers, which appeared to show that cannabis made immunotherapies work less well in patients. Scientific papers should describe what they did in the Methods section and then carefully adhere to this in the Results section. However, we found many instances where there were disagreements between the Methods and Results sections in Bar-Sela’s report. It is also riddled with errors in arithmetic, including unconventional rounding. The prior study did not consider the role of tobacco smoking, which may have been more important than cannabis use in their findings. We propose that artificial intelligence could be used by journals to identify mathematical and statistical errors and prevent studies that are not trustworthy from being published.

Abstract

The small (N = 102) prospective study by Bar-Sela and colleagues at Emek Medical Center in Israel) regarding diminished efficacy of immunotherapy in the setting of cannabis use would be an important discovery which could theoretically benefit the outcomes of oncology patients if verified by independent research teams, including by basic scientists. However, if this finding was spurious, clinical practice guidelines could recommend that oncology patients receiving immunotherapies be erroneously denied an evidence-based treatment for pain and chemotherapy-induced nausea and vomiting. Our full-length manuscript identified dozens of instances of unverifiable statistical information and even errors in arithmetic in this report. More briefly, our concerns regarding this well-cited (123 times) paper are as follows: (1) non-verifiable non-parametric statistics, including some that would change the statistical inferences; (2) difficulties with determining percentages; (3) switching from two-tailed tests in the Methods to one-tailed in the Results; (4) engaging in the unusual practice of floor rounding but not reporting this in the Methods; and (5) not reporting smoking, which could be a key confound. These concerns are serious errors that undermine the validity of the results and invalidate the conclusions that can be drawn from this prospective study about cannabis and immunotherapy.

1. Introduction

The small (N = 102) prospective study by Bar-Sela and colleagues regarding a purported drug interaction between immunotherapy and cannabis [1] would be an important discovery which could theoretically benefit the outcomes of oncology patients, if verified by independent research teams, including basic scientists. The hypothetical interaction results in the immunotherapy being less effective and could indirectly result in poorer outcomes, including mortality. However, if this finding were spurious [2], clinical practice guidelines [3,4] could recommend that oncology patients receiving immunotherapies be erroneously denied an evidence-based treatment for pain and chemotherapy-induced nausea and vomiting [5]. Our full-length manuscript [6] identified dozens of instances of unverifiable statistical information, as well as errors in arithmetic in [1]. More briefly, our concerns regarding this well-cited (123 times) paper are as follows:

A.

Table 1 and the associated text is the foundation of this report [1]. If this baseline information is inaccurate and the groups differ significantly on other variables besides cannabis, then it becomes very challenging to meaningfully assess the rest of this study. They reported (page 3, Section 2.1) that liver metastasis of the immunotherapy group (19% of N = 68 is 13) vs. the immunotherapy–cannabis group (67% of N = 34 is 23) had a p = 0.89. Using GraphPad Prism [7] and the two statistics listed in their Methods section (chi-square and Fisher’s exact tests) with the N provided, we were unable to verify this result. We determined that chi-square (1) = 23.375, p < 0.0001 and Fisher’s p < 0.0001.

Readers may also be confused by the numbers cited in the text for liver metastases (19% I-G, 67% IC-G) not matching what is in Table 1 (19.1% I-G, 32.3% IC-G). Nor do the p values match (p = 0.89 in text, and p = 0.2157 in the table).

B.

Table 1 notes that 76% of the immunotherapy–cannabis group (76% of N = 34 is 26) were given immunotherapy as the second line of treatment versus 54% (54% of N = 68 is 37) of immunotherapy only, which had a p-value of “0.05178”. Again, we were unable to verify this with chi-square (1) = 4.67, p = 0.0317 and Fisher’s p = 0.0334.

C.

Similarly, only one of the 22 p-values (4.5%) in the corrected Table 1 could be verified using the same statistics, as were reported in the Methods section [1]. Based on p-value agreement to at least three decimal places, the authors appear to have reported a different statistic (Yates) than was in their Methods section (chi-square or Fisher’s) on 18 of 22 (81.8%) of these statistics. This is a deviation from common scientific reporting conventions.

D.

Although perhaps less impactful for their core conclusions, there appears to have been some difficulties in calculating percentages (i.e., a difference of 16.2% between the reported 22.0% and our calculated value of 38.2% for chronic diseases = 0 for cannabis users; a difference of 4.1% between the reported 34.1 and calculated 38.2% for high blood pressure for cannabis users; and a 10.3% difference between the reported 13.2% and the calculated 23.5% for bone metastasis for cannabis users) [6], which suggests an inattention to detail in the “corrected” Table 1. Further, the total percentages for chronic diseases for cannabis users (0 = 22.0%, 1 = 20.5%, ≥2 = 41.1%) adds up to 83.6% instead of the usual 100.0%.

E.

The original authors also employed a type of “floor rounding” on 19 occasions [6]. For those who are unfamiliar (as we were), a floor function would take an input (e.g., 4.9) and convert it to the next lowest whole integer (4.0) [8]. As applied here, a number like 0.049999 if calculated to two decimal places would be listed as “0.04” with floor rounding. In retrospect, unusual practices like this should be noted in the Methods section.

F.

The Methods section includes the standard statement that “We computed 2-tailed p-values, where p < 0.05 was considered a statistically significant result.” The sample size calculation sub-section reinforced the reader’s expectation of two-tailed statistical tests by stating “alpha of 5% for the two-tailed test.” for the complete response and partial response calculations. However, the Results section then notes that there was “(one-tailed p = 0.08) (Table 2).” [1]. Although we found [6,7] the one-tailed Fisher’s was “0.089”, which would commonly be reported as 0.09, this could be attributed to floor-rounding. The alkaline phosphatase level was reported as “0.09”, but we obtained a one-tailed chi-square of 0.0687, one-tailed Fischer’s of 0.1089, and even a one-tailed Yates of 0.1079. Switching the data analysis plan from the Methods to the Results section (i.e., from two-tailed to one-tailed) is incongruent with typical practice in scientific reporting. If the authors made their deidentified raw data and a detailed data dictionary publicly available on GitHub at https://github.com/ (assuming that this were acceptable to the Rambam IRB), this could allow for independent re-analysis of this and other statistics (Table 4, and Figure 3 and Figure 4).

G.

Tobacco smoking is a defining theme of oncology that impacts cancer risk, diagnosis, and treatment [9,10]. This includes lung cancer, with a relative risk of seven among smokers [11]. Lung cancer patients that quit smoking may have a 20 to 35% improved survival [10]. Current smokers had a greater (Hazard Ratio = 1.48) likelihood of melanoma-associated death than non-smokers [12]. A meta-analysis revealed that current smokers with renal cell carcinoma had poorer progression-free survival (HR = 2.94) than non-smokers [13]. Note that about half the Bar-Sela sample from Israel consisted of non-small cell lung cancer, one-third melanoma, and the rest were renal cell carcinoma or other malignancies [1]. Among Israeli cancer survivors, significantly more cannabis users also used tobacco [14]. Adding further complexity, about one-third of cancer survivors with an active cannabis permit in Israel reported mixing tobacco with cannabis [14]. The Bar-Sela team should be commended for reporting smoking in Table 2 of their retrospective study [15] as this could be an important confound [9,10,11,12,13,14]. However, this key variable was never mentioned, and therefore, any variance due to this was not removed in their prospective study [1].

H.

There is a very limited sample size within the cannabis group (n = 34), who were more likely to receive second line treatment, were more likely to have impaired performance statuses (ECOG ≥ 2) and had higher rates of baseline lymphopenia (all associated with worse clinical outcomes). Although none of these differences reach statistical significance, it does raise questions about important differences that may exist between the groups that may not have been apparent with this limited sample size. A median overall survival of six months seems unusually low in the modern immunotherapy era for these malignancies, which likely suggests additional unmeasured confounds associated with cannabis use and cancer outcomes.

2. Potential Solutions

We do not fault the reviewers, including the statistician reviewer (if there was one), for not identifying issues A–D. However, we do not believe it would be impossible for journals like Cancers to use software [16] or artificial intelligence to calculate percentages, chi-squares, and t-tests to assist the reviewers in flagging potential issues, particularly ones that change statistical decision making, earlier. Leveraging Large Language Models (LLMs) like OpenAI’s ChatGPT o1 model could appreciably reduce statistical errors in manuscripts, as demonstrated in Appendix A. With minimal input, such as copying relevant tables, specifying statistical tests, and simple prompt engineering, these models efficiently validate percentages, recalculate p-values, and flag discrepancies in parametric or non-parametric tests. The o1 model, highlighted in OpenAI’s report on learning to reason with LLMs [17], outperforms earlier models like 4o in statistical reasoning, often exhibiting the precision of a statistician by recognizing when tests like Fisher’s exact test were employed. Our conversation threads with 4o and o1 (Appendix A) demonstrate how o1 consistently identified errors and recalculated values that 4o missed, showcasing its ability to navigate complex statistical tasks with minimal guidance. This level of accuracy may not have been achievable with simpler models, but advanced LLMs like o1 offer valuable tools for both authors and reviewers, assisting peer review by automating statistical verification as a double-check and reducing the likelihood of publishing flawed findings.

Of course, AI models will need to be thoroughly tested and extensively evaluated to ensure their accuracy before being incorporated in the biomedical review process. Journals could begin this process by testing whether prompting authors to use AI to double-check the math of their preprint prior to submission would produce a quantifiable reduction in PubPeer comments or published corrections. Later, this could advance to more complicated tasks like identifying discrepancies between a reported correlation coefficient, or t-test, and the p-value when the degrees of freedom are provided. Repeating even simple (i.e., 2 × 2) non-parametric analyses would require greater standardization of how these are written than biomedical journals currently require. Authors could rebut erroneous AI feedback or make adjustments at the pre-submission stage. The utilization of LLMs preserves expert judgment for nuanced interpretation and contextual critique, while using AI for rapid, standardized checks that streamline the process for both journal staff and authors. AI can act as a first check for tedious math-checking, so reviewers can focus their mental energy on evaluating science.

Some reputable journals in psychology (Psychological Science and the Journal of Experimental Social Psychology) have incorporated the R application statcheck as a standard part of their peer review process [16]. The statistical standards, as well as transparency, of oncology should be at least as rigorous as those employed in the social sciences. Of course, human reviewers will continue to be crucial to thoroughly vetting methodologically suspect manuscripts, including identifying the omission of potentially key confounds (A, B, and G) prior to publication. If these are identified, then observational studies could use standard multivariate statistics to remove the variance attributable to them or, if the sample size was sufficient, stratify the sample accordingly (e.g., by nicotine use). Further, observational studies, with their potential for multiple unmeasured confounds, occasional reliance on self-reported data (e.g., smoking), and lack of random assignment, should not be interpreted in terms of causality. Controlled basic science investigations do not suffer from these limitations. A small (N = 7–9/group) mouse non-small cell lung cancer investigation revealed that survival was non-significantly longer in the anti-PD-1 antibody and tetrahydrocannabinol group (54 days) relative to the anti-PD-1 only group (31 days) [17].

More broadly, “cannabis” is not a single homogenous substance. One of the main cannabis producers in Israel, Tikun Olam Ltd., cultivates sixteen different strains, including ten chemotypes with more than 18% tetrahydrocannabinol and up to 1% cannabidiol, as well as two types with 1% tetrahydrocannabinol and ≥18% cannabidiol [18]. The US situation may be even more challenging, with almost two-thousand “strains” in an online database [19]. The impact of THC and CBD on the immune system is understudied [20]. More specifically, further research on whether checkpoint inhibitors differ in their potential interaction with certain cannabinoids is needed. Future efforts to examine the benefits, or harms, of cannabis would benefit from characterizing a single formulation with a known daily dose of the major cannabinioids with a single route of administration [21], which would facilitate the ability to draw meaningful inferences based on more defined pharmacokinetic and pharmacodynamic properties.

3. Conclusions

The authors of future narrative and systematic reviews, meta-analyses, and clinical practice guidelines may find ambiguous situations like these to be challenging. The writings of JPA Ioannidis may provide some guidance [2,22]. Issues A-C, F, and potentially G, appear to us to be serious errors that undermine the validity of the results and invalidate the conclusions. This commentary is not intended to question the integrity of Bar-Sela and colleagues. We leave it to others to infer whether A-G reflect typos, an inattention to detail, misinterpretation on our part or whether the underlying drug interaction is suspect. The retrospective [15] and prospective reports [1] have been criticized because cannabis use may only be a surrogate for a high burden of symptomatic disease [17]. Put more plainly, cannabis may be used as palliative care for those who recognize they are nearing the end of their lives. It also did not escape notice that one author of a recent review on phytocannabinoids as chemotherapy adjuncts [23] elected to omit the study in [15] from consideration due to statistical concerns. This may be a prudent approach. While writing from the perspective of an oncology researcher (MB), a pharmacist (DXD), an applied mathematician (JG), a radiation oncologist (TMC), and a cancer patient that has failed two immunotherapies (BJP), we would like to express our gratitude to Bar-Sela and colleagues for their thought-provoking manuscripts [1,15].