Truth and Pitfalls of Evidence-Based Medicine

The practice of medicine needs to be evidence-based. But what constitutes evidence? Evidence is anything that can be used to prove something. But the evidence can be wrong, subject to various interpretations, or (rarely) fabricated.

Randomized controlled trials (RCTs) are considered the gold-standard to achieve evidence. However, many RCTs often have problems pertaining to their planning, conduct, analysis and the interpretation of data; for example, planning a placebo as a control rather than standard interventions. The interpretation of data is often biased and intended to maximize advantages and minimize adverse events.

But the major problems are in the presentation of data. Most effects are reported as relative risk reduction (RRR) rather than absolute risk reduction (ARR). The number needed to treat (NNT = 100/AAR), i.e. the number of patients that need to be treated to benefit one, is ignored. The number needed to harm (NNH) is seldom reported. The concept of NNH/NNT (harm vs. benefit) is not even considered.

Consider the scenario in which 10% of people with a disease die and 90% survive, and assume that drug X reduces death by 2% (from 10% to 8%), this results in an RRR of 20%, but the ARR is only 2% and the NNT is 50. Reporting the data as RRR will not reveal that 98% of people will receive the drug with no benefit and potential harm.

To better underscore these problems, I will mention few examples.

(1) Inhibitors of the renin–angiotensin system (RASi) are commonly used in the management of patients with cardiovascular (CV) and chronic kidney disease (CKD) on the assumption that this system plays a “major” causal role in these disorders. Experimental evidence in animals strongly supports this assumption. If this evidence fully applies to clinical medicine, the benefits should be generalized, dose-dependent, independent of effects on blood pressure (BP), and result in reduced mortality. This is not always the case.

The HOPE study [1] was planned to prove that, in high-risk patients, ACEi would prevent CV events independently of effects on BP. Patients were randomly assigned to ramipril or placebo groups. Baseline BP in both groups was 139/79 mmHg; approximately 50% of patients had hypertension. In the ramipril group, 14.0% patients developed endpoints as compared with 17.8% in the placebo group. The RRR was 27%, ARR was 3.8%, and NNT was 26. At the end of the study, despite sporadic measurements, and the fact that BP was measured during the day while the drug was administered in the evening, BP was −3/−1 mmHg lower in the treated than in the untreated group. The authors concluded that CV benefits occurred independently of BP. This conclusion is not substantiated by facts. In a sub-study, 38 patients underwent 24 h ambulatory BP monitoring before and one year after randomization; in the treated group, 24 h BP was reduced by 10/4 mm Hg, and nighttime BP was reduced by 17/8 mm Hg [2].

The problems with this study are quite evident. BP should have been measured more systematically, considering that ramipril is a short-acting ACE inhibitor and it was administered in the evening. Given that ~50% of participants had hypertension, the authors should have anticipated a difference in BP among groups, and the control group should have received a non-RAS antihypertensive drug, not a placebo. In fact, the ALLHAT study that compared an ACEi with other antihypertensive drugs showed no differences in outcome among patients treated with lisinopril, amlodipine, or chlortalidone [3].

The RENAL study [4]: To assess the role of the RAS in CKD progression, patients with type 2 diabetes and proteinuria > 0.5 g/d were randomized to losartan or a placebo. The primary outcome was the composite of a doubling of serum creatinine, ESRD, or death. The primary outcome was reached in 43.5% of patients in the losartan and 47.1% in the placebo group. The RRR was 16%, ARR was 3.6%, and NNT was 28. This means that the benefits were not generalized but limited only to 3.6% of patients. The authors concluded that the benefits occurred independently of BP effects, despite differences in BP of 2–4 mmHg between treated and untreated groups, that could account for the observed difference in outcome. Indeed, in the HOT study, among diabetic patients, a difference in diastolic or systolic BP of 4 mmHg could account for a difference in CV outcome exceeding 50% [5].

The problem with the RENAL study was that the treated group received an additional antihypertensive drug, whereas the control group received a placebo. The investigators should have predicted a difference in BP and treated the control groups with a non-RASi antihypertensive drug, and not with a placebo.

The IDNT study: in this study, irbersartan was compared with amlodipine and a placebo in 1715 hypertensive patients with nephropathy due to type 2 diabetes. The target BP was 135/85 mm Hg or less in all groups. The primary endpoint was a composite of a doubling of the baseline serum creatinine, the development of end-stage renal disease, or death from any cause.

Treatment with irbesartan was associated with a 20% RRR of the primary composite endpoint compared with the placebo group and 23% lower than that in the amlodipine group. The mean BP was significantly higher (by 3.3 mm Hg) in the placebo group than in the two active-treatment groups (p = 0.001). The final BP was 140/77 mm Hg in the irbesartan group and 141/77 mm Hg in the amlodipine group (not significant). The authors concluded that irbesartan is effective in protecting against the progression of nephropathy due to type 2 diabetes, and this protection is independent of the reduction in BP. This conclusion is not supported by the data, since there were differences in BP between the irbersartan and the placebo group, and BP was not at its target value in all groups. The study does not prove superiority of irbersartan versus a placebo independent of BP. One could also interpret the data to indicate that amlodipine (when given without an RASi?) provides less or no renal protection when compared to a placebo [6].

If the activation of the RAS plays a “major” role in CKD progression, the benefits should be dose-dependent and result in reduced mortality. Studies combining an ACEi with an ARB have demonstrated no additive benefits [7]. Finally, and surprisingly, a meta-analysis of 19 RCT involving 41,042 diabetic patients has shown that RASi do not reduce mortality [8].

The conclusions that one can draw from these studies are that ACEis are useful drugs in the management of hypertension, particularly in patients with significant proteinuria. However, their benefits can largely be explained by their effects on BP, are not generalized, are not dose-dependent, and do not result in reduced mortality.

(2) A second example is represented by the extensive use of statins for the primary prevention of CVD.

The USPSTF (US Preventive Services Task Force) reviewed 22 trials of statin use for primary prevention. Statin therapy was associated with a decreased risk of all-cause mortality (RRR = 8%, ARR = −0.35%, NNT = 286) and fatal or nonfatal myocardial infarction (RRR 33%, ARR −0.89%, NNT = 112) [9]. This means that 286 patients must be treated to prevent one death and 112 to prevent one myocardial infarction episode. There was no mention of the NNH. The conclusion that one can draw from these data is that statins can reduce CV events and mortality in a limited number of patients, and their use for primary prevention should probably be restricted to those with strong family history of CVD or serious comorbidities.

(3) SGLT2 inhibitors are widely utilized in the management of CV disease and CKD.

The CKD-EMPA Kidney Study examined the effects of empagliflozin compared to a placebo in patients with CKD due to diabetes or other causes. The primary outcome was a composite of progression of kidney disease or death from renal or CV causes. The safety committee prematurely stopped this study because of the clear superiority of empagliflozin. At the end of the study, 13.1% of patients in the EMPA group and 16.9% in the placebo group reached the endpoint. The RRR was 28%, ARR was 3.8%, and NNT was 26. There was a difference in SBP of −2.6 ± 0.3 and DBP −0.5 ± 0.2 mmHg. The benefit was evident only in patients with UACR > 300 mg/g [10]. The authors concluded that, among a wide range of patients with CKD who were at risk for disease progression, empagliflozin therapy led to a lower risk of progression of kidney disease or death from CV than a placebo. The claim for an effect on CV death was not substantiated by the statistical analysis provided.

The problem with this study is that the authors should have known that SGLT2 inhibitors are natriuretic and diuretic and that this would reduce BP. Regardless, they used a placebo in the control group rather than a different natriuretic agent.

(4)Finerenone (FIDELIO Trial) [11]: This was a double-blind trial that randomly assigned 5734 patients with CKD and type 2 diabetes to the finerenone or placebo group. The primary composite outcome was kidney failure, a sustained decrease in eGFR, or death from renal causes. During a median follow-up of 2.6 years, a primary outcome event occurred in 17.8% patients in the finerenone group and 21.1% patients in the placebo group. The RRR was 18%, ARR was 2.4, and NNT was 42. In the finerenone group, the mean SBP at month 12 was −2.1 mm Hg lower than that in the placebo group. The problem of this study was in its planning. The authors knew quite well that finerenone has a diuretic and an antihypertensive effect as well. They should have performed a comparison with a diuretic and not with a placebo.

Conclusions:

Evidence-based Medicine remains the most reliable way to assess the efficacy and the adverse events of a medication or procedure. Fortunately, many trials are very well designed and analyzed taking in consideration mainly the interest of the patients. Unfortunately, a substantial part of the available evidence is based on RCTs that are biased, not correctly planned and interpreted, and presented in misleading ways. Studies should have adequate controls, and their data should provide the ARR, NNT, and NNH and not only the RRR. Editorial Boards of Medical Journals should also be more attentive in pointing out these discrepancies and inadequate presentations of data.

The hope for the future is that more efforts will be dedicated to determining predictors of responders and non-responders to a given treatment, aiming to a more personalized use of drugs, and avoiding the cost and potential side-effects of drugs in patients that do not benefit from them. The patients and the health care services could obtain a great advantage from this approach.