Synthesizing and Assessing Influenza Vaccine Evidence: Strengths and Limitations of the Recent ECDC Report on the Effectiveness of New and Enhanced Influenza Vaccines. Communication on: The “European Centre for Disease Prevention and Control. Systematic Review Update on the Efficacy, Effectiveness and Safety of Newer and Enhanced Seasonal Influenza Vaccines for the Prevention of Laboratory Confirmed Influenza in Individuals Aged 18 Years and Over. Stockholm: ECDC; 2024”

Abstract

High quality research is critical for evidence-based decision making in public health and fundamental to maintain progress and trust in immunization programs in Europe. In 2024 the European Centre for Disease Prevention and Control (ECDC) conducted an update of the 2020 systematic review to capture more recent evidence on of the efficacy, effectiveness of influenza vaccines in individuals aged 18 years and older in the prevention of laboratory-confirmed influenza. While this report was highly anticipated due to the strength of the protocol and processes put in place, during our assessment, we expressed two chief concerns. We are concerned by the grading of the evidence certainty applied and being unable to reproduce some data extracted in the report from the primary sources. While the systematic review benefited of strong methods and processes, the execution of the research protocol warrants revision due to the issues discussed. We encourage the ECDC to work towards an updated review within a reasonable time frame to avoid misinterpretation by decision-making bodies across Europe.

High quality research is critical for evidence-based decision making in public health and fundamental to maintain progress and trust in immunization programs in Europe. In 2020, the European Centre for Disease Prevention and Control (ECDC) conducted a systematic literature review (SLR) of the efficacy, effectiveness and safety of influenza vaccines in individuals aged 18 years and older in the prevention of laboratory-confirmed influenza [1]. In 2024 the ECDC subsequently conducted an update of the 2020 systematic review to capture more recent evidence [2]. This review was highly anticipated by influenza experts in the National Immunization Technical Advisory Groups (NITAG) and healthcare practitioners, since it provides an important scientific summary to facilitate decision making by individual Member States. The previous report was instrumental in that regard, and we were encouraged by the improvements made to the protocol, that further strengthened the value of the report. By focusing on specific, laboratory confirmed, and hard clinical outcomes in addition to the emphasis on randomized trials, the design of the updated SLR was indeed strengthened. The choice of using “ROBINS-I” tool for evaluating risk of bias in estimates of the effectiveness from studies that did not use randomization to allocate interventions also strengthened the methodology of quality assessment. The broad range of expertise from across EU/EEA Member States in the implementation of the protocol ensured comprehensive assessment of the document.

While this report was highly anticipated due to the strength of the protocol and processes put in place, during our evaluation, we had two chief concerns. First, we are concerned by the grading of the evidence certainty of several studies, and second, we are concerned as we were unable to reproduce an important proportion of the data extracted in the ECDC report from the primary sources. Considering the impact this report can have on the public health of older adults in the European Union, we felt compelled to communicate on our evaluation of the ECDC report.

Our primary concern relates to determination of certainty of evidence using GRADE methodology. The new report may have inaccurately determined the certainty of evidence of Domnich et al. 2022 [3], a post-hoc analysis part of the DRIVE (Development of Robust and Innovative Vaccine Effectiveness) project. In their assessment, the ECDC Report 2024 authors gave a ‘moderate’ certainty for that post-hoc observational study with 512 participants that performed un-reported adjustments to move relative vaccine effectiveness from -92% to +59% [4]. Recently, Domnich et al. acknowledged the limitations of their post hoc analysis, recognized that their results are influenced by confounding factors and emphasized that the choice of statistical method impacted the obtained results [5]. To put this into perspective, the same level of certainty (moderate) given to Domnich et al. was also given to a prospective trial for regulatory registration, powered, double-blinded and individually randomized with 31,989 participants [6], as well as to a prospective, individually randomized, double-blinded trial with 9003 participants [7]. It is well established across disciplines of medicine that individual randomization in study design significantly improves the quality of evidence, and that observational studies carry inherent challenges of confounding and bias. We believe that the strongest evidence should inform influenza vaccine policy and that the certainty of evidence provided by observational studies should not be considered equal to a randomized clinical trial.

Our other concerns relate to the scientific reproducibility of the report. Thanks to the published protocol, we recreated the literature search. While the ECDC 2024 SLR identified only one study of MF59 adjuvanted vaccine preventing “influenza hospitalizations”, indeed there are three more studies incorrectly classified in the report as “influenza” endpoint [8,9,10]. Further inaccuracies also complicated interpretation of the results: Table 21 of the report provides estimates of relative vaccine effectiveness (rVE) of -1% (-122 to 59%) from Bellino et al., whereas Table 3 from the original publication shows absolute VE [11]. Some other rVE values referenced in the report could not be located in the original publications. For example, Table 21 of the report provides rVE against influenza A(H3N2) of 88% (51-100) citing Rondy 2017b, while the original publication states that the low number of A(H3N2) cases “did not allow us to compute IVE against this subtype” [10]. Similarly, Pebody 2020b in Table 21 of the report provides a rVE of 16% (176 to 75%) which we were unable to find in the original publication [12]. Furthermore, data from an observational study on influenza laboratory-confirmed hospitalization were not extracted, and the study was not evaluated, despite the fact that the study met the inclusion criteria and was published within the review period (Zimmerman et al. 2023) [13].

The discussion section could be improved to help readers understand the limitation of the PICO of the ECDC 2024 SLR. By excluding non-influenza vaccine comparators, as such, the review did not capture an important, well powered randomized clinical trial (RCT) by Beran et al. that contained laboratory confirmed outcomes [14]. Similarly, the focus on laboratory confirmed outcomes is understandable, due to the increased specificity it offers on estimates, but by excluding non-laboratory-confirmed outcomes this review ignores the benefit that vaccines provide by preventing serious complications of influenza infections. The impact of vaccinations beyond influenza infection, such as cardio-respiratory events, has been well established and forms part of national guidelines. This decision also fails to recognize that diagnosis of hospitalized influenza from some environments (notably the US) is a specific endpoint with high rates of laboratory confirmation. By excluding non-lab-confirmed outcomes, this review misses a critical study that can aid in evaluating vaccine effectiveness, namely a RCT by Johansen et al. [15] that showed lower incidence of hospitalization for pneumonia and influenza and all-cause mortality. While we understand the focus of this SLR, at least the authors should discuss the strengths and limitations of this focus on a small aspect of public health burden prevention. This is not currently the case.

Lastly, the ECDC report uses the Cochrane GRADE framework to assess certainty of evidence but deviates from its well-established methodology by downgrading evidence solely based on the source of funding. This means that gold-standard, FDA/EMA supervised, and fully audited studies based on transparent and reproducible prespecified methods that were funded by manufacturers receive poor evaluation by default. This unusual deviation from GRADE should be made more prominent to readers to understand what aspect of the certainty of evidence could be affected by funding source, and justify their deviation of established Cochrane methodology.

In conclusion, by equalizing the certainty of evidence from observational studies and randomized trials, the ECDC 2024 report goes against established principles which need to be carefully considered. The inaccuracies in data extractions affect the overall reliability of the report and its practical implementation. While we continue our evaluation, we encourage the ECDC to already work towards an updated report within a reasonable timeframe and with more transparency, as the current report is already influencing decision-making bodies across the EU as exemplified by the recent reports from the authorities in Ireland [16] and Belgium [17].