Efficacy and Effectiveness of SARS-CoV-2 Vaccines: A Systematic Review and Meta-Analysis

The coronavirus disease 2019 (COVID-19) pandemic has threatened global health and prompted the need for mass vaccination. We aimed to assess the efficacy and effectiveness of COVID-19 vaccines to prevent mortality and reduce the risk of developing severe disease after the 1st and 2nd doses. From conception to 28 June 2021, we searched PubMed, Cochrane, EBSCO, Scopus, ProQuest, Web of Science, WHO-ICTRP, and Google Scholar. We included both observational and randomized controlled trials. The pooled vaccine efficacy and effectiveness following vaccination, as well as their 95 percent confidence intervals (CI), were estimated using the random-effects model. In total, 22 of the 21,567 screened articles were eligible for quantitative analysis. Mortality 7 and 14 days after full vaccination decreased significantly among the vaccinated group compared to the unvaccinated group (OR = 0.10, ([95% CI, 0.04–0.27], I2 = 54%) and (OR = 0.46, [95% CI, 0.35–0.61], I2 = 0%), respectively. The probability of having severe disease one or two weeks after 2nd dose decreased significantly (OR = 0.29 [95% CI, 0.19–0.46], I2 = 25%) and (OR = 0.08 [95% CI, 0.03–0.25], I2 = 74%), respectively. The incidence of infection any time after the 1st and 2nd doses diminished significantly (OR = 0.14 [95% CI, 0.07–0.4], I2 = 100%) and (OR = 0.179 [95% CI, 0.15–0.19], I2 = 98%), respectively. Also, incidence of infection one week after 2nd dose decreased significantly, (OR = 0.04, [95% CI (0.01–0.2], I2 = 100%). After meta-regression, the type of vaccine and country were the main predictors of outcome [non-mRNA type, ß = 2.99, p = 0.0001; country UK, ß = −0.75, p = 0.038; country USA, ß = 0.8, p = 0.02]. This study showed that most vaccines have comparable effectiveness, and it is purported that mass vaccination may help to end this pandemic.

The major determinants of vaccine acceptance are vaccine safety and efficacy [25]. Most COVID-19 vaccines have mild side effects, such as pain at the site of injection, tiredness, headache, fever, or shivering for 1-2 days after vaccination. Very rare side effects include allergic reactions and blood clotting problems, the latter affecting a small number of people who had the Oxford/AstraZeneca vaccine [26]. Vaccine efficacy is defined as the degree to which a vaccine prevents disease, and possibly, also its transmission under ideal and controlled circumstances; this is determined by comparing a vaccinated group with a placebo group in a randomized controlled trial (RCT). Vaccine effectiveness also refers to how well the vaccine performs in the real world based on observational studies [27]. The aim of this systematic review and meta-analysis was to shed light on different studies evaluating the efficacy and effectiveness of COVID-19 vaccines after phase III trials.

Materials and Methods
This study was conducted in accordance with the Preferred Reporting Items of the Systematic Review and Meta-Analysis (PRISMA) checklist [28]. All steps were performed with strict compliance to the Cochrane Handbook of Systematic Review and Meta-Analysis [29]. Supplementary Table S1. Reference [1] is cited in Supplementary Materials.

Inclusion and exclusion criteria
All studies that met the following criteria were included: • Reported COVID-19 vaccine efficacy (RCTs) or effectiveness (observational studies). • Had a comparator group receiving either a placebo or another vaccine.

•
The intervention group were either partially vaccinated (received only one dose of COVID-19 vaccine) or fully vaccinated.

•
No restriction regarding country, race, gender, or age.

•
We excluded abstract only letters to the editor, reviews, conference reports, study protocols, author responses, case reports, case series, and surveillance studies with no control group, in addition to any studies that had unreliable data for extraction or duplicates.
Secondary outcomes: • Efficacy and effectiveness of the vaccine to prevent severe disease.

Operation cases definition
Confirmed cases are persons who had a positive nucleic acid amplification test (NAAT), a person with a positive SARS-CoV-2 rapid diagnostic antigen test [30] and fulfill probable or suspected criteria of WHO case definitions, or a positive SARS-CoV-2 antigen (by rapid diagnostic test) asymptomatic patient but in close contact to probable or confirmed case [31].
Severe COVID-19: Adult patients categorized as having severe COVID-19, if matching one of the following criteria: oxygen saturation less than 90% in room air, a respiratory rate more than 30 breaths per minute, or had signs of severe respiratory distress [32].
Critical COVID-19: Adult patients with acute severe acute respiratory distress, septic shock, or any life-threatening condition needing critical care admission or mechanical ventilation [32].
Test negative cases control design: The best study design to detect risk factors of severe COVID-19 illness. In this study type, symptomatic COVID-19 patients are tested using a polymerase chain reaction (PCR) test, then categorized into cases (test positive patients) and controls (test-negative patients) [33].
Search methods for identification of studies Electronic searches: The following databases were searched: Scopus, EBSCO, MED-LINE central/PubMed, Cochrane Central Register for Clinical Trials (CENTRAL), WHO International Clinical Trials Registry Platform (ICTRP), Web of Science (WOS), ProQuest Coronavirus database, and Google scholar. Search terms were determined and approved after the consultation with PubMed. The following keywords were used in our search, after adapting according to each database search strategy, ('coronavirinae' OR 'coronaviridae infection' OR 'coronavirus disease 2019' OR 'coronavirus' OR 'coronavirus infection') AND (Vaccin * efficacy * OR Vaccin * effective * OR Vaccin * immune *). We searched these databases to compile all studies on COVID-19 vaccination that were available up to 28 June 2021.
Searching other resources: In addition to searching the grey literature, a manual search of studies by checking reference lists of all eligible papers was undertaken to ensure that we did not miss any relevant study.

Data Collection and Extraction
Two independent reviewers searched the included databases (S.H. and R.M.G.). After which, all citations were exported to Endnote 20 to remove duplicates. Title and abstract screening was conducted by (R.A., A.A., N.H., D.M., and O.A.R.) and disagreement was resolved by (R.M.G). The inter-reviewer agreement was substantial (K = 0.8). Two reviewers (R.M.G. and S.H.) delineated accepted papers eligible for full-text screening. The two reviewers then extracted data related to patient characteristics and outcomes (authors, year of publication, country, inclusion, and exclusion criteria, when the study was conducted, study design, sample size, type of vaccine and number of doses, time point of analysis, primary and secondary outcomes). The Supplementary Materials of eligible articles were also reviewed for relevant data and the extracted data were checked and confirmed by a third author (Y.E.).

Data Analysis
Assessment of publication bias: We assessed publication bias through a visual inspection of the funnel plot and Egger's test.
Quality assessment: The risk of bias was assessed using two different tools according to the type of study: Cochrane risk of bias for randomized controlled trials (RoB2) [34] and the National Heart, Lung, and Blood Institute's quality assessment tool for cohort, cross-sectional, and casecontrol studies [35]. D.M. and N.H. reviewed the quality of the studies and any disagreement was resolved by R.M.G. and S.H. Effect size measurement: We used the random-effects model to study the proposed outcomes due to the significant heterogeneity. The effect size was reported as odds ratio (OR) and 95% confidence interval level.
Assessment of heterogeneity: • Visual inspection of the forest plot was carried out to analyze the consistency of intervention effects across the included studies. If the same intervention effect is estimated, there should be an overlap between the confidence intervals for each effect estimate on the forest plot. However, if the overlap is weak, or there are outliers, statistical heterogeneity is likely to be present.

•
Statistical test for variation: heterogeneity was assessed by inspecting the forest plots to detect overlapping confidence intervals (CIs) and the I 2 statistic used to denote levels of heterogeneity as defined in the Cochrane Handbook for Systematic Reviews of Interventions [30].
Sensitivity analysis: To conduct a sensitivity analysis, we recalculated the results of our meta-analysis K times, leaving out one study each time. This analysis also provides a classification for what is considered influential. We ordered studies in the plot via I 2 . Here we identified the studies with the highest heterogeneity as well as the final heterogeneity when these studies were removed.
Subgroup analysis: The included studies were divided into subgroups based on research design (RCT and observational), and the study outcomes were compared between the subgroups of studies.
Meta-regression: The outcome variable was the effect estimate (COVID-19 vaccine efficacy and effectiveness). The explanatory variables were study design (RCT/observational), type of vaccine (mRNA/not mRNA), and country where the study was conducted.

Study Selection Process
A total of 21,567 articles were found after searching nine different databases. Out of these, 8088 articles were excluded either because they were duplicates as found by Endnote X8 or because they were published before 2019. The title and abstract screening of 13,479 papers resulted in exclusion of 13,284 irrelevant papers and 195 manually found duplicates. A total of 78 articles were screened for eligibility. Finally, 22 papers were deemed eligible for the meta-analysis ( Figure 1). List of papers rejected and causes of rejection are provided as Supplementary Table S2. References  are cited in Supplementary Materials.

Publication Bias
We assessed publication bias of mortality associated with COVID-19 (primary outcome) through visual inspection of a funnel plot ( Figure 2) and by conducting Egger's test (t = 0.844, p = 0.43). We found that data was symmetric with low risk of publication bias.

•
Main finding: incidence of SARS-CoV-2 infection in observational studies decreased significantly after vaccination within 14 days of 1st dose.       Thus, we carried out a meta-regression analysis to understand the main predictors of this heterogeneity. Again, the type of vaccine and the country were responsible for 88.21% of the heterogeneity; [non-mRNA vaccine β = 3.519, p = 0.004; country Spain β = 2.6256, p = 0.028: Israel was the reference country) ( Figure 1A). • Seven studies reported the incidence of symptomatic infection 14 days after the 2nd dose.

•
Main findings: incidence of SARS-CoV-2 cases (total cases and asymptomatic cases) two weeks after the 2nd dose was significantly reduced. The test for subgroup differences suggested that there was no statistically significant subgroup effect (p = 0.98), meaning that the type of study did not significantly modify the efficacy and effectiveness of vaccination. Vaccination decreased the number of cases regardless of the study design, although the protective effect was greater in RCT than in observational studies. There was no heterogeneity between the results of the RCT studies, while the heterogeneity of observational studies was I 2 = 99% ( Figure 13).

•
Main findings: vaccination against SARS-CoV-2 decreased incidence of infection after the 2nd dose regardless of the duration.

Discussion
The aim of vaccine development is to provide a weapon that protects people from getting infected or becoming a source of transmission. By the end of 2020, several COVID-19 vaccines had become available for use across the world, with over 40 different vaccines in human trials, and over 150 in preclinical trials. An updated list of vaccine candidates under evaluation is maintained by the WHO [58]. Although some of the vaccines were approved for emergency use by the FDA in the USA and the respective health departments of other countries across the world, the efficacy and effectiveness should be periodically assessed due to the ongoing antigenic drift. It is worth noting that while vaccinations are still being administered worldwide, the vaccinated population (received at least one dose of vaccine) represents around three-fifth of the entire population [59], with safety and effectiveness representing the main concern and points of hesitation for many people [25]. Another main concern affecting vaccination coverage is COVID-19 vaccine inequity; vaccine supply will have a long-term and severe impact on socioeconomic recovery in low-and lower-middleincome countries (LMIC) unless immediate action is taken to increase supply and provide equal access for all countries. If LMIC had similar vaccination rates as high-income countries (HIC), an acceleration in scaling up manufacturing and providing adequate vaccine doses might have added costs. A high price per COVID-19 vaccine dose in comparison to other vaccines, as well as delivery costs, including those for the health workforce surge, could put a huge strain on fragile health systems, undermining vaccination programs and essential health services, and causing alarming spikes in measles, pneumonia, and diarrhea [60].
Several studies recruited different numbers of participants to study various outcomes with variable endpoints providing different doses of vaccine with variable intervals. Studying the efficacy and effectiveness of several types of COVID-19 vaccines in reducing mortality and severity were the focus of this meta-analysis aiming to provide strong evidence to decision-makers in health policy sectors to deal with the ongoing pandemic.
We reviewed a total of 22 articles to study the desired outcomes, among which 10 were RCTs, 4 were case-control, and 8 were cohort studies. The highest number of recruited subjects in a single study was 6,538,911 [49], while the smallest number was 268 subjects [48].

Mortality and Severe COVID-19
Based on the findings of this meta-analysis, the mortality related to COVID-19 two weeks after vaccination was significantly decreased (OR = 0.46, [95% CI, 0.35-0.61], I 2 = 0%). Similarly, mortality one week after vaccination dropped significantly (OR = 0.10, [95% CI, 0.04-0.27], I 2 = 54%). In RCTs, the odds ratio for severe COVID-19 was 0.14 [95% CI, 0.03-0.75], I 2 = 30%), whereas in observational studies, the odds ratio was 0.06 [0.02-0.24], I 2 = 85%. In the same line, the odds ratio of having severe COVID-19 after the 1st dose was 0.15 [0.10-0.25], I 2 = 26%. Analyzing our results, different studies reported a significant reduction in SARS-CoV-2 infection, hospitalizations, and fatalities among those who had been fully vaccinated compared to those who had not been fully vaccinated [61][62][63]. Fiolet et al. [64] recently published a review on different COVID-19 vaccines effectiveness; when the strain was not sequenced, the effectiveness of the mRNA vaccination against hospitalization and mortality was over 87-94%. Similarly, inactivated viral COVID-19 vaccine (CoronaVac) was extremely effective against hospitalization (87.5%) and death (86.3%). In addition, if a breakthrough occurs in a vaccinated individual, the events are usually less severe than in an unprotected person [65]. Similarly, a recently published metaanalysis highlighted that the BNT162b2 and mRNA-1273 vaccines had the best effectiveness in preventing symptomatic COVID-19. The efficacy of comparing different vaccines in preventing serious illness was not different. Moreover, there was no difference in the efficacy of vaccinations to prevent symptomatic COVID-19 among the elderly [66]. Unfortunately, this protective effect wanes with time-5 months or more after vaccination-and vaccine effectiveness decreased against hospitalization and deaths (80.0 and 84.8% with the ChAdOx1-S) and (91.7% and 91.9% with BNT162b2), respectively [67].

Infection after Vaccination
It is worthy to note that Alagoz et al. [68] hypothesized that if there is a strong adherence to non-pharmacological interventions in the community, the controllable spread of SARS-CoV-2 can be reached sooner than when a substantial part of the population gets vaccinated (e.g., 70-80%). In the current study, COVID-19 vaccines effectively reduced the incidence of symptomatic and asymptomatic infection. On the same lines, the WHO reported that unvaccinated persons account for the great majority of the current SARS-CoV-2 infection [65]. Virus-neutralizing antibodies are principally responsible for the protection provided by presently available vaccinations. These antibodies often inhibit the virus's binding with its cellular receptor or prevent the virus from undergoing the conformational changes essential for fusion with the cell membrane [69] We found that vaccination against COVID-19 decreased the number of cases reported within a week of the 2nd dose (OR = 0.06 (95% CI, 0.02-0.21), I 2 = 98%). Type of vaccine and country where study was conducted were the main predictors of vaccine efficacy and effectiveness. Similarly, the total number of cases diagnosed within 14 days of the 2nd dose decreased significantly, (OR = 0.01 [95% CI, 0.01-0.02], I 2 = 0%). In terms of cases reported 7 days after 2nd dose, the total number of cases decreased significantly with vaccination (OR = 0.03 [95% CI, 0.02-0.05], I 2 = 73%). About 100% of this heterogeneity was explained by meta-regression (vaccine type and country). Regarding symptomatic cases diagnosed 7 days after the 2nd dose, COVID-19 vaccine was effective in reducing the number of symptomatic cases in comparison to placebo or control group (OR = 0.02 [95% CI, 0.02-0.02], I 2 = 0%). The odds ratio of cases reported 14 days after the 2nd dose among vaccinated versus unvaccinated subjects was OR = 0.08, [95% CI, 0.02-0.34], I 2 = 100%). Confirmed cases reported after the 1st and 2nd dose regardless of the duration decreased significantly, OR = 0.14 (95% CI, 0.07-0.4) I 2 = 100% and 0.18 (95% CI, 0.15-0.19), I 2 = 98%, respectively.
In the same vein, many reviews addressed vaccine effectiveness and efficacy. Pormohammad et al. [70] included 25 studies in phase II/III RCTs, the efficacy of mRNA-based and adenovirus-vectored COVID-19 vaccines was 94.6% and 80.2%, respectively. After 3 weeks of vaccinations, the adenovirus-vectored vaccine had the maximum efficacy against receptor-binding domain (RBD) antigen after the 1st and 2nd doses (97.6% and 98.2% respectively). Similarly, a review of phase III studies showed a significant increase in neutralizing antibodies with the 2nd dose of the vaccine [71]. However, it was also advised that when vaccine supply is scarce, countries should vaccinate with a single dose. This may provide better overall protection in the population than vaccinating half the number of individuals with both doses [72].
Many factors can explain the observed difference in efficacy and effectiveness of the COVID-19 vaccines. The Center for Disease Control and Prevention [73] demonstrated that in the real-world, vaccine effectiveness can be affected by several factors, including population host factors (e.g., those who were not included in clinical trials) and virus factors (e.g., variants) as well as programmatic factors (e.g., adherence to dosing schedules or vaccine storage/handling) [74]. Thompson et al. [75] reported that under real-world conditions, complete immunization (14 days after 2nd dose) was 90% effective against SARS-CoV-2 infection, while partial immunization (14 days after 1st dose but before 2nd dose) was 80% effective. In addition, the effectiveness of vaccination varied according to the types of vaccine; Pilishvili et al. [76] stated that vaccine effectiveness for Pfizer-BioNTech and Moderna were 77.6% (95.6% CI, 70.9-82.7) and 88.9% (95.9% CI, 78.7-94.2) after the 1st dose and were 88.8% (95% CI, 84.6-91.8) and 96.3% (95.3-98.4) after the 2nd dose, respectively. Of note, when the SARS-CoV-2 Delta variant became prevalent, the percentage of completely vaccinated people who got SARS-CoV-2 infection grew higher than predicted [63]. The effectiveness of the mRNA vaccine against COVID-19 was 88-100% against Alpha, 76-100% against Beta/Gamma, 47.3-88% against Delta, and 89-100% when the SARS-CoV-2 strain was not sequenced. Oxford/AstraZeneca (AZD1222) was 74.5% effective against Alpha and 67% effective against Delta. CoronaVac was effective against the Alpha/Gamma/D614G strain in 36.8-73.8% of cases [64].
Unfortunately, new data consistently demonstrated that vaccine efficacy against SARS-CoV-2 infection declines with time following immunization [77]. It is worth noting that according to a recently published systematic review and meta-analysis, immunization efficacy against severe COVID-19 infection dropped by around 8% (95% CI, 4-15) during the 6-months period in all age groups. Over the same time, vaccine efficacy against serious illness declined by around 10% (95% CI, 6-15%) in individuals over the age of 50. Vaccine efficacy against symptomatic illness fell by 32% (95% CI, 11-69%) in individuals over the age of 50 [78]. Consequently, WHO has already recommended administering a booster dose of vaccine to people aged 60 years or older as part of the main series to strengthen initial protection [65]. Therefore, people should adhere to public health and social measures even though they have received vaccines to avoid COVID-19 infection and its consequences [79].

Strengths and Limitations
This systematic review has some limitations. First, there is no evidence of the longterm effectiveness of the vaccine. Due to the urgency of vaccine development, most trials only followed up participants for 28 days after vaccination. Second, this metanalysis cannot give solid evidence on the efficacy and effectiveness of COVID-19 vaccines on the variant strain B.1.351. This variant strain can escape neutralizing relevant antibodies. Consequently, more studies need to be conducted to assess the effectiveness and efficacy of COVID-19 vaccines against variants of concern like delta strain and omicron [64]. Third, due to scarce of literature, we neither included all approved vaccines nor all age groups (elderly, adolescents, and children).
The points of strength in this systematic review are that we did not include preprinted documents, studies that were not peer-reviewed, and studies with missing data. Due to the scarcity of RCTs, observational studies were included, as were retrospective case analyses. Animal studies were excluded, and we did not have lingual restrictions. Our analysis has identified numerous critical components to consider when planning a real-world efficacy trial of COVID-19 vaccinations, such as the appropriate study design, study population, outcome, and period for follow-up. The majority of studies identified were from HICs, frequently utilizing national databases (which may not exist or may be of lower quality in LMICs), and the vast majority assessed mRNA vaccines, which are more prevalent in HICs. These findings highlight the need for pressing for real-world efficacy studies on all licensed COVID-19 vaccines in a variety of LMIC contexts using different study designs.

Conclusions
This systematic review and meta-analysis summarized the results of clinical trials related to the COVID-19 vaccines, showing that most vaccines had comparable effectiveness and efficacy. It is believed that vaccination can effectively reduce COVID-19 related deaths and severe cases. The incidence of COVID-19, either symptomatic or asymptomatic, decreased significantly after vaccination by one or two doses. However, in the light of the ongoing appearance of novel variants, the efficacy/effectiveness of vaccination against COVID-19 infection needs to be re-assessed.
Supplementary Materials: The following supporting information can be downloaded at: https: //www.mdpi.com/article/10.3390/vaccines10030350/s1, Table S1: PRISMA 2020 Checklist. Table S2: Summary of studies not met inclusion criteria of qualitative and quantitative analysis. Table S3: Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies.

Data Availability Statement:
The data presented in this study are available on request from the corresponding author.

Conflicts of Interest:
The authors declare no conflict of interest.