Next Article in Journal
Micronutrients to Support Vaccine Immunogenicity and Efficacy
Next Article in Special Issue
A Social Network Analysis of Tweets Related to Mandatory COVID-19 Vaccination in Poland
Previous Article in Journal
Factors Associated with Non-Severe Adverse Reactions after Vaccination against SARS-CoV-2: A Cohort Study of 908,869 Outpatient Vaccinations in Germany
Previous Article in Special Issue
Vaccinations and Healthy Ageing: How to Rise to the Challenge Following a Life-Course Vaccination Approach
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Vaccines
Volume 10
Issue 4
10.3390/vaccines10040567
vaccines-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

Monitoring and Evaluation of National Vaccination Implementation: A Scoping Review of How Frameworks and Indicators Are Used in the Public Health Literature

by
Manar Marzouk
1,†,
Maryam Omar
2,†,
Kanchanok Sirison
3,*,
Aparna Ananthakrishnan
3,
Anna Durrance-Bagale
1,4,*,
Chatkamol Pheerapanyawaranun
3,
Charatpol Porncharoen
3,
Nopphadol Pimsarn
3,
Sze Tung Lam
1,
Mengieng Ung
1,
Zeenathnisa Mougammadou Aribou
1,
Saudamini V. Dabak
3,
Wanrudee Isaranuwatchai
3,5 and
Natasha Howard
1,4
1
Saw Swee Hock School of Public Health, National University of Singapore, National University Health System, 12 Science Drive 2, Singapore 117549, Singapore
2
Barts Health NHS Trust, Newham University Hospital, London E13 8SL, UK
3
Health Intervention and Technology Assessment Program, Ministry of Public Health, Nonthaburi 11000, Thailand
4
London School of Hygiene and Tropical Medicine, 15-17 Tavistock Place, London WC1H 9SH, UK
5
Institute of Health Policy, Management and Evaluation, University of Toronto, 155 College St., Toronto, ON M5T 3M6, Canada
*
Authors to whom correspondence should be addressed.
†
These authors contributed equally to this work.
Vaccines 2022, 10(4), 567; https://doi.org/10.3390/vaccines10040567
Submission received: 23 February 2022 / Revised: 1 April 2022 / Accepted: 4 April 2022 / Published: 6 April 2022
(This article belongs to the Special Issue New Insight in Vaccination and Public Health)
Browse Figures
Review Reports Versions Notes

Abstract

:
An effective Monitoring and Evaluation (M&E) framework helps vaccination programme managers determine progress and effectiveness for agreed indicators against clear benchmarks and targets. We aimed to identify the literature on M&E frameworks and indicators used in national vaccination programmes and synthesise approaches and lessons to inform development of future frameworks. We conducted a scoping review using Arksey and O’Malley’s six-stage framework to identify and synthesise sources on monitoring or evaluation of national vaccination implementation that described a framework or indicators. The findings were summarised thematically. We included 43 eligible sources of 4291 screened. Most (95%) were in English and discussed high-income (51%) or middle-income (30%) settings, with 13 in Europe (30%), 10 in Asia-Pacific (23%), nine in Africa (21%), and eight in the Americas (19%), respectively, while three crossed regions. Only five (12%) specified the use of an M&E framework. Most (32/43; 74%) explicitly or implicitly included vaccine coverage indicators, followed by 12 including operational (28%), five including clinical (12%), and two including cost indicators (5%). The use of M&E frameworks was seldom explicit or clearly defined in our sources, with indicators rarely fully defined or benchmarked against targets. Sources focused on ways to improve vaccination programmes without explicitly considering ways to improve assessment. Literature on M&E framework and indicator use in national vaccination programmes is limited and focused on routine childhood vaccination. Therefore, documentation of more experiences and lessons is needed to better inform vaccination M&E beyond childhood.

1. Introduction

Improving national vaccination programme implementation requires collection and analysis of data on relevant vaccination components. Monitoring and evaluation (M&E) or more recent Monitoring, Evaluation, Accountability and Learning (MEAL) frameworks [1] support decision-making by consolidating available information on agreed indicators, benchmarked targets, and methods to collect, analyse, and report necessary data to strengthen vaccination programmes [2]. M&E frameworks are usually aggregated into pre-, peri-, and post-vaccination phases, and include elements of vaccine procurement, transport, storage, staff training, communication, coverage, adverse effects, and identification of successes and failures [3].
Planning effective M&E for national rollout of new vaccines, such as for COVID-19, can be strengthened by learning from previous vaccination experiences, particularly those targeted beyond routine childhood populations [4]. A virtual expert roundtable, hosted by the Saw Swee Hock School of Public Health in January 2021, identified key M&E framework components to inform COVID-19 vaccination. This included best practice guidelines, particularly by the World Health Organization (WHO) [5], but few lessons or experiences of using M&E frameworks and selecting and appropriately benchmarking indicators within vaccination programme M&E. Practical details of these experiences could help governments and technical partners in planning, implementing, and assessing M&E for new vaccine implementation. Lessons learnt from assessment experiences worldwide could help inform national efforts to improve routine and vaccine-specific data collection and analyses and support vaccination programme strengthening, particularly in resource-constrained settings, which aligns with the Immunization Agenda 2030 goal to make vaccination available to everyone, everywhere [6].
We thus aimed to synthesise the literature on M&E frameworks and indicators used for vaccination implementation. The objectives were to: (i) summarise the scope of existing primary literature on M&E frameworks or indicators used; (ii) identify any useful indicators to inform development or adaptation of M&E frameworks; and (iii) synthesise lessons to inform M&E framework development for national rollouts of vaccination.

2. Materials and Methods

2.1. Study Design

We conducted a scoping literature review using Arksey and O’Malley’s six-stage framework with Levac and colleagues’ revisions and Khalil and colleagues’ refinements [7,8,9]. We selected this method because, as Munn et al. suggested, scoping reviews are useful to map and identify evidence in emerging topics and help identify key concepts and gaps [10].

2.2. Identifying the Research Question (Stage 1)

Our research question was: “What are the scope (i.e., extent, distribution, nature), main findings, and key lessons of literature on M&E frameworks and indicators for vaccination implementation?” Table 1 provides our study definitions.

2.3. Identifying Relevant Sources (Stage 2)

To ensure breadth, we included multiple electronic databases and websites. First, we searched five databases systematically (i.e., Medline, Embase, Web of Science, Scopus, Eldis). Second, we searched eight relevant websites purposively (i.e., WHO, Australian Department of Health, National Advisory Committee on Immunization Canada, India Ministry of Health and Family Welfare, Philippines Department of Health, Singapore Ministry of Health, UK Joint Committee on Vaccination and Immunisation, US Centers for Disease Control and Prevention). For both databases and websites, we used search terms for ‘vaccine’ (i.e., vaccin*, immuniz*, immunis*) AND ‘monitoring’ AND ‘evaluation’ (i.e., monitor* and evaluat*, M&E, Monitor*, evaluat*) and related terminology adapted to subject headings.

2.4. Selecting Sources (Stage 3)

We established eligibility criteria based on our research question and discussion with experts (Table 2). We included primary research sources focused on vaccine implementation in national settings and including content on M&E frameworks or indicators. Thus, we also included conference abstracts, commentaries, book chapters, and reviews that provided research data not already included in a research article. We did not exclude on language (if an English abstract was accessible), publication year, study design, or participants.
We screened 4288 potential sources using Covidence and EndNote software. After removing 2089 duplicates, all authors first screened 2199 titles and abstracts against eligibility criteria and excluded 1995 ineligible sources. We then screened 204 full texts, excluding another 163 ineligible sources. We added two eligible website sources to 41 eligible database sources, thus including 43 in total.

2.5. Extracting (Charting) Data (Stage 4)

We extracted data from 43 sources into Excel using the following headings: lead author, publication year, source type (i.e., article, abstract, book, report), language, country/ies included, aim, study information (i.e., design, participants, data collection, analysis), and findings (i.e., M&E tool/framework used, indicators included, lessons described). Indicators were subcategorised as coverage (i.e., targeting, population estimation, equity, disaggregation, uptake/coverage, attitude/behaviours), operational (i.e., health service capacity, human resources, vaccine supply chain (e.g., availability, allocation, transport, storage, delivery, wastage, disposal)), clinical (i.e., vaccine safety, vaccine demand), or others (i.e., costs, additional indicators) as described in the WHO-UNICEF monitoring framework for COVID-19 vaccines [12].

2.6. Collating and Summarising Findings (Stage 5)

First, we summarised sources extent (i.e., database/website origin, publication year), distribution (i.e., publication language, countries included), and nature (i.e., type, topic, study design, outcomes included). Second, we synthesised data thematically under framework and indicator headings guided by our research question and stakeholder consultation.

2.7. Consulting Stakeholders (Stage 6)

We discussed initial review findings with 16 high-level stakeholders in the Thai Ministry of Public Health in December 2021. Stakeholders were asked how findings could be made most useful and for any additional potential sources (none were identified). Inputs informed final synthesis.

3. Results

3.1. Scope of the Literature

Figure 1 provides the PRISMA flow diagram of the 43 eligible sources of 4291 screened. Databases provided 4288 (i.e., 1122 in Medline, 1746 in Embase, 600 in Web of Science, 820 in Scopus, 0 in Eldis) and the UK Joint Committee on Vaccination and Immunisation website provided two [13,14].
Figure 2 shows the extent of sources by publication year. None were found before 1987 or in 1991–2005. From 2006, publications slowly increased, with two notable increases in 2010 and 2012, to a peak of nine in 2018–2019, and then decreased.
Forty single-country sources were distributed across 26 countries, while three multi-country sources included Bahrain, Kuwait, Oman, Qatar, Saudi Arabia, United Arab Emirates [15]; China, Indonesia, Viet Nam [16]; and Bangladesh, Mozambique, Uganda, and Zambia [17]. Over half of sources described high-income settings (22/43; 51%), while 13 (30%) described middle-income settings and only 8 (19%) described low-income settings.
Most (41/43; 95%) were published in English, with one each in French and Spanish. Most (40/43; 93%) were journal articles, and three (7%) were conference abstracts. Study designs and methodology were often unclear, but 34 sources (79%) appeared to use primarily quantitative, six (14%) used mixed-method, and three (7%) used qualitative approaches. Methods were somewhat better described and included surveys, document analysis, observations, interviews, and focus group discussions.

3.2. Synthesised Findings

We synthesised findings under: (i) description of any framework and how it was used; (ii) coverage indicators; (iii) operational indicators; and (iv) clinical indicators. As most sources did not detail the specific frameworks or indicators used, we instead reported on ways they were used and any lessons within each sub-section (Table 3).

3.2.1. Frameworks

Five (12%) sources explicitly described using any type of M&E framework, while the rest either did not use one or were unclear about whether or how one was used and what it included. Thus, framework usage was minimally described and heterogeneous, depending on requirements and objectives. For example, Al Awaidy et al. used the WHO M&E framework for hepatitis B in reviewing vaccination in several Gulf countries [15]. Dang et al. used the mHealth Assessment and Planning for Scale (MAPS) toolkit to assess scale-up of an electronic immunisation registry in Viet Nam [18]. Hutubessy et al. used the Cervical Cancer Prevention and Control Costing (C4P) tool to determine cost-effectiveness of HPV vaccination in Tanzania [19]. Ijsselmuiden et al. used the WHO Extended Programme on Immunisation framework to determine vaccination coverage and cold chain maintenance in South Africa [20]. Non-WHO M&E frameworks included Manyazewal and colleagues’ use of the Plan-Do-Check-Act (PDCA) cycle with Continuous Quality Improvement, a prospective quasi-experimental interrupted time-series design to evaluate effectiveness of a continuous quality improvement intervention for a vaccination programme in Ethiopia [21] and Aceituno and colleagues’ use of a logical framework to determine participant engagement in Bolivia [22].
Four (9%) sources did not specify the use of a formal framework, instead describing an assessment process or method (e.g., use of registers, sampling approaches, surveys). For example, Lanata et al. used lot quality assurance sampling to determine vaccine coverage in Peru [23], and Tuells et al. used a WHO process for cold chain temperature monitoring in Spain [24]. Sources seldom explicitly distinguished between routine data collection sources (e.g., national infectious disease surveillance), dedicated disease-specific registers (e.g., for rabies, tetanus, HIV), general surveys (e.g., national-level demographic and health surveys or multiple-indicator cluster surveys), or vaccine-specific (e.g., post-introduction evaluation surveys).
General lessons included incorporating national staff in monitoring meetings to improve M&E ownership and accountability [22], improving contextual dynamics tailored to the specific vaccination programme to improve coverage [21], and multi-disciplinary co-production and inclusion of M&E staff during decision-making to improve outcomes [17].

3.2.2. Coverage Indicators

Most (32/43; 74%) described elements of target population estimation, equity, and uptake, primarily of routine childhood vaccines. Thirteen (30%) included targeting or population estimation, although most did not describe indicators in depth and how they were used differed by setting and resource availability. For example, Lacapere et al. roughly estimated measles-rubella vaccination coverage by dividing the number of vaccine doses given by the estimated population for each district in Haiti [25], while D’Ancona et al. used an immunisation register to estimate coverage in Italy [26]. Bianco et al. determined the number of foreign workers eligible for vaccination in Italy through screening [27]. Manyazewal et al. used WHO Reaching Every Community (REC) mapping of community locations and characteristics to help estimate coverage targets for five vaccines in Ethiopia [21].
Only three (7%) included indicators to examine equity or disaggregate data. For example, Sarker et al. compared immunisation coverage among children aged 12–59 months in Bangladesh across socioeconomic and demographic factors, finding disparities by parental education and mothers’ access to media [28]. Wattiaux et al. considered equity aspects of vaccination rollout by comparing hepatitis B immunisation incidence between indigenous and non-indigenous Australians [29]. Geoghegan et al. examined whether women received COVID-19 vaccine when pregnant or received routinely recommended vaccines in pregnancy in Ireland [30].
Sixteen sources (37%) discussed uptake indicators. Most focused on general target populations with minimal discussion on vaccine coverage for migrants and refugees and none on the elderly, people with disabilities, or other potentially vulnerable groups. However, Bawa et al. estimated oral polio vaccination coverage among underserved hard-to-reach communities in Nigeria, defining them based on difficulty of terrain, any local or state border, scattered households, nomadic, water-logged/riverine, or conflict-affected and thus requiring outreach services [31]. More generally, Lacapere et al. calculated numbers of municipalities that reported achieving 95% vaccination coverage for measles-rubella vaccine in Haiti [25]. Only two sources (7%) described coverage indicators beyond the second year of life. Muhamad et al. calculated total HPV vaccine doses delivered through school-based outreach to evaluate the effectiveness of free vaccination for schoolgirls in Malaysia [32], while Beard et al. estimated 40–50% coverage of pertussis vaccination among pregnant women in Australia, based on number of births and consent forms returned centrally [33].
Coverage lessons were varied. Aceituno et al. described challenges of collecting high-quality data in resource-constrained settings [22]. Soi et al. noted that using feedback loops to guide policy decision must be pragmatic, as they are often too slow—e.g., Gavi’s HPV demonstration project policy required countries to demonstrate adequate coverage before applying for rollout funding, which could take years [17]. Alam et al. found automation of EPI scheduling can improve coverage and enhance monitoring, particularly in remote areas [34]. Edelstein et al. found that data triangulation and inclusion of routine data, in countries with good national records, can help identify vulnerable groups and monitoring of vaccine coverage [35]. Lanata et al. found lot quality assurance sampling helped identify small areas with poorer vaccination coverage in rural areas of Peru with dispersed populations, thus improving coverage and equity monitoring [23]. Aceituno et al. found staff understanding of cultural-linguistic context improved coverage and vaccination continuity in Bolivia [22].

3.2.3. Operational Indicators

Only two (5%) sources mentioned health service capacity indicators. Manyazewal et al. assessed immunisation services availability, regular static immunisation services delivered, adequate outreach sites, catchment area mapped for immunisation, separate and adequate rooms for immunisation services and storing supplies, all planned outreach sessions conducted, health education on immunisation provided, and immunisation services availability in all catchment health posts in Ethiopia [21]. Walker et al. assessed surveillance feedback reports, timely reporting, and number of districts with populations not receiving immunisation services [36].
Three (7%) sources mentioned supply chain and logistics indicators. Walker et al. used cold chain and logistics data from facility inventory logs for routine immunisation to identify gaps in vaccine supplies and equipment, such as the number of facilities with insufficient supply of syringes and diluent, and number of facilities with inventory logs consistent with vaccine supply [36]. Hipgrave et al. reviewed evidence on thermostability of hepatitis B vaccine for pregnant women when stored outside the cold chain in China [16]. Özdemir et al. assessed cold chain storage and gaps for a hepatitis B vaccine in Turkey [37]. Manyazewal et al. assessed adequacy of fridge-tag 2 units for temperature monitoring, refrigerator spare parts, vaccine request and report forms, and inventory documents in Ethiopia [21].
Five (12%) sources mentioned human resource indicators. Hall et al. assessed the number of English health-workers vaccinated against COVID-19 stratified by dose, manufacturer, and day [13]. Cherif et al. assessed numbers of health-workers participating in vaccination activities, e.g., epidemiological surveillance, adverse event monitoring training, and supervisions in Abidjan [38]. Carrico et al. assessed numbers of states mandating health-worker vaccination in the US [39]. Manyazewal et al. assessed numbers of experts assigned for immunisation and numbers of immunisation focal persons to evaluate effectiveness of system-wide continuous quality improvement for national immunisation programme performance [21]. Walker et al. assessed numbers of supervisory visits conducted, documented in writing, surveillance guidelines observed, surveillance discussed at supervisory visits, and if an operational plan was observed [36].
Two (5%) sources included indicators for vaccination costing. Hutubessy et al. calculated the incremental costs to the health system of HPV vaccination for adolescent girls through schools, health facilities, and other outreach strategies in Tanzania [20]. Walker et al. measured the number of districts in Kenya with insufficient financial resources for key surveillance elements for acute flaccid paralysis [36].
Multiple sources discussed operational lessons. D’Ancona et al. used an observational survey to show that decentralised health systems such as in Italy could result in fragmented immunisation registries and information flow across regions [26]. Ward et al. described how data improvement teams, allocated to all districts, enhanced the quality of vaccine administrative data in Uganda by helping identify data inaccuracies and providing on-the-job data collection training [40]. Dang et al. used qualitative research to describe an approach to optimise vaccination information in Vietnam, which included establishing a partnership between the Vietnamese Ministry of Health and mobile network operators [17]. Soi et al. described the importance of physically co-locating evaluators from different disciplinary backgrounds, and suggested that including evaluators in decision-making could enrich outcomes [17].

3.2.4. Clinical Indicators

Five (12%) sources described clinical indicators, primarily counting numbers of adverse events following immunisation (AEFI). For example, Aceituno et al. assessed monthly reporting of adverse events and severe adverse events, details of any deaths, reasons for all withdrawals, and infant and maternal death rates below Demographic and Health Survey rates for Bolivia [22]. Loughlin et al. conducted a post-marketing evaluation to assess the number of confirmed cases of intussusception or Kawasaki disease among infants who received Rotavirus vaccine in the US compared with historical cohort data from diphtheria-tetanus-acellular pertussis vaccination [41].
Lessons were relatively limited. Vivekanandan et al. described the positive role of health-workers in assessing vaccine safety indicators in India [42]. Cherif et al. similarly noted that improving AEFI system performance required improved health-worker training, data analysis, and community engagement [38]. Ijsselmuiden et al. suggested that vaccination targeting at-risk populations, such as for Hepatitis B, should ensure concomitant disease surveillance to reduce morbidity and mortality [20]. Similarly, Beard et al. suggested combining AEFI and syndromic surveillance in emergency departments to monitor numbers of pertussis vaccine adverse events and supplementing maternal influenza vaccination AEFI monitoring with mobile phone text messages in Australia [33].

4. Discussion

This initial review of the use of M&E frameworks and indicators in vaccination highlights the relatively limited literature on this topic. M&E frameworks are important for consolidating selected indicators and described as essential in the Global Vaccine Action Plan (GVAP) [4], yet their use was seldom explicit or clearly defined in our sources. Most sources described assessment methods (e.g., survey, lot quality assurance) rather than the use of a formal M&E framework, suggesting overreliance on individual methods without the benefit of an overarching assessment framework. Similarly, while indicators were described more frequently, they were rarely fully defined or benchmarked against targets, and sources focused on ways to improve vaccination programmes without explicitly considering ways to improve assessment. Given indicators and benchmarked targets are crucial to national vaccination programme M&E these are noteworthy gaps.
Limited description of M&E framework or indicator use outside routine childhood vaccination was perhaps unsurprising and some of the lessons identified in our review could inform development of monitoring or evaluation for COVID-19 vaccination, or other vaccines, beyond routine childhood populations [43]. It is worth reiterating that most sources described high- or middle-income settings, and none described the use of M&E frameworks or indicators in fragile or conflict-affected settings despite the risk of poorer routine vaccination coverage, weakened health system responses, and infectious disease outbreaks in these settings. For example, 14 million ‘zero-dose’ children, who did not receive an initial dose of required vaccines, live in conflict-affected African countries [44,45]. The limited documentation of equity indicators was unexpected given that improving equity in immunisation is both essential [46] and expected by donors such as Gavi [47], yet equity monitoring seemed limited and ill defined. Historical and socio-cultural influences and biases can influence the effectiveness of data collection and assessment on equity, and thus vaccination programme success. For instance, if some high-risk cohorts are not considered acceptable or relevant and hence not recorded (e.g., men who have sex with men in Gulf states), accurate and detailed assessment becomes impossible [15]. An approach used by M’Bangombe et al. analysed historical data to augment current data on high-risk populations, expanding the cohort of those identified as being at risk for cholera in Malawi [48].
Another surprising gap was the limited documentation of operational indicators, given their importance in effective vaccination management. The Organisation for Economic Co-operation and Development (OECD) report on lessons from government evaluations of COVID-19 responses similarly highlighted gaps in cost and health system indicators [43]. Additionally, we found that sources used different types of routine and ad hoc data, including surveillance, disease registers, household surveys, and vaccine-specific surveys. While this is understandable, depending on setting and programming needs [49], justification was not always explicit.
Less surprisingly, our review showed a preference for quantitative assessment methods, with only three sources using qualitative and five using mixed methods. However, qualitative and mixed-method social science approaches offer deeper insights into how processes work and are particularly useful for equity analyses. For example, Dutta et al. interviewed vaccination decision-makers in India to examine the importance of engaging with communities to promote health equity, and found this required formulating policies and guidelines that clearly define community engagement and its related evaluation metrics [50]. Qualitative methods can also help amplify perspectives and groups that may be less visible, which can be particularly important in reaching zero-dose children [45].
What is often missing in evaluations is the impact of vaccination on the general population, particularly for lower efficacy vaccines (e.g., against malaria, cholera, or influenza) that have relatively low demographic impacts even with high vaccine coverage. It may be worth exploring this demographic impact further, as was observed for smallpox vaccination [51,52]. Therefore, further documentation of assessment methods, experiences, and lessons appears necessary to expand the evidence base and help inform ongoing and future vaccination assessment.
Several potential limitations should be considered. First, while we included five databases and eight websites, we may still have missed relevant sources. It is likely that much of the research on this topic remains unpublished, as evaluations conducted by non-academic bodies (e.g., government, consultants) may not be in the public domain, though sites such as bioRxiv could be useful for manuscripts awaiting peer review. Second, included sources were not assessed for quality, as the purpose was to scope existing literature, and this would have eliminated too many documents. Third, we excluded sources on vaccine development, so may have missed some that discussed vaccine safety. Fourth, many sources only assessed one or more components of the vaccination programme (e.g., financing, equity, personnel), and thus we did not attempt to make direct comparisons of assessments. Fifth, methods were often insufficiently described, so we chose to categorise broadly as “quantitative, qualitative, mixed-methods” approaches rather than trying to provide more detail. Sixth, we focused on the public health literature as most likely to contain M&E frameworks and indicators, rather than conducting a broader search of various social science literatures. We thus may have missed some qualitative frameworks or indicators, e.g., for vaccine hesitancy. Finally, we chose not to include MEAL framework components for advocacy or learning, which could be relevant for future research.
Overall, our review identified minimal literature describing M&E frameworks or indicators for use in vaccination programme implementation. Numbers of relevant publications increased during the past decade, particularly after the 2012 Middle East Respiratory Syndrome epidemic, but numbers are still small and focused on high- and middle-income countries. Further research and documentation are therefore needed to identify additional public health lessons.

Funding

Study funding was provided by the Health Systems Research Institute of Thailand (grant number HSRI 64-061). K.S., A.A., C.K., N.P., C.P., S.D., and W.I. are employed by the Health Intervention and Technology Assessment Program (HITAP), a semi-autonomous research unit of the Thai Ministry of Public Health and funded by national and international public funding agencies, including the Access and Delivery Partnership—hosted by the United Nations Development Programme and funded by the Government of Japan. HITAP’s international work is supported by the International Decision Support Initiative (iDSI), funded by the Bill & Melinda Gates Foundation (OPP1202541), the UK Foreign, Commonwealth, and Development Office, and Rockefeller Foundation. Funders were not involved in study implementation, analysis, or preparation of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Please contact the corresponding author for data requests.

Acknowledgments

We wish to thank Russell Burke, Assistant Librarian at the London School of Hygiene and Tropical Medicine, for invaluable help in developing database search structures and Sirikorn Sujinnaprum for helping with title/abstract screening.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

References

  1. WHO. COVID-19 Strategic Preparedness and Response Plan. Monitoring and Evaluation Framework; World Health Organization: Geneva, Switzerland, 2021. [Google Scholar]
  2. ECDC. Monitoring and Evaluation Framework for COVID-19 Response Activities in the EU/EEA and the UK. 2020. Available online: https://www.ecdc.europa.eu/en/publications-data/covid-19-monitoring-and-evaluation-framework-response-activities (accessed on 22 February 2022).
  3. WHO. The Monitoring and Evaluation/Accountability Framework; World Health Organization: Geneva, Switzerland.
  4. WHO. Assessing and Improving the Accuracy of Target Population Estimates for Immunization Coverage; World Health Organization: Geneva, Switzerland, 2015. [Google Scholar]
  5. World Health Organisation. Immunization Agenda 2030: A Global Strategy to Leave No One Behind. 2020. Available online: https://www.who.int/teams/immunization-vaccines-and-biologicals/strategies/ia2030 (accessed on 22 February 2022).
  6. IA2030. Immunization Agenda 203: Monitoring and Evaluation Framework 2021; World Health Organization: Geneva, Switzerland, 2021. [Google Scholar]
  7. Arksey, H.; O’Malley, L. Scoping studies: Towards a methodological framework. Int. J. Soc. Res. Methodol. 2005, 8, 19–32. [Google Scholar] [CrossRef] [Green Version]
  8. Khalil, H.; Peters, M.; Godfrey, C.M.; McInerney, P.; Soares, C.B.; Parker, D. An Evidence-Based Approach to Scoping Reviews. Worldviews Evid. Based. Nurs. 2016, 13, 118–123. [Google Scholar] [CrossRef]
  9. Levac, D.; Colquhoun, H.; O’Brien, K.K. Scoping studies: Advancing the methodology. Implement. Sci. 2010, 5, 69. [Google Scholar] [CrossRef] [Green Version]
  10. Munn, Z.; Peters, M.D.J.; Stern, C.; Tufanaru, C.; McArthur, A.; Aromataris, E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med. Res. Methodol. 2018, 18, 143. [Google Scholar]
  11. CDC. Immunization: The Basics Definition of Terms. 2018. Available online: https://www.cdc.gov/vaccines/vac-gen/imz-basics.htm (accessed on 22 February 2022).
  12. WHO-UNICEF. Monitoring COVID-19 Vaccination Considerations for the Collection and Use of Vaccination Data. 2021. Available online: https://apps.who.int/iris/bitstream/handle/10665/339993/WHO-2019-nCoV-vaccination-monitoring-2021.1-eng.pdf (accessed on 22 February 2022).
  13. Hall, V.J.; Foulkes, S.; Saei, A.; Andrews, N.; Oguti, B.; Charlett, A.; Wellington, E.; Stowe, J.; Gillson, N.; Atti, A.; et al. COVID-19 vaccine coverage in health-care workers in England and effectiveness of BNT162b2 mRNA vaccine against infection (SIREN): A prospective, multicentre, cohort study. Lancet 2021, 397, 1725–1735. [Google Scholar] [CrossRef]
  14. Bernal, J.L.; Andrews, N.; Gower, C.; Robertson, C.; Stowe, J.; Tessier, E.; Simmons, R.; Cottrell, S.; Roberts, R.; O’Doherty, M.; et al. Effectiveness of the Pfizer-BioNTech and Oxford-AstraZeneca vaccines on covid-19 related symptoms, hospital admissions, and mortality in older adults in England: Test negative case-control study. BMJ 2021, 373, n1088. [Google Scholar] [CrossRef]
  15. Al Awaidy, S.T.; Ezzikouri, S. Moving towards hepatitis B elimination in Gulf Health Council states: From commitment to action. J. Infect. Public Health 2020, 13, 221–227. [Google Scholar] [CrossRef]
  16. Hipgrave, D.B.; Maynard, J.E.; Biggs, B.A. Improving birth dose coverage of hepatitis B vaccine. Bull. World Health Organ. 2006, 84, 65–71. [Google Scholar] [CrossRef]
  17. Soi, C.; Shearer, J.C.; Budden, A.; Carnahan, E.; Salisbury, N.; Asiimwe, G.; Chilundo, B.; Sarma, H.; Gimbel, S.; Simuyemba, M.; et al. How to evaluate the implementation of complex health programmes in low-income settings: The approach of the Gavi Full Country Evaluations. Health Policy Plan 2020, 35 (Suppl. S2), ii35–ii46. [Google Scholar] [CrossRef]
  18. Dang, H.; Dao, S.; Carnahan, E.; Kawakyu, N.; Duong, H.; Nguyen, T.; Nguyen, D.; Nguyen, L.; Rivera, M.; Ngo, T.; et al. Determinants of Scale-up From a Small Pilot to a National Electronic Immunization Registry in Vietnam: Qualitative Evaluation. J. Med. Internet Res. 2020, 22, e19923. [Google Scholar] [CrossRef]
  19. Hutubessy, R.; Levin, A.; Wang, S.; Morgan, W.; Ally, M.; John, T.; Broutet, N. A case study using the United Republic of Tanzania: Costing nationwide HPV vaccine delivery using the WHO Cervical Cancer Prevention and Control Costing Tool. BMC Med. 2012, 10, 136. [Google Scholar] [CrossRef] [Green Version]
  20. Ijsselmuiden, C.B.; Küstner, H.G.; Barron, P.M.; Steinberg, W.J. Notification of five of the EPI target diseases in South Africa. An assessment of disease and vaccination reporting. South Afr. Med. J. 1987, 72, 311–316. [Google Scholar]
  21. Manyazewal, T.; Mekonnen, A.; Demelew, T.; Mengestu, S.; Abdu, Y.; Mammo, D.; Abebe, W.; Haffa, B.; Zenebe, D.; Worku, B.; et al. Improving immunization capacity in Ethiopia through continuous quality improvement interventions: A prospective quasi-experimental study. Infect. Dis. Poverty 2018, 7, 119. [Google Scholar] [CrossRef] [Green Version]
  22. Aceituno, A.M.; Stanhope, K.K.; Rebolledo, P.A.; Burke, R.M.; Revollo, R.; Iniguez, V.; Suchdev, P.S.; Leon, J.S. Using a monitoring and evaluation framework to improve study efficiency and quality during a prospective cohort study in infants receiving rotavirus vaccination in El Alto, Bolivia: The Infant Nutrition, Inflammation, and Diarrheal Illness (NIDI) study. BMC Public Health 2017, 17, 911. [Google Scholar] [CrossRef] [Green Version]
  23. Lanata, C.F.; Stroh, G.; Black, R.E.; Gonzales, H. An evaluation of Lot Quality Assurance sampling to monitor and improve immunization coverage. Int. J. Epidemiol. 1990, 19, 1086–1090. [Google Scholar] [CrossRef]
  24. Tuells, J. The fragile beginning of the vaccine cold chain in Spain. Gac Sanit. 2010, 24, 354–357. [Google Scholar] [CrossRef] [Green Version]
  25. Lacapere, F.; Magloire, R.; Danovaro-Holliday, M.C.; Flannery, B.; Chamoulliet, H.; Celestin, E.P. The use of rapid coverage monitoring in the national rubella vaccination campaign, Haiti 2007-2008. J. Infect. Dis. 2011, 204 (Suppl. 2), S698–S705. [Google Scholar] [CrossRef]
  26. D’Ancona, F.; Gianfredi, V.; Riccardo, F.; Iannazzo, S. Immunisation Registries at regional level in Italy and the roadmap for a future Italian National Registry. Ann. Ig. 2018, 30, 77–85. [Google Scholar]
  27. Bianco, P.; Ieraci, R.; Comito, M.; Anzelmo, V. Controindications and adverse events of immunizations at workplace. Multidiscipunary management in workers for 2005–2010. G. Ital. Di Med. Del Lav. Ed Ergonomia 2012, 34, 631–634. (In Italian) [Google Scholar]
  28. Sarker, A.R.; Akram, R.; Ali, N.; Sultana, M. Coverage and factors associated with full immunisation among children aged 12-59 months in Bangladesh: Insights from the nationwide cross-sectional demographic and health survey. BMJ Open 2019, 9, e028020. [Google Scholar] [CrossRef]
  29. Wattiaux, A.L.; Yin, J.K.; Beard, F.; Wesselingh, S.; Cowie, B.; Ward, J.; Macartney, K. Hepatitis B immunization for indigenous adults, Australia. Bull. World Health Organ. 2016, 94, 826–834. [Google Scholar] [CrossRef] [PubMed]
  30. Geoghegan, S.; Stephens, L.C.; Feemster, K.A.; Drew, R.J.; Eogan, M.; Butler, K.M. This choice does not just affect me. Attitudes of pregnant women toward COVID-19 vaccines: A mixed-methods study. Hum. Vaccines Immunother. 2021, 17, 3371–3376. [Google Scholar] [CrossRef] [PubMed]
  31. Bawa, S.; McNab, C.; Nkwogu, L.; Braka, F.; Obinya, E.; Galway, M.; Mirelman, A.J.; Hammanyero, K.I.; Safiyanu, G.; Chukwuji, M.; et al. Using the polio programme to deliver primary health care in Nigeria: Implementation research. Bull. World Health Organ. 2019, 97, 24–32. [Google Scholar] [CrossRef] [PubMed]
  32. Muhamad, N.A.; Buang, S.N.; Jaafar, S.; Jais, R.; Tan, P.S.; Mustapha, N.; Aris, T.; Sulaiman, L.H.; Murad, S. Achieving high uptake of human papillomavirus vaccination in Malaysia through school-based vaccination programme. BMC Public Health 2018, 18, 1402. [Google Scholar] [CrossRef] [PubMed]
  33. Beard, F.H. Pertussis immunisation in pregnancy: A summary of funded Australian state and territory programs. Commun. Dis. Intell. Q Rep. 2015, 39, E329–E336. [Google Scholar] [PubMed]
  34. Alam, A.; Capoor, A.K.; Rao, L.V. Evaluation of adjuvanticity of promising new synthetic MDP analogues. Immunol. Lett. 1991, 27, 53–57. [Google Scholar] [CrossRef]
  35. Edelstein, M.; White, J.; Bukasa, A.; Saliba, V.; Ramsay, M. Triangulation of measles vaccination data in the United Kingdom of Great Britain and Northern Ireland. Bull. World Health Organ. 2019, 97, 754–763. [Google Scholar] [CrossRef]
  36. Walker, A.T.; Sodha, S.; Warren, W.C.; Sergon, K.; Kiptoon, S.; Ogange, J.; Ahmeda, A.H.; Eshetu, M.; Corkum, M.; Pillai, S.; et al. Forewarning of poliovirus outbreaks in the Horn of Africa: An assessment of acute flaccid paralysis surveillance and routine immunization systems in Kenya. J. Infect. Dis. 2014, 210 (Suppl. S1), S85–S90. [Google Scholar] [CrossRef] [Green Version]
  37. Özdemir, H.; Özer, A.Y. Investigating cold-chain system and efficacy of vaccines reaching the end user in Turkey and related regulations. Fabad J. Pharm. Sci. 2010, 35, 93–104. [Google Scholar]
  38. Cherif, D.; Ken, N.G.; Coulibaly, D.; Zengbe-Acray, P.; Ekra, K.D.; Traore, A.; Tiembré, I. Evaluation of epidemiological monitoring of post-vaccination adverse reactions in Abidjan. Sante Publique 2018, 30, 411–417. (In French) [Google Scholar]
  39. Carrico, R.M.; Sorrells, N.; Westhusing, K.; Wiemken, T. Monitoring of health care personnel employee and occupational health immunization program practices in the United States. Am. J. Infect. Control 2014, 42, 66–68. [Google Scholar] [CrossRef] [PubMed]
  40. Ward, K.; Mugenyi, K.; Benke, A.; Luzze, H.; Kyozira, C.; Immaculate, A.; Tanifum, P.; Kisakye, A.; Bloland, P.; MacNeil, A. Enhancing Workforce Capacity to Improve Vaccination Data Quality, Uganda. Emerg. Infect. Dis. 2017, 23 (Suppl. 1), S85. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  41. Loughlin, J.; Mast, T.C.; Doherty, M.C.; Wang, F.T.; Wong, J.; Seeger, J.D. Postmarketing evaluation of the short-term safety of the pentavalent rotavirus vaccine. Pediatric Infect. Dis. J. 2012, 31, 292–296. [Google Scholar] [CrossRef] [PubMed]
  42. Vivekanandan, K.; Prakash, J.; Singh, G.N. Pharmacovigilance programme of India. Arch. Pharm. Pract. 2012, 3, 229–232. [Google Scholar] [CrossRef]
  43. OECD. First Lessons from Government Evaluations of COVID-19 Responses: A Synthesis. 2022. Available online: https://www.oecd.org/coronavirus/policy-responses/first-lessons-from-government-evaluations-of-covid-19-responses-a-synthesis-483507d6/ (accessed on 22 February 2022).
  44. WHO. Progress and Challenges with Achieving Universal Immunization Coverage. 2020. Available online: https://www.who.int/immunization/monitoring_surveillance/who-immuniz.pdf (accessed on 22 February 2022).
  45. Greene, J.C. Qualitative program evaluation. Handb. Qual. Res. 1994, 530, 544. [Google Scholar]
  46. WHO. Equity in Immunisation. 2022. Available online: https://www.euro.who.int/en/health-topics/disease-prevention/vaccines-and-immunization/activities/immunization-systems/equity-in-immunization (accessed on 22 February 2022).
  47. The Vaccine Alliance. The Equity Goal. 2021. Available online: https://www.gavi.org/our-alliance/strategy/phase-5-2021-2025/equity-goal (accessed on 22 February 2022).
  48. M’Bangombe, M.; Pezzoli, L.; Reeder, B.; Kabuluzi, S.; Msyamboza, K.; Masuku, H.; Ngwira, B.; Cavailler, P.; Grandesso, F.; Palomares, A.; et al. Oral cholera vaccine in cholera prevention and control, Malawi. Bull. World Health Organ. 2018, 96, 428–435. [Google Scholar] [CrossRef]
  49. Teerawattananon, Y.; Anothaisintawee, T.; Pheerapanyawaranun, C.; Botwright, S.; Akksilp, K.; Sirichumroonwit, N.; Budtarad, N.; Isaranuwatchai, W. A systematic review of methodological approaches for evaluating real-world effectiveness of COVID-19 vaccines: Advising resource-constrained settings. PLoS ONE 2022, 17, e0261930. [Google Scholar] [CrossRef]
  50. Dutta, T.; Meyerson, B.E.; Agley, J.; Barnes, P.A.; Sherwood-Laughlin, C.; Nicholson-Crotty, J. A qualitative analysis of vaccine decision makers’ conceptualization and fostering of ‘community engagement’ in India. International. J. Equity Health 2020, 19, 185. [Google Scholar] [CrossRef]
  51. Guignard, A.; Praet, N.; Jusot, V.; Bakker, M.; Baril, L. Introducing new vaccines in low- and middle-income countries: Challenges and approaches. Expert Rev. Vaccines 2019, 18, 119–131. [Google Scholar] [CrossRef] [Green Version]
  52. Bonanni, P. Demographic impact of vaccination: A review. Vaccine 1999, 17 (Suppl. S3), S120–S125. [Google Scholar] [CrossRef]
Vaccines 10 00567 g001 550
Figure 1. PRISMA flow diagram.
Figure 1. PRISMA flow diagram.
Vaccines 10 00567 g001
Vaccines 10 00567 g002 550
Figure 2. Number of sources by publication year.
Figure 2. Number of sources by publication year.
Vaccines 10 00567 g002
Table
Table 1. Study definitions.
Table 1. Study definitions.
TermsDefinitions
EvaluationThe systematic assessment of an activity, project, programme, strategy, policy, topic, theme, sector, operational area or institution’s performance to determine its relevance, effectiveness, efficiency, impact, and/or sustainability [11].
FrameworkShows how the programme or activity is intended to work by organising out the components of the initiative and the order or the steps needed to achieve the desired results. A framework increases understanding of the programme’s goals and objectives, defines the relationships between factors key to implementation, and articulates the internal and external elements that could affect the programme’s success.
ImmunisationA process by which a person becomes protected against a disease through vaccination or recovery from infection [11].
MonitoringThe systematic process of collecting, analysing, and using information to track progress toward objectives and guide management decisions [10].
M&E framework A matrix compiling goal/purpose, outcomes, and outputs, along with the defined and measurable indicators with specified targets/thresholds necessary to achieve success.
VaccinationThe management and administration of vaccines pre-/peri-/post-vaccination to provide people with the most effective immunisation [11].
VaccineA product, usually administered through needle injection, by mouth, or sprayed into the nose, that stimulates a person’s immune system to produce immunity to a specific disease, protecting the person from that disease [11].
Table
Table 2. Eligibility criteria.
Table 2. Eligibility criteria.
CriteriaIncludedExcluded
1. Context
  • National settings
  • Other (e.g., subnational, international)
2. Topic
  • Vaccine implementation
  • Vaccine production
  • Vaccine R&D
3. Outcomes
  • Framework
  • Indicators
  • Lessons
  • Impact
  • Other
4. Source type
  • Primary research articles
  • Review articles that include studies not included individually
  • Commentaries/editorials if they include primary research
  • Conference abstracts that include primary research
  • Book chapters that include primary research
  • Audio/video reports
  • Conference abstracts covering the same material as an available publication
  • Social media, blogs, media articles
  • Guidance/legal documents
5. Time-period
  • Any
  • NA
6. Language
  • All languages
  • NA
7. Study design
  • Any
  • NA
8. Participants
  • Any
  • NA
Table
Table 3. Synthesised findings by source.
Table 3. Synthesised findings by source.
Lead Author, YearTypeCountry/iesApproachM&E FrameworkCoverage IndicatorsOperational IndicatorsClinical IndicatorsLessons Learnt
Targeting/EstimationEquityUptakeService CapacityVaccine Supply Human ResourcesM&E CostsVaccine SafetyVaccine Demand
Aceituno, 2017ArticleBoliviaQuantitativeX X X X
Al Awaidy, 2020Articlemulti (Bahrain, Kuwait, Oman, Qatar, Saudi Arabia, United Arab Emirates)QuantitativeX X X
Alam, 2018ArticleBangladeshQuantitative X X
Ashish, 2017ArticleNepalQuantitative X X
Bawa, 2019ArticleNigeria Quantitative X X X
Beard, 2015ArticleAustraliaQuantitative X X X X
Bednarczyk, 2019ArticleUSQuantitative X
Bernal, 2021ArticleUKQuantitative X
Bhatnagar, 2016ArticleIndiaQuantitative X
Bianco, 2012AbstractItalyQuantitative X X
Carrico, 2014ArticleUSMixed X X
Checchi, 2019ArticleUKQuantitative X X X
Cherif, 2018ArticleIvory CoastQuantitative X X
Cutts, 1988ArticleMozambiqueQuantitative X
D’Ancona, 2018ArticleItalyQuantitative X X
Dang, 2020ArticleViet NamQuantitativeX X
Edelstein, 2019ArticleUKQuantitative X X
Geoghegan, 2021AbstractIrelandQualitative X
Hall, 2021ArticleUKQuantitative X X
Hipgrave, 2006Articlemulti (China, Indonesia, Viet Nam)Quantitative X
Hutubessy, 2012ArticleTanzaniaQuantitative X X
Ijsselmuiden, 1987ArticleSouth AfricaMixed X X
Imoukhuede, 2007ArticleGambiaQuantitative X
Lacapere, 2011ArticleHaitiQuantitative X X X
Lanata, 1990ArticlePeruQuantitative X X
Loughlin, 2012ArticleUSQuantitative X X
Maina, 2017ArticleKenyaMixed X X
Manyazewal, 2018ArticleEthiopiaMixedXX XXXX X
McCarthy, 2013ArticleUSQuantitative X X
Muhamad, 2018ArticleMalaysiaQuantitative X X
Ozdemir, 2010ArticleTurkeyQuantitative X X X
Raji, 2019AbstractNigeriaQualitative X
Richard, 2008ArticleSwitzerlandQuantitative X X
Sarker, 2019ArticleBangladeshQuantitative X X
Soi, 2020Articlemulti (Bangladesh, Mozambique, Uganda, Zambia)QuantitativeX X
Tanton, 2017ArticleUKMixed X X
Tuells, 2010ArticleSpainQualitative X
van Wijhe, 2018ArticleNetherlandsQuantitative X X X
Vivekanandan, 2012ArticleIndiaQuantitative X
Walker, 2014ArticleKenya Quantitative XXXX X
Ward, 2017ArticleUgandaQuantitative X
Watson, 2010ArticleUSQuantitative X
Wattiaux, 2016ArticleAustraliaMixed XX X
Totals51331623525039
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Marzouk, M.; Omar, M.; Sirison, K.; Ananthakrishnan, A.; Durrance-Bagale, A.; Pheerapanyawaranun, C.; Porncharoen, C.; Pimsarn, N.; Lam, S.T.; Ung, M.; et al. Monitoring and Evaluation of National Vaccination Implementation: A Scoping Review of How Frameworks and Indicators Are Used in the Public Health Literature. Vaccines 2022, 10, 567. https://doi.org/10.3390/vaccines10040567

AMA Style

Marzouk M, Omar M, Sirison K, Ananthakrishnan A, Durrance-Bagale A, Pheerapanyawaranun C, Porncharoen C, Pimsarn N, Lam ST, Ung M, et al. Monitoring and Evaluation of National Vaccination Implementation: A Scoping Review of How Frameworks and Indicators Are Used in the Public Health Literature. Vaccines. 2022; 10(4):567. https://doi.org/10.3390/vaccines10040567

Chicago/Turabian Style

Marzouk, Manar, Maryam Omar, Kanchanok Sirison, Aparna Ananthakrishnan, Anna Durrance-Bagale, Chatkamol Pheerapanyawaranun, Charatpol Porncharoen, Nopphadol Pimsarn, Sze Tung Lam, Mengieng Ung, and et al. 2022. "Monitoring and Evaluation of National Vaccination Implementation: A Scoping Review of How Frameworks and Indicators Are Used in the Public Health Literature" Vaccines 10, no. 4: 567. https://doi.org/10.3390/vaccines10040567

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop