Next Article in Journal
COVID-19 Booster Uptake among First Responders and Their Household Members May Be Lower than Anticipated
Next Article in Special Issue
Public Vaccination Reluctance: What Makes Us Change Our Minds? Results of A Longitudinal Cohort Survey
Previous Article in Journal
Effectiveness of COVID-19 Vaccination in Preventing All-Cause Mortality among Adults during the Third Wave of the Epidemic in Hungary: Nationwide Retrospective Cohort Study
Previous Article in Special Issue
Factors Influencing the Knowledge Gap regarding Influenza and Influenza Vaccination in the Context of COVID-19 Pandemic: A Cross-Sectional Survey in China
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:

Prioritization of Vaccines for Inclusion into China’s Expanded Program on Immunization: Evidence from Experts’ Knowledge and Opinions

National Immunization Program, Chinese Center for Disease Control and Prevention, Beijing 100050, China
Shanghai Municipal Center for Disease Control and Prevention, Shanghai 201204, China
Zhuhai Center for Disease Control and Prevention, Zhuhai 519000, China
Chinese Field Epidemiology Training Program (CFETP), Chinese Center for Disease Control and Prevention, Beijing 100050, China
Division of Infectious Disease, Chinese Center for Disease Control and Prevention, Beijing 100050, China
Author to whom correspondence should be addressed.
These authors contributed equally to this work.
Vaccines 2022, 10(7), 1010;
Received: 3 June 2022 / Revised: 20 June 2022 / Accepted: 21 June 2022 / Published: 24 June 2022
(This article belongs to the Special Issue New Insight in Vaccination and Public Health)


Background: Vaccine developers in China have made an increasing number of infectious diseases preventable through vaccination. An appropriate decision-making procedure is necessary for making wise decisions on whether to introduce new vaccines into the Expanded Program on Immunization (EPI). When there are several vaccines that could potentially be considered, a scientifically justifiable mechanism is needed for prioritizing and sequencing vaccines for consideration. Methods: We used a modified Delphi technique (MDT) to develop and refine an indicator system to prioritize vaccines and make policy recommendations concerning their introduction into China’s EPI system. From January through May 2021, thirty-nine experts were recruited and participated in a two-round Delphi survey that was based on a set of candidate indicators obtained through a literature review and reference to the WHO vaccine introduction recommendations. Using the resulting indicator system, we conducted a third consultation with a multi-disciplinary group of experts who scored five program-eligible candidate vaccines to determine prioritization and sequencing for consideration of inclusion into the EPI. Results: Response rates of the thirty-nine experts were 100% and 97.4% across the two rounds. Authority coefficients from rounds one to three were over 0.70, reflecting the high accuracy and reliability of the consultation. Coordination coefficients of importance scores for primary, secondary, and tertiary indicators were 0.486, 0.356, 0.275 in round one, and 0.405, 0.340, and 0.236 in round two. According to the scores from 30 experts using our indicator system, the sequence and scores (1–10 scale, 10 highest) of 5 candidate vaccines were varicella (6.91), meningococcal conjugate AC (6.83), Hib (6.74), influenza (6.56), and EV71 (6.17) vaccines. Conclusions: A modified Delphi technique effectively built a scientific, rational, comprehensive, and systematic indicator system for prioritizing vaccine candidates for consideration of inclusion into the EPI. The rank order will be used by the technical working groups of China’s National Immunization Advisory Committee to sequentially develop and present Evidence-to-Recommendation tables for making policy recommendations.

1. Introduction

China launched the National Immunization Program (NIP)—also called the Expanded Program on Immunization (EPI)—in 1978. The program initially protected children against six vaccine-preventable diseases (VPDs) with four vaccines: bacille Calmette–Guerin vaccine (BCG), oral poliovirus vaccine (tOPV), diphtheria–tetanus–pertussis vaccine (DTP), and measles vaccine (MV). In 2002, protection was extended against an additional VPD by including the hepatitis B vaccine (HepB), and in 2007, against five more VPDs by including hepatitis A (HepA), meningococcal polysaccharide (MPV), Japanese encephalitis (JEV), and rubella and mumps vaccines [1]. The EPI is guided by the National Health Commission (NHC), which has a responsibility to make evidence-based decisions regarding further expansion of the EPI, the introduction of non-EPI vaccines into the EPI system, and the replacement of current EPI vaccines with new ones.
In China, the government has a duty to provide safe and effective EPI vaccines at no cost to families. Parents and legal guardians have a corresponding duty to vaccinate their children with EPI or equivalent non-EPI vaccines. Non-EPI vaccines are paid for by families, and families are under no obligation to use non-EPI vaccines [2]. Coverage levels of EPI vaccines are consistently and uniformly high [3], but coverage of non-EPI vaccines is lower and variable, resulting in less effective disease prevention. For example, one dose of mumps vaccine was included in the EPI, but some provinces included two doses, resulting in superior protection [4]; varicella vaccine is not included in the EPI, but some provinces provide one or two doses, resulting in lower incidence rates [5]; and during the 6 years prior to the 2016 introduction of inactivated poliovirus vaccine into the EPI, 157 children were paralyzed by tOPV-caused vaccine-associated paralytic polio [6].
Over the years, vaccine developers have made an increasing number of infectious diseases preventable through vaccination, with more than 30 non-EPI vaccines currently market authorized in China. However, new vaccine introduction has lagged the availability of new vaccines, missing important opportunities for preventing serious VPDs. There are at least eight vaccines that could be considered for introduction into the current set of EPI vaccines: Hemophilus influenzae type b (Hib), inactivated influenza vaccine (IIV), varicella vaccine (VarV), meningococcal polysaccharide conjugate (MPCV), enterovirus type 71 (EV71), pneumococcal conjugate vaccine (PCV), human papillomavirus vaccine (HPV), and rotavirus vaccine (Rota). Decisions for vaccine introduction must be science-based. Using evidence for immunization decision-making procedures and national policy development is a global health priority. National Immunization Technical Advisory Groups (NITAGs) are independent technical advisory bodies for this purpose [7]. The National Immunization Advisory Committee (NIAC), China’s NITAG, was established in 2017 to adjust the vaccines included in the EPI [8]. To this end, appropriate decision-making procedures must be used to make wise decisions on vaccines to be included in the EPI.
The World Health Organization (WHO) developed guidance for making decisions and plans to introduce a vaccine into a national immunization program [9]. However, there is no tool for prioritizing and sequencing vaccines for introduction when there are several vaccines eligible for consideration. Expert consultation has been used to make strategic work plans for NITAGs. The Delphi method is a systematic and qualitative way to elicit opinions from a group of experts through several rounds of questions. It is applicable to public health problems that cannot be solved by commonly used analytic methods [10], and it is widely used in the construction of evaluation indicator systems [11,12,13].
As a deliberative committee, NIAC cannot consider the inclusion of all potentially program-eligible vaccines simultaneously—vaccines will be sequentially considered and introduced if favorably recommended. To develop a rational medium-term policy work plan for NIAC, we used a modified Delphi technique (MDT) to systematically obtain and synthesize the opinions of experts to prioritize and sequence potentially program-eligible vaccines for consideration of EPI introduction. We report results from the MDT study and discuss the next steps for the evidence-based introduction of new vaccines into China’s EPI system.

2. Methods

From January through May 2021, we used a modified Delphi technique to refine a set of quantitative indicators for prioritizing vaccines to be considered for inclusion in China’s EPI system. Indicators are specific lines of evidence useful for making vaccine policies. Examples include the burden of disease preventable by the vaccine, the economic value of the vaccine, the safety of the vaccine, and the willingness of parents to vaccinate their children with the vaccine. Through a literature review, discussion with EPI experts, and reference to the WHO principles and considerations for adding a vaccine to a national immunization program [9], we assembled a list of 36 candidate indicators for considering the inclusion of a vaccine into China’s EPI system (Supplementary Table S1). From January through April 2021, we conducted two rounds of surveys with a panel of experts to refine the set of indicators. Subsequently, thirty experts were asked to score EPI-eligible vaccines using the refined indicator set.

2.1. Selection of Delphi Experts

We used a purposive sampling technique [10] to select knowledgeable individuals to serve on a panel of experts. Selected experts were researchers from disease prevention and control institutions with the academic rank of associate senior or above, with at least a bachelor or higher degree, who had more than 10 years of experience in vaccination practices and immunization research, and who were willing to participate continuously throughout the consultation. Ultimately, 39 experts in public health, immunology, epidemiology and health statistics, health economics, clinical medicine, and infectious diseases were recruited to participate in the Delphi process.

2.2. Delphi Survey to Establish Indicator System

The purpose of the Delphi survey was to develop and refine an indicator system for prioritizing and sequencing vaccines for introduction into the EPI system.

2.2.1. Questionnaire Development

We categorized candidate indicators into primary, secondary, and tertiary indicators, with lower-level indicators being more fine-grained than upper-level ones. The five top-level (primary) indicators were characteristics of the disease prevented by the vaccine, characteristics of the vaccine, ability of the vaccination to be implemented in the EPI system, international experience with the vaccine, and potential societal impact of the vaccine (Supplementary Table S1).
The Delphi questionnaire consisted of four parts: (1) instructions for the Delphi survey to introduce the study domains and describe how to complete the survey; (2) the questionnaire, with candidate indicators for evaluating the inclusion of a vaccine into China’s EPI system; (3) questions about experts’ authority in the EPI, including familiarity with indicators and judgment criteria for the indicator; and (4) additional question about the experts, such as age, professional title, position, education, major, and duration of work in their professional field.

2.2.2. Expert Consultations

We conducted both rounds of expert consultation using either electronic questionnaires or face-to-face interviews. In the first round, experts scored the indicators on their assessments of importance, familiarity, and judgment of appropriateness. We utilized a five-point Likert scale [12,14] for the importance of each indicator and calculated means, covariances, and full-mark ratios (proportion of responses giving “full marks”, i.e., maximum scores in the Likert scale) for each candidate indicator. We used a 5-point score (1.0, 0.8, 0.6, 0.4, 0.2), ranging from extremely familiar to unfamiliar, to indicate experts’ familiarity. For experts’ judgment basis and degree of influence, we used three-point scores that weighted practical experience (0.5, 0.4, or 0.3) higher than theoretical knowledge (0.3, 0.2, or 0.1), which was weighted higher than referring to domestic and international literature (0.1, 0.1, or 0.1) and “instinctual feeling” (0.1, 0.1, or 0.1) [15,16].
In round one, an indicator was retained if it met at least one of three threshold criteria, which were based on means, covariances, and full-mark ratios of the importance scores for candidate indicators. Round one questionnaires were analyzed to achieve a consensus, and new candidate indicators suggested by the experts during round one were included in round two. The round one consensus was based solely on importance.
In round two, experts assigned weights to the candidate indicators according to their familiarity with the prioritization study and their sources of knowledge about the indicators (i.e., direct experience with the VPD, theoretical knowledge, opinions of others from the literature, or their own feeling or opinion).

2.3. Prioritization of Candidate Vaccines

With the indicator system established and refined, we conducted a third expert consultation to determine the prioritization of candidate vaccines for NIAC’s consideration for policy recommendation of inclusion into the EPI system.

2.3.1. Candidate Vaccines

We used 3 criteria to screen 41 non-EPI vaccines for inclusion in the third round of expert consultations: (1) whether the potential candidate vaccine is a childhood vaccine, as we are only considering childhood vaccines for inclusion into China’s EPI system; (2) whether the vaccine is produced by at least three domestic manufacturers; and (3) whether the vaccine is recommended by the WHO for inclusion into every national immunization program or for widespread use in countries with epidemiological situations similar to those in China. Five vaccines met all three criteria: VarV, MPCV-AC, Hib, IIV, and EV71 vaccines.

2.3.2. Expert Consultation

We selected at random 30 of the 39 experts who participated in rounds one and two to participate in round three. The experts assigned scores from one to ten to the tertiary indicators; scores were multiplied by weighting coefficients and summed across the indicators to yield a single numeric value for each candidate vaccine. Relative values of the scores were used to develop a sequence for NIAC deliberation on policy recommendations.

2.4. Data Analysis

The intent of the analyses was to use experts’ knowledge and self-confidence in their knowledge and refine the three-level set of indicators so that the indicators can be used in a quantitative survey of experts that will yield a prioritization and sequencing of vaccines for consideration of their inclusion in the EPI.
Data entry and statistical analyses were performed using Excel 2016 and IBM SPSS 22.0. We calculated frequencies, rates, means, full mark ratios, standard deviations, and coefficients of variation for the importance scores of indicators at each level.
The reliability and representativeness of the expert consultations were measured as follows: The positivity coefficient (response rate), calculated as the ratio of the number of questionnaires delivered to the number of those returned, was used to appraise experts’ enthusiasm. The authority coefficient (Cr), calculated as Cr = (Cs + Ca)/2, where Ca and Cs are arithmetical means of experts’ familiarity scores and their sources of knowledge scores, was used to assess experts’ perceived degree of authority [17]. Expert consultation results were considered reliable when Cr ≥ 0.70 [18]. Consensus across expert panel participants was evaluated using Kendall’s coefficient of concordance (W), which evaluates the degree of consistency among the experts and consistency of scoring results during a Delphi process [19]. Kendall’s W ranges from 0 (no agreement) to 1 (complete agreement) [19,20,21]; non-parametric testing (chi-square) was used to determine whether the resulting W values were due to chance. Arithmetic means, coefficients of variation, and full-mark ratios (the proportion of experts giving the maximum score of an item) of the importance scores reflect the degree of concentration of expert opinions on each indicator. An indicator was considered highly acceptable when the standard deviation of the indicator’s importance score was <1, and the coefficient of variation was <0.2 [22]. For all analyses, p < 0.05 was considered statistically significant. The hybrid weighting coefficients of each indicator were calculated using the product method, in which indicator coefficients at the three levels were serially multiplied to yield a hybrid weighting.

2.5. Ethical Considerations

The study was approved by the Ethics Committee of China CDC (ERB number 2020). Experts provided written informed consent for participation in the study. Data were anonymized prior to analysis.

3. Results

3.1. Demographic Characteristics of the Experts

Table 1 shows the characteristics of the 39 round one and two senior experts. The experts included people in international organizations; national-, provincial-, and municipal-level CDCs; health administrative departments; and scientific research institutions. In total, 33 (84.61%) experts had master’s degrees or above, and 23 (58.98%) had more than 20 years of working experience in their field. The 30 (76.92%) experts who were selected at random from the round one/two expert panel to participate in the round three Delphi survey were similar to the original 39 experts.

3.2. Reliability and Representation of Expert Consultation

Table 2 shows the positivity coefficients, authority coefficients, importance scores of indicators, coefficients of variation, and concordance coefficients (W) of consultation for rounds one, two, and three. These measures assess the reliability and representation of expert consultation. All authority coefficients from rounds one through three were over 0.70, reflecting high accuracy and reliability of consultation. In round one, standard deviations of the importance scores were >1 for 35% of the indicators, and coefficients of variation were >0.2 for 50% of the indicators. In round two, standard deviations of all indicators became <1, and coefficients of variation for only four indicators were >0.2. Thus, the indicators became highly acceptable. The Kendall coordination coefficients (W) evaluated for all three levels of indicators in the first two consultation rounds and for the five candidate vaccines in the third consultation round showed that the experts’ scoring levels were highly similar (all p < 0.001).

3.3. Indicator System Construction

Table 3 shows the boundary values for the importance of the candidate indicators. After the first Delphi survey round, four candidate indicators were removed because they were not within the boundary values—one primary-level indicator, two secondary-level indicators, and one tertiary-level indicator. Through expert consultation and face-to-face interviews, six more tertiary-level indicators were removed, three pairs of tertiary indicators were merged, and three secondary-level and two tertiary-level indicators were added. After the second Delphi survey, two indicators were removed that were not within the boundary values—one primary level and one secondary level indicator. After two rounds of expert consultation, the final indicator system consisted of 3 primary-, 13 secondary-, and 26 tertiary-level indicators with weighting coefficients (Table 4).

3.4. Prioritization of Vaccines

The thirty questionnaires received in round three all met validity criteria. The order and score of the five candidate vaccines were VarV (6.91), MPCV-AC (6.83), Hib (6.74), IIV (6.56), EV71 (6.17), in which the possible range of scores was from one to nine. The resulting scores were in a narrow range of values that indicated all five vaccines were considered favorable by the thirty experts. Table 4 shows the final set of indicators at the primary, secondary, and tertiary levels with the means of the experts’ indicator values shown for each of the five candidate vaccines.

4. Discussion

Our study used a modified Delphi technique and developed a survey-based system for achieving a common expert perspective that can be used to prioritize vaccines for introduction into China’s EPI system. Factors most influential to the experts were similar to the WHO guidelines for new vaccine introduction [9]: features of the disease (epidemiological characteristics, burden of disease, and importance for prevention and control); features of the vaccine (vaccine characteristics, performance, availability, cost-effectiveness, and international and provincial experience); and features of the EPI program itself (financing, cold chain capacity, and public confidence). Concordance among the experts was high and increased with successive rounds of consultation. Using the resulting indicator system, a panel of thirty experts prioritized varicella, meningococcal conjugate AC, Hib, influenza, and EV71 vaccines for rank-order consideration of policy recommendations for inclusion into China’s EPI system.

4.1. Study in Context

Expert consultations have been used to make strategic work plans for NITAGs. For example, in 2005, US CDC’s Advisory Committee on Immunization Practices (ACIP) held an independent consultation for expanding influenza vaccine recommendations to additional target populations that would eventually lead to universal influenza vaccine recommendations [23]. The consultation helped focus several years of effort by the ACIP influenza working group.
There are technical aids for decision-making procedures regarding immunization programs. One of the best known is PROVAC, from the Pan American Health Organization (PAHO). PROVAC was developed in 2004 and incorporates a health and economic model for determining whether to include a vaccine into an EPI system [24]. A major innovation of PROVAC is the use of economic reasoning for decision making [25]. In contrast, the modified Delphi technique we used is a more general means to achieve a quantitative decision. Candidate factors for decision making were prespecified in our MDT based on the performed literature review and consultation with experts, but their weightings were dynamic and allowed for the reflection of experts’ self-assessments of knowledge, strengths, and limitations.
The US Institute of Medicine (now the National Academy of Medicine) developed a quantitative, multi-attribute ranking tool (SMART) in 2012 to support decisions on vaccine development. The tool was an adjunct for expert decision making to quantify the potential values of a vaccine yet to be developed [26]. Similar to our use of MDT to prioritize the ordering of introducing vaccines into the EPI, SMART prioritized vaccine development. While SMART is a tool for upstream decision making, our MDT is for downstream decision making—only vaccines that are “ready” to be included in China’s EPI were considered in our study. Vaccines we considered had to be licensed, be in use as a non-program vaccine, and have at least three domestic vaccine manufacturers—a reduced scope that streamlines the problem being solved by MDT. For example, the domestic production capacity and security of supply of pneumococcal conjugate, HPV, and rotavirus vaccines are currently insufficient for consideration of their inclusion into China’s EPI system. Therefore, these vaccines were not included in our MDT study.
Robust frameworks exist for determining whether or not to include a vaccine in a national EPI system. The most mature and frequently emulated framework is the WHO Evidence to Recommendation (E2R) Framework [27]. Individual vaccine decision making is a critically important problem; the use of the E2R framework requires a substantial amount of resource-intensive scientific work. Our MDT study was not intended to replace the E2R framework. Rather, it was intended to focus the scientific support work for NIAC by concentrating on vaccines that are most “ready” and most necessary for China’s EPI system. Completing E2R tables for these five vaccines will be the next work of NIAC’s secretariat. The usefulness of our MDT study is that the upcoming E2R work can be placed into a multi-year work plan.

4.2. Strengths and Limitations

A strength of our study is the quantification of expert opinions using a known methodology, and the use of quantitative indicators on evidence that will be needed by NIAC for consideration of vaccine inclusion. Measures of concordance and consistency were used to understand the degree of consensus among the experts, adding confidence to the validity of the results. Weightings of the indicators were adjusted based on experts’ self-assessments of their own knowledge, strengths, and weaknesses, in effect, adjusting weightings to the expert’s self-confidence in their knowledge, and thus reducing guesswork. A limitation is that only one panel of experts was used for this consultation. A different panel with differing areas of expertise could yield different prioritizations of the vaccines. However, the MDT does not influence NIAC policy work—only the sequencing of vaccines for consideration of policy recommendations.

4.3. Program Implications and Next Steps

Although vaccines will be recommended by NIAC for inclusion in the EPI based on their own evidence and merit, prioritization from our study provides a set of five vaccines that are “ready” to have their evidence summarized in an E2R table and considered by NIAC for recommendation into China’s EPI. Being “ready” implies that there is production capacity for national introduction and that experts consider these vaccines to be beneficial to children and meritorious for the EPI. Our study results will inform the overall work plan for NIAC technical working groups, focusing the work to expedite policy recommendations on the inclusion of vaccines into the EPI.
Our study identified scientific areas in which the evidence base is insufficient. Cost-effectiveness data for including vaccines into the EPI are insufficient for some of the prioritized vaccines, and improvement in disease burden surveillance for bacteria disease (Hib) is needed. Partnership with academic institutions and the development of “in-house” expertise will be necessary to ensure NIAC has the evidence needed for policy making. For vaccines that are included in the EPI system, additional research is warranted on vaccine product selection strategies that can help ensure a healthy market environment for the program and manufacturers.
In conclusion, using a modified Delphi technique, 39 experts refined a set of indicators that enabled the National Immunization Program to obtain an expert prioritization for sequencing vaccines into the EPI system in China. In rank order of priority, from the experts’ points of view, varicella, meningococcal conjugate AC, Hib, influenza, and EV-71 vaccines should be considered by NIAC working groups to make policy recommendations for inclusion in the EPI for routine vaccination of children. The modified Delphi technique may also be applicable for the best use of multiple COVID-19 vaccines in the National Immunization Program.

4.4. Highlights

New vaccine introduction into China’s EPI system has lagged the availability of new vaccines, resulting in missed opportunities for preventing serious VPDs. A modified Delphi technique was used to redefine a set of quantitative indicators for prioritizing vaccines to be considered for inclusion in China’s EPI system. Using the resulting Delphi indicators, experts rank-ordered varicella, meningococcal conjugate AC, Hib, influenza, and EV71 vaccines for consideration of introduction into the EPI. The results will inform the overall work plan for NIAC technical working groups, focusing the work to expedite policy recommendations on the inclusion of prioritized vaccines into the EPI.

Supplementary Materials

The following supporting information can be downloaded at:, Table S1: Candidate indicators for considering inclusion into the EPI before the first round of expert consultation.

Author Contributions

Data curation, N.W.; formal analysis, N.W.; investigation, N.W., Y.W. and Y.S.; methodology, C.M., J.L., C.Z., Z.A., L.R. and Z.Y.; project administration, C.M., Z.A. and Z.Y.; resources, C.M., Y.W., Y.S., C.Z., Z.A. and Z.Y.; visualization, X.Z. and L.R.; writing—original draft preparation, C.M., J.L. and N.W.; writing—review and editing, C.M., J.L., X.Z., Z.A., L.R. and Z.Y. All authors have read and agreed to the published version of the manuscript.


Chinese Center for Disease Control and Prevention, The Emergency Response Mechanism Operation Program (131031001000200001) and the Bill & Melinda Gates Foundation (Grant Number: OPP 1193638).

Institutional Review Board Statement

The study was approved by the Ethics Committee of China CDC (ERB number 2020).

Informed Consent Statement

Informed consent was obtained from all experts involved in the study.

Data Availability Statement

Not applicable.


We thank all experts who agreed to take part in this study.

Conflicts of Interest

The authors declare no conflict of interest.


  1. Wang, H.; An, Z.J.; Yin, Z.D. Achievements in prevention and control of seven infectious diseases targeted by the National Immunization Program in China across 70 years. Chin. J. Vaccines Immun. 2019, 25, 359–367. [Google Scholar]
  2. NPCC. Vaccine Administration Law of the People’s Republic of China. 2019. Available online: http://enmojgovcn/2021-06/26/c_636456htm (accessed on 21 April 2022).
  3. Cui, J.; Cao, L.; Zheng, J.; Cao, L.; Duan, M.; Xiao, Q. Reported coverage of vaccines in the National Immunization Program of China, 2015. Chin. J. Vaccines Immun. 2017, 23, 601–607. [Google Scholar]
  4. Hong, Y.; Chao, M.; Yaxing, D.; Zhi, L.; Xiaodong, L.; Fuzhen, W.; Yuanqiu, L.; Ning, W.; Lixin, H. Comparison of mumps incidence in areas of China with different immunization strategies, 2012–2019. Chin. J. Vaccines Immun. 2021, 27, 242–245. [Google Scholar]
  5. Pumei, D.; Miao, W.; Yanmin, L. Epidemiological characteristics of varicella in China, 2016–2019. Chin. J. Vaccines Immun. 2020, 26, 403–406. [Google Scholar]
  6. Wu, W.; Wang, H.; Li, K.; Nuorti, J.P.; Liu, D.; Xu, D.; Ye, J.; Zheng, J.; Fan, C.; Wen, N.; et al. Recipient vaccine-associated paralytic poliomyelitis in China, 2010–2015. Vaccine 2018, 36, 1209–1213. [Google Scholar] [CrossRef]
  7. Adjagba, A.; Senouci, K.; Biellik, R.; Batmunkh, N.; Faye, P.C.; Durupt, A.; Gessner, B.D.; da Silva, A. Supporting countries in establishing and strengthening NITAGs: Lessons learned from 5 years of the SIVAC initiative. Vaccine 2015, 33, 588–595. [Google Scholar] [CrossRef]
  8. Ma, C.; Rodewald, L.; An, Z.; Yin, Z.; Feng, Z. The National Immunization Advisory Committee in China: Roles of National Experts in Making Evidence-Based Recommendations for Immunization. China CDC Wkly. 2019, 1, 28–30. [Google Scholar] [CrossRef]
  9. WHO. Principles and Considerations for Adding a Vaccine to a National Immunization Programme—Firm Decision to Implementation and Monitoring; WHO: Geneva, Switzerland, 2014. [Google Scholar]
  10. Nasa, P.; Jain, R.; Juneja, D. Delphi methodology in healthcare research: How to decide its appropriateness. World J. Methodol. 2021, 11, 116–129. [Google Scholar] [CrossRef]
  11. Lloyd, A.R.; Hickie, I.; Boughton, C.R.; Spencer, O.; Wakefield, D. Prevalence of chronic fatigue syndrome in an Australian population. Med. J. Aust. 1990, 153, 522–528. [Google Scholar] [CrossRef]
  12. Gopal, C.P.; Ranga, A.; Joseph, K.L.; Tangiisuran, B. Development and validation of algorithms for heart failure patient care: A Delphi study. Singap. Med. J. 2015, 56, 217–223. [Google Scholar] [CrossRef] [PubMed]
  13. Wang, Y.; Liu, J.; Zhang, J.; Xie, X.; Jiang, X. Construction of nursing quality evaluation system in pediatrics outpatient department. J. Nurses Train. 2019, 34, 2036–2040. [Google Scholar]
  14. Liu, M.; He, W.; Li, K.; Deng, Y. Study on the combination of delphi method and analytic hierarchy process to construct the evaluation index system of discipline construction in an affiliated hospital. China Med. Pharm. 2019, 9, 18–22. [Google Scholar]
  15. He, Y.; Yang, X. To construct an evaluation system of mental health service accessibility based on improved Delphi method. Chongqing Univ. 2018, 47, 993–995. [Google Scholar]
  16. Chunzhi, W.; Qin, S. Research on data statistical processing method and its application in Delphi method. Coll. J. Inn. Mong. Univ. Financ. Econ. 2011, 9, 92–96. [Google Scholar]
  17. Qiu, Y.; Shen, J.; Lu, H. The Development of Criteria for the Selection of Chinese Peer Educators in HIV Management: A Delphi Study. Scientifica 2016, 2016, 9521313. [Google Scholar] [CrossRef]
  18. Li, Y.; Ehiri, J.; Hu, D.; Zhang, Y.; Wang, Q.; Zhang, S.; Cao, J. Framework of behavioral indicators for outcome evaluation of TB health promotion: A Delphi study of TB suspects and Tb patients. BMC Infect. Dis. 2014, 14, 268. [Google Scholar] [CrossRef] [PubMed]
  19. Rankin, G.; Rushton, A.; Olver, P.; Moorec, A. Chartered Society of Physiotherapy’s identification of national research priorities for physiotherapy using a modified Delphi technique. Physiotherapy 2012, 98, 260–272. [Google Scholar] [CrossRef] [PubMed]
  20. Guang, Z.; Hui, L. Modern Epidemiological Method and Application; Peking Union Medical College Press: Beijing, China, 1994; pp. 254–259. [Google Scholar]
  21. Okoli, C.; Pawlowski, S.D. The Delphi method as a research tool: An example, design considerations and applications. Inf. Manag. 2004, 42, 15–29. [Google Scholar] [CrossRef]
  22. Dai, F.; Wei, K.; Chen, Y.; Ju, M. Construction of an index system for qualitative evaluation of undergraduate nursing students innovative ability: A Delphi study. J. Clin. Nurs. 2019, 28, 4379–4388. [Google Scholar] [CrossRef] [PubMed]
  23. Schwartz, B.; Hinman, A.; Abramson, J.; Strikas, R.A.; Allred, N.; Uyeki, T.; Orenstein, W. Universal influenza vaccination in the United States: Are we ready? Report of a meeting. J. Infect. Dis. 2006, 194, S147–S154. [Google Scholar] [CrossRef]
  24. Jauregui, B.; Garcia, A.G.; Bess Janusz, C.; Blau, J.; Munier, A.; Atherly, D.; Mvundura, M.; Hajjeh, R.; Lopman, B.; Clark, A.D.; et al. Evidence-based decision-making for vaccine introductions: Overview of the ProVac International Working Group’s experience. Vaccine 2015, 33, A28–A33. [Google Scholar] [CrossRef] [PubMed]
  25. Jauregui, B.; Janusz, C.B.; Clark, A.D.; Sinha, A.; Garcia, A.G.F.; Resch, S.; Toscano, C.M.; Sanderson, C.; Andrus, J.K. ProVac Global Initiative: A vision shaped by ten years of supporting evidence-based policy decisions. Vaccine 2015, 33, A21–A27. [Google Scholar] [CrossRef] [PubMed]
  26. Knobler, S.; Bok, K.; Gellin, B. Informing vaccine decision-making: A strategic multi-attribute ranking tool for vaccines-SMART Vaccines 2.0. Vaccine 2017, 35, A43–A45. [Google Scholar] [CrossRef]
  27. WHO. Guidelines for the Development of Evidence-Based Vaccination-Related Recommendations. Available online: (accessed on 15 May 2022).
Table 1. Demographic characteristics of the experts.
Table 1. Demographic characteristics of the experts.
CharacteristicNumber (%)
Rounds 1 and 2Round 3
Male20 (51.28)15 (50.00)
Female19 (48.72)15 (50.00)
Highest educational degree
Doctor18 (46.15)14 (46.67)
Master15 (38.47)11 (36.67)
Undergraduate6 (15.38)5 (16.67)
Work experience (years)
10–2016 (41.02)12 (40.00)
21–3014 (35.90)9 (30.00)
31–409 (23.08)9 (30.00)
Title a
Senior title34 (87.18)26 (86.67)
Deputy senior title5 (12.82)4 (13.33)
Research field
Epidemiology and health statistics11 (28.21)1 (3.33)
Public health10 (25.64)11 (36.67)
Immunization program6 (15.38)10 (33.33)
Health economics5 (12.82)3 (10.00)
Clinical medicine4 (10.26)3 (10.00)
Infectious diseases3 (7.69)2 (6.67)
a Senior title included chief physician, researcher, and professor. Deputy senior titles include deputy chief physician, deputy researcher, and associate professor.
Table 2. Reliability and representation of expert consultation.
Table 2. Reliability and representation of expert consultation.
FactorsRound 1Round 2Round 3
Positivity coefficients100%--97.44%--100%--
Authority coefficients0.84--0.86--0.85--
Importance scores of indicators3.44–4.95--3.49–4.83-----
Standard deviation0.22–1.25--0.33–0.98-----
Coefficients of variation0.05–0.36--0.07–0.28-----
Concordance coefficients (W)
Primary indicators0.48675.852<0.0010.40530.758<0.001---
Secondary indicators0.356138.659<0.0010.340155.242<0.001---
Tertiary indicators0.275374.802<0.0010.236224.091<0.001---
Table 3. Boundary values for the importance of all-level indicators.
Table 3. Boundary values for the importance of all-level indicators.
FactorRound 1Round 2
Threshold value 3.953.823.764.173.734.01
Coefficient of variation0.05–0.250.05–0.290.07–0.360.09–0.170.07–0.280.07–0.24
Threshold value0.
Full mark ratio (%)28.21–94.8720.51–92.3112.82–84.6210.53–52.635.26–68.422.63–78.95
Threshold value30.5026.7225.5413.907.0113.70
Standard deviation0.22–0.960.22–1.030.34–1.250.41–0.690.33–0.980.36–0.94
Table 4. Indicator system and Delphi results for vaccine inclusion into EPI.
Table 4. Indicator system and Delphi results for vaccine inclusion into EPI.
Level of IndicatorInterpretation of the Tertiary IndicatorCombined WeightCandidate Vaccine
Diseases prevented by the vaccine-0.420
Epidemiological characteristics-0.126
Endemic areaLarger endemic area → higher possibility of including the vaccine.0.0287.175.978.388.376.73
Morbidity rateHigher morbidity → higher possibility of considering the vaccine.0.0296.005.207.637.706.30
Population mortality rate Higher mortality → higher possibility of considering the vaccine.0.0376.007.335.074.435.47
Case disability rate Higher disability rate → higher possibility of considering the vaccine.0.0325.737.384.374.204.90
Economic burden 0.120
Direct economic burdenHigher direct economic burden → higher possibility of considering the vaccine.0.0696.436.706.306.075.87
Indirect economic burdenHigher indirect economic burden → higher possibility of considering the vaccine.0.0516.476.536.076.105.70
Public health priority 0.115
In national public health listWhether control and prevention of the disease are included in China’s national public health priority list. If yes, the disease has a higher possibility of being considered.0.0626.636.687.227.436.30
Public health emergency eventWhether the disease can cause public health emergency events. If yes, the disease has a higher possibility of being considered.0.0535.076.277.577.576.73
Non-vaccine interventions (NVI) 0.059
Cost of NVIHigher costs of NVI (for example, hand washing, face mask, medicines) → higher possibility of being considered.0.0205.535.775.905.975.67
Effectiveness of NVILess effective NVI → higher possibility of being considered.0.0205.805.905.736.075.67
SustainabilityLess sustainable of implementing NVI → higher possibility of being considered.0.0195.575.605.936.205.70
Candidate vaccine 0.371
Vaccine performance and characteristics 0.118
Efficacy and Effectiveness More effective → higher possibility of considering the vaccine.0.0447.807.906.338.007.07
Persistence Longer vaccine protection persistence → higher possibility of considering the vaccine.0.0317.437.404.607.476.40
Safety Safer → higher possibility of considering the vaccine.0.0448.508.
Cost-effectiveness 0.083
Vaccination cost Lower cost → higher possibility of considering the vaccine.0.0376.476.436.906.406.17
Cost-effectiveness More cost-effective → higher possibility of considering the vaccine.0.0467.406.926.677.476.53
Availability of vaccine supply 0.081
Maximum supply Vaccine supply meets NIP need → higher possibility of being considered.0.0407.777.407.107.537.10
SustainabilitySupply sustainable → higher possibility of being considered.0.0407.637.477.207.507.33
International experience 0.042
WHO recommends Recommended by WHO → higher possibility of being considered.0.0248.937.777.277.574.40
Most countries include the vaccine in NIP Other countries have introduced the vaccine → higher possibility of being considered.0.0188.677.136.806.903.73
Domestic experience 0.046
Provincial inclusion in local immunization program Introduced into local immunization program by some provinces → higher possibility of being considered.0.0465.0766.235.776.834.33
Vaccination implementation 0.209
Financial issues 0.081
Financial affordabilityOperational costs are affordable → higher possibility of being considered.0.0816.476.576.106.535.97
Acceptability 0.047
Willingness for vaccinationPublic willing to receive the vaccination → higher possibility of being considered.0.0477.237.336.807.706.63
Ethical consideration 0.038
Benefit–risk ratioBenefit far exceeds risk → higher possibility of being considered.0.0387.707.387.177.536.43
Capability of implementation 0.043
Cold chain administration Cold chain affordable → higher possibility of being considered.0.0205.906.
Immunization service system Immunization services (e.g., human resources, information system, and surveillance) are affordable → higher possibility of being considered.0.0236.806.706.376.776.67
Total scores for each vaccine6.746.835.566.916.17
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Ma, C.; Li, J.; Wang, N.; Wang, Y.; Song, Y.; Zeng, X.; Zheng, C.; An, Z.; Rodewald, L.; Yin, Z. Prioritization of Vaccines for Inclusion into China’s Expanded Program on Immunization: Evidence from Experts’ Knowledge and Opinions. Vaccines 2022, 10, 1010.

AMA Style

Ma C, Li J, Wang N, Wang Y, Song Y, Zeng X, Zheng C, An Z, Rodewald L, Yin Z. Prioritization of Vaccines for Inclusion into China’s Expanded Program on Immunization: Evidence from Experts’ Knowledge and Opinions. Vaccines. 2022; 10(7):1010.

Chicago/Turabian Style

Ma, Chao, Junhong Li, Nan Wang, Yamin Wang, Yudan Song, Xiang Zeng, Canjun Zheng, Zhijie An, Lance Rodewald, and Zundong Yin. 2022. "Prioritization of Vaccines for Inclusion into China’s Expanded Program on Immunization: Evidence from Experts’ Knowledge and Opinions" Vaccines 10, no. 7: 1010.

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop