Monitoring the Performance of National Immunization Programs: Innovative Methodology and Tool for Countries’ Self-Assessment

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This is a very extended document on the development and application of the MD-PAI instrument in the countries of the PAHO. The process on how to develop the instrument has been well detailed and reported (huge work). It is a very extended program to get there and the impression is that the whole work has been well conceived by experts in doing this. In that respect I don’t think that my review can utter any critic on what has to be assessed and how or what is missing. I like to say that this is an excellent example on how to do the evaluation. I have a few points to needs more clarification in the text about the reporting structure of the information:

1. It is a little confusing that the project starts with mentioning 6 strategic lines (line 56), move to 5 steps to build the tool (line 80), came to 13 components (line 99-180, Table 2), ends with 3 phases (Figure 2), and 17 documents/annexes (Table 3). It would be helpful to bring all those numbers aligned into one scheme, if possible, so that people better know what this all means. Example: there are the elements of general policy of EPI (strategic lines) in which monitoring the vaccine coverage and effect is critical. There is a new tool to be developed for an adequate monitoring in 5 steps to get the tool operational. To work with the tool at country level, you need 3 phases (preparation, implementation, follow-up). The content of the tool is about 13 components. The support to the tool is 17 documents/annexes.
2. In the result and method sections there is confusion that some tables/figures are results in the method section and that there are method issues in the result section. It should therefore be better to first define in the method section 2 parts. One is about how to develop the instrument/tool and built up the content with the 5 steps. That should be seen in the Result section as well with a first section on the development of the tool in 5 steps and in each step, the quantifiable results. The second part of your method is about in introduction of the tool at country level in 3 steps. If you follow that approach -as an example-, you can mention in the method section under a, the documents reviewed, but in the Result section under the new section of product development, part ‘a’ on the review, you then mentioned the 744 questions and 13 components from the review. Everything that is quantified in the method section should be presented as a result in the result section. Things become then for an external reader more clear that your paper is focussing on the one hand on the tool development and content and on the other hand on how to introduce the tool at country level. I could have done all the changes in the text to make clear what is a result and what is a method issue, but this would have bring me to far in doing this.
3. The denominator of the countries enrolled in the PAHO organisation is not disclosed (I guess it is 35 countries) which helps to understand that 14 countries have so far going into the direction of introducing MD-PAI (40%). Is that sufficient or was a higher number expected? It also raises the question why 60% did not respond? Would be good to have that point highlighted in the discussion.
4. It is great to have such a tool developed, but the most critical point is not reported in the paper which is whether this tool, after implementation, has improved the coverage? Also here it would be important to have that discussed in the discussion section: to early to see a result, by when do we expect to see an impact?
5. This tool is certainly not cheap to implement and to make a good follow-up. It would be good for other countries that did not apply the tool yet, to have a feeling about how much it costs for working that out. You mentioned the time needed to perform the program. Somewhere it is mentioned that a consultant is needed at the level of a ministry of health.
6. There are many different ways to implement vaccination programmes in a country going at one extreme about a very centralised process getting control on the different levels up to a very decentralised and local program where the initiative is taken by the people in the field locally as long as there enough money locally to start with. I have no idea which scheme is most applicable here in the countries of the PAHO, but is the MD-PAI best applicable to the more centralised run vaccination programs or not? I asked that question because the approach here developed would/could be useful for other continents like Africa where there is a wide range of different ways to implement vaccination. The recommendation from your side would be helpful to identify where it will be most easy to apply the program.
7. Detail: you use the term ‘symmetrical’ on line 260 in the document, but I guess that you want to say ‘systematic’. I don’t see where symmetry should happen in the EPI program rather that there is a systematic approach possible but not always applicable.

Author Response

Please see the attachment. 

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

 This manuscript presents the development and implementation of the MD-PAI, a self-assessment tool designed by the Pan American Health Organization (PAHO) to support countries in evaluating the performance of their national immunization programs. The subject is timely and highly relevant, particularly in the context of post-pandemic immunization recovery and declining vaccination coverage trends. The initiative aligns with global priorities led by the World Health Organization and with regional immunization strategies. The manuscript is well organized and reflects substantial operational experience across 14 countries. However, from a scientific perspective, the methodological foundation requires significant strengthening before the work can meet the standards of an international peer-reviewed journal.

The introduction (lines 31–78) provides historical and institutional background, tracing the development of the Expanded Program on Immunization and linking it to contemporary policy frameworks. While this contextual narrative is informative, it remains largely programmatic and does not sufficiently engage with the academic literature on health system performance measurement, governance evaluation, or maturity models. The manuscript does not clearly articulate a research question or define its scientific objective. In lines 72–78, the authors state that the article describes the development of the MD-PAI tool, but it is unclear whether the intent is to present an instrument validation study, an implementation science report, or a descriptive multi-country analysis. This lack of conceptual positioning affects the coherence of the methodological framework.

The methods section (lines 79–83) outlines five development steps but does not define the study design. There is no explicit statement of methodological approach, which limits clarity. The review of existing tools described in lines 84–98 lists several evaluation frameworks, yet no systematic review process is described. There is no information about search strategy, inclusion criteria, or structured analytical comparison. The selection process for reducing 744 questions to 196 (lines 99–112) is attributed to subject-matter expert prioritization, but the manuscript does not report the number of experts involved, how they were selected, whether a formal consensus technique was used, or how agreement was measured. This weakens reproducibility and increases the risk of institutional bias.

The five-level maturity scale described in lines 116–131 is conceptually aligned with international frameworks but lacks psychometric validation. There is no evidence of internal consistency testing, construct validation, or factor analysis. The calibration phase (lines 132–147) mentions external consultation but does not describe the methodology used to assess reliability or document revisions. Similarly, the pilot testing in Guyana and Guatemala (lines 148–155) lacks methodological detail regarding participant profiles, structured feedback analysis, or documented changes following the pilot phase.

The sampling approach at subnational level (lines 214–220) explicitly states that randomness is not required. While practical constraints are understandable, this decision introduces potential selection bias and limits comparability. The consensus-based self-assessment approach described in lines 221–231 raises additional concerns regarding social desirability and political bias. Although the authors acknowledge this risk in lines 300–306, no formal mitigation strategy is described. Another significant methodological issue concerns the aggregation of ordinal scale scores. In lines 191–194, averages and medians are calculated and presented graphically. Given that the scale defined in lines 116–124 is ordinal, calculating means assumes equal intervals between levels. The authors acknowledge this limitation in lines 307–312 but do not provide a statistical justification or alternative analytical approach.

The results section (lines 239–260) demonstrates adoption across 14 countries and identifies component-level strengths and weaknesses. However, the reporting remains descriptive. No quantitative comparative tables, measures of dispersion, or inferential analyses are presented. Importantly, the manuscript does not evaluate impact. In lines 334–343, the authors acknowledge that the effect of MD-PAI on vaccination coverage remains to be evaluated, confirming that outcome assessment has not yet been conducted.

The discussion section (lines 270–351) appropriately highlights governance strengthening and alignment with health security frameworks. The acknowledgment of methodological limitations, particularly regarding self-assessment bias (lines 300–306) and ordinal scale aggregation (lines 307–312), is commendable. However, the discussion would benefit from deeper engagement with academic literature and stronger reflection on institutional bias, particularly given that all authors were involved in tool development and implementation (lines 356–364), despite the declaration of no conflicts of interest (line 376).

To enhance scientific rigor and ensure suitability for publication, the following revisions are recommended:

• Clearly define the study design and scientific objective (e.g., methodological development study, implementation science study, descriptive evaluation).

• Provide a structured description of the review process used to analyze existing evaluation tools, including search strategy and selection criteria.

• Specify the number, selection process, and expertise of subject-matter experts involved in question prioritization.

• Describe whether a formal consensus method (e.g., Delphi process) was used and how agreement was assessed.

• Provide greater methodological detail on the calibration phase, including procedures used to assess internal consistency.

• Describe the pilot testing methodology in detail, including participant profiles, data collection methods, feedback analysis, and documented revisions.

• Justify the use of mean scores for ordinal data or provide alternative aggregation methods (e.g., median-based analysis or categorical reporting).

• Discuss the limitations of non-probabilistic subnational sampling more explicitly and clarify its implications for generalizability.

• Elaborate on mitigation strategies for self-assessment bias (e.g., external validation mechanisms or triangulation).

• Provide more detailed quantitative reporting of country-level results, including summary statistics and measures of variability.

• Clarify ethical considerations related to workshop participation and data governance.

• Expand the discussion to include comparison with existing maturity models and performance assessment frameworks in the academic literature.

• Explicitly address potential structural institutional bias arising from developer-led implementation and evaluation.

Author Response

Please see the attachement

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Congrat. I'm fine with the corrections made. It is now much clearer as a presentation. There was one small error on line 303: Anex should be Annex. 

Author Response

Please, see the attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

Thank you for submitting the revised version of your manuscript entitled “Monitoring the Performance of National Immunization Programs: Innovative Methodology and Tool for Countries’ Self-Assessment.” The manuscript presents an important and timely initiative led by PAHO to strengthen national immunization program performance through the MD-PAI maturity model. The revised version improves clarity in several areas, particularly in describing the conceptual framework and implementation experience across multiple countries.

 

However, several methodological and interpretative issues remain that require clarification or strengthening before the manuscript can be considered for publication.

 

Major Comments

1. Methodological validation of the MD-PAI tool remains insufficiently demonstrated

You state that the MD-PAI was developed through a structured process including review of existing methodologies, prioritization of questions, piloting, and calibration to ensure validity, completeness, reliability, and usability (lines 12–19) . While this description is useful, the manuscript does not provide sufficient methodological detail on how these properties were empirically evaluated.

Specifically, it remains unclear:

• How construct validity was assessed,
• Whether inter-rater reliability or reproducibility was formally measured,
• Whether quantitative performance metrics were used to evaluate the tool’s sensitivity and specificity in detecting programmatic gaps.

The manuscript refers to piloting and calibration (lines 12–19) but does not provide methodological criteria, statistical validation procedures, or structured evaluation results. Providing even basic validation metrics (e.g., reproducibility testing, inter-observer agreement, or pilot evaluation outcomes) would substantially strengthen the scientific robustness of the tool.

2. Lack of clear sampling methodology and representativeness at subnational level

The manuscript acknowledges that subnational and local information is collected, but also states that these results are not formally included in the aggregated MD-PAI results (lines 349–356) . Furthermore, it suggests that probabilistic sampling could reduce selection bias but does not indicate that such sampling was implemented.

 

This raises important concerns regarding representativeness and potential selection bias. Specifically:

• The criteria used to select subnational units are not clearly defined,
• It is unclear whether participating sites are representative of national program performance,
• The potential for selection bias introduced by voluntary participation is not adequately addressed.

 

Since the tool is intended to support national performance evaluation, the representativeness of assessed units is critical for ensuring generalizability.

Clarification of site selection methodology, including whether systematic or convenience sampling was used, is essential.

3. Results are presented primarily descriptively, without quantitative performance analysis

The manuscript reports that fourteen countries implemented the MD-PAI and developed action plans (lines 17–23, 272–276) . However, the results remain largely descriptive and lack quantitative evaluation of performance outcomes.

For example:

• No numerical summary scores, distributions, or comparative analyses are presented,
• There is no quantitative assessment of improvement in program performance,
• The manuscript does not evaluate whether the tool leads to measurable improvements in immunization outcomes.

 

Without quantitative performance indicators, it is difficult to assess the operational impact and effectiveness of the tool. Even descriptive summary statistics of maturity scores across components would greatly enhance the analytical value.

4. Potential subjectivity and bias in maturity model scoring is insufficiently addressed

The MD-PAI relies on consensus-based evaluation involving national coordination teams and stakeholders (lines 338–347) . While participatory approaches are valuable, they also introduce potential subjectivity and reporting bias.

The manuscript does not sufficiently describe:

• Procedures to standardize scoring across countries,
• Mechanisms to ensure scoring consistency,
• Independent verification or external validation procedures.

 

Without standardized scoring controls, there is a risk of systematic overestimation or variability in maturity assessments. Clarification on quality control procedures is recommended.

5. Interpretation of effectiveness and impact remains overstated relative to available evidence

The manuscript concludes that the MD-PAI is an “agile, efficient, and sensitive methodology” for detecting gaps and strengthening immunization programs (lines 24–27). However, the manuscript does not present direct evidence demonstrating improved vaccination coverage, improved operational performance, or measurable programmatic outcomes resulting from MD-PAI implementation. While the tool appears promising, its effectiveness remains supported primarily by implementation experience rather than quantitative outcome evaluation.

The conclusions should therefore be more cautiously framed and clearly distinguish between:

• feasibility,
• acceptability,
• and demonstrated effectiveness.

 

Minor Comments

1. Clarify operational definitions and scoring methodology

The manuscript describes the maturity model concept and progressive levels of performance (lines 321–329), but does not clearly define:

• scoring criteria,
• thresholds for each maturity level,
• or aggregation methods across components.

Providing a concise description of scoring methodology would improve transparency.

2. Clarify data sources and data quality control procedures

You indicate that national coordination teams collect and validate data (lines 421–427) , but specific quality assurance mechanisms are not described.

Clarifying validation procedures would strengthen methodological transparency.

 

3. Clarify generalizability of findings

The manuscript presents implementation results from 14 countries (lines 272–276, 330–335) . However, variability in health systems, governance, and resources may affect generalizability.

Explicit discussion of these limitations would improve interpretative balance.

Author Response

Please see the attachment

Author Response File: Author Response.pdf