Next Article in Journal
Laser Tooth Preparation for Pit and Fissure Sealing
Previous Article in Journal
Group versus Individualised Minimum Velocity Thresholds in the Prediction of Maximal Strength in Trained Female Athletes
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:

Mixed and Multi-Methods Protocol to Evaluate Implementation Processes and Early Effects of the Pradhan Mantri Jan Arogya Yojana Scheme in Seven Indian States

Heidelberg Institute of Global Health, Medical Faculty and University Hospital, Heidelberg University, 69120 Heidelberg, Germany
German Development Institute/Deutsches Institut für Entwicklungspolitik (DIE), 53113 Bonn, Germany
Centre for Global Health and Health Partnerships, School of Population Health and Environmental Sciences, King’s College London, London SE5 9RJ, UK
IQVIA Consulting and Information Services India, New Delhi 110001, India
Nielsen India Private Limited, Gurugram 122002, India
Indo-German Social Security Programme (IGSSP), Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH, New Delhi 110029, India
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2020, 17(21), 7812;
Submission received: 28 September 2020 / Revised: 15 October 2020 / Accepted: 22 October 2020 / Published: 26 October 2020


In September 2018, India launched Pradhan Mantri Jan Arogya Yojana (PM-JAY), a nationally implemented government-funded health insurance scheme to improve access to quality inpatient care, increase financial protection, and reduce unmet need for the most vulnerable population groups. This protocol describes the methodology adopted to evaluate implementation processes and early effects of PM-JAY in seven Indian states. The study adopts a mixed and multi-methods concurrent triangulation design including three components: 1. demand-side household study, including a structured survey and qualitative elements, to quantify and understand PM-JAY reach and its effect on insurance awareness, health service utilization, and financial protection; 2. supply-side hospital-based survey encompassing both quantitative and qualitative elements to assess the effect of PM-JAY on quality of service delivery and to explore healthcare providers’ experiences with scheme implementation; and 3. process documentation to examine implementation processes in selected states transitioning from either no or prior health insurance to PM-JAY. Descriptive statistics and quasi-experimental methods will be used to analyze quantitative data, while thematic analysis will be used to analyze qualitative data. The study design presented represents the first effort to jointly evaluate implementation processes and early effects of the largest government-funded health insurance scheme ever launched in India.

1. Introduction

In line with global efforts aimed at leaving no one behind while advancing universal health coverage (UHC), recent health sector reforms in India have strived to expand social health protection for the most vulnerable segments of society while simultaneously strengthening delivery of quality healthcare services. This strategy has largely been pursued through the implementation of government-funded health insurance (GFHI) schemes implemented at either federal or state levels [1], aimed at improving access to quality health care while offering financial protection to end users. The expansion of these schemes has led to a steady rise in population insurance coverage, from approximately 5% of households in 2005 to 29% in 2015 [2].
Evidence on the effect of these Indian GFHI schemes on health service use, financial protection, quality of service provision, and health outcomes presents a diverse view. A recent systematic review on the impact of Indian GFHI schemes found positive effects on health service use but mixed effects on out-of-pocket expenditures (OOPEs) and financial risk protection [3]. Only one study demonstrated better health outcomes in enrolled versus non-enrolled households for covered conditions [4]. The few studies on gender effects of Indian GFHI schemes report the persistence of lower access, enrolment, and use among women than men [5,6,7,8,9]. There is little evidence on the effect of in-built quality control mechanisms in improving the quality of service provision [10,11,12]. Likewise, little is known on scheme implementation processes and their role in shaping impacts [13,14,15,16]. Even fewer studies explore factors relating to scheme ideation, design, and conceptualization [17]. A review of numerous impact studies on one specific Indian scheme, Rashtriya Swasthya Bima Yojana (RSBY), highlights methodological weaknesses in existing studies and calls for more rigorous evaluations [18]. This paucity of evidence is surprising considering that the scope of GFHI implementation in the Indian context can offer numerous opportunities for learning at the global level.
Our study is situated against this background and aimed to generate scientific evidence on both implementation processes and early effects of the largest GFHI scheme in India, the Pradhan Mantri Jan Arogya Yojana (PM-JAY). The ambition of our study, which combines process and impact evaluation elements to look at the first 18 months of operation, was to generate evidence to guide future implementation of the scheme while simultaneously drawing lessons to inform social health protection developments in other settings.
Hereafter, we first describe the scheme and its complexities and then present details of our study design, including both the conceptual model and the methodology guiding our evaluation work.

2. Materials and Methods

2.1. Pradhan Mantri Jan Arogya Yojana

In September 2018, the Government of India (GoI) launched PM-JAY [19]. The GoI expects PM-JAY to consolidate gains made by previous and ongoing GFHI schemes, primarily RSBY, by removing formal enrolment procedures and expanding coverage to approximately 500 million people listed as most vulnerable in the Socio-Economic Caste Census (SECC) of 2011 [20]. Insurance coverage under this new scheme is no longer subject to a fixed household size but instead covers year INR 500,000 (EUR 6150 in May 2020) per household with no member ceiling; the benefit package includes 1350 inpatient procedures provided at secondary and tertiary care levels. Beneficiaries can avail these services at any public and empaneled private hospital across all Indian states implementing PM-JAY. The scheme is governed by the National Health Authority (NHA) at the federal level and implemented by the State Health Agency (SHA) in each state, either through existing insurance companies, or publicly run autonomous trusts/nodal agencies, or a combination of these two (referred to as mixed models), with a scope for state-specific adjustments. Currently, 32 out of 36 states and union territories in India are implementing the scheme [21]. PM-JAY seeks both to expand health care access and financial protection, and to promote efficiency, transparency, and accountability by adopting output-based financing measures.
PM-JAY implementers ultimately expect the scheme to increase affordable access to quality healthcare services for the most vulnerable segments of society. They anticipate the scheme to produce significant changes on all three UHC dimensions: population coverage (through comprehensive eligibility and removal of any formal enrolment procedure for the most vulnerable population segments); service coverage (through a generous benefit package covering high-cost services offered through a wide network of PM-JAY-accredited and empaneled providers); and financial protection (through cashless provision and high insurance ceiling, reducing out-of-pocket payments).

2.2. Conceptual Framework and Study Objectives

In order to assess the effect of PM-JAY on outcomes related to health and healthcare, we rely on a conceptual framework (Figure 1), which explicitly acknowledges access to healthcare services as the product of the interaction between supply- and demand-side factors [22].
Our study design aligns with this conceptual postulation and explores both demand- and supply-side elements shaping population coverage first, and access to care and financial protection afterwards. In addition, our design recognizes the socio-political dimension of policy implementation as an element that affects intervention outcomes by including a policy-level process documentation aimed at disentangling how the overall policy context and implementation realities ultimately shape behavior, both on the demand and on the supply sides.
The study objectives were:
  • To assess and understand the effects of PM-JAY on the target population’s awareness of PM-JAY features, coverage, health service use, and financial protection, including unraveling effect heterogeneity by population sub-groups (e.g., along socio-economic and gender dimensions);
  • To assess the effects of PM-JAY on quality of service delivery and to explore healthcare providers’ experiences with the implementation of PM-JAY; and
  • To document implementation processes at the national level, as well as in selected states transitioning from either no or a prior GFHI to PM-JAY.
Each study objective corresponds to a component within the study design (Figure 2). Study components 1 and 2 look respectively at the effects of the scheme on the demand and on the supply of secondary and tertiary care services. Study component 3 documents the process of designing and implementing PM-JAY.

2.3. Study Design

This study adopts a concurrent triangulation mixed methods design [23] (Figure 3), relying on a multi-method approach [24]. The term ‘mixed methods’ is used to refer to the inclusion of both quantitative and qualitative methods of data collection and analysis. The term ‘multi-method’ refers to the fact that our design incorporates diverse methods of data collection and analysis within each strain of research, i.e., qualitative and quantitative [24]. Specifically, in a triangulation design, quantitative and qualitative methods hold equal weight and are used to explore different facets of the same phenomenon [23]. In our specific case, we selected triangulation above any other mixed-methods design because we focus on different facets of a single phenomenon, the implementation and effects of the PM-JAY. We used quantitative methods in a quasi-experiment to quantify scheme awareness and population coverage among the target community and to assess the scheme effect on outcomes of interest (health service use and financial protection); we used qualitative methods to explore causal mechanisms of change through an analysis of implementation processes. The term ‘concurrent’ refers to the fact that quantitative and qualitative data are collected in parallel, without waiting for the findings from one strain of research to be available to inform data collection for the other strain [23]. The design was set exclusively by the authors.

2.4. Study Setting and State and District Selection

The study took place in seven Indian states, namely Bihar, Chhattisgarh, Gujarat, Karnataka, Meghalaya, Tamil Nadu, and Uttar Pradesh. Both states and districts within states were purposively selected by policy stakeholders in consultation with study funders (the Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH) to reflect the geographical diversity and landscape of the earlier RSBY as well as the diversity of implementation models within PM-JAY itself: one state (Meghalaya) is implementing the scheme through insurance companies, three states (Bihar, Karnataka, and Uttar Pradesh) are implementing the scheme though state agencies, and the remaining three states (Chhattisgarh, Gujarat, and Tamil Nadu) are implementing the scheme through mixed models involving both state agencies and insurance companies. While prior to PM-JAY, Chhattisgarh and Meghalaya states had successfully used the RSBY platform to extend coverage to nearly their entire population via additional state funding, the states of Gujarat, Karnataka, and Tamil Nadu had successfully implemented their own state-funded tertiary care schemes. Bihar and Uttar Pradesh are two states in which RSBY implementation had already been discontinued for several years. Similarly, two districts were selected in each state (three districts in Bihar and Uttar Pradesh) to account for diverse performance levels within states (Table 1 and Figure 4).
Hereafter, we describe each of the study components in detail.

2.5. Study Component 1: Demand-Side Household and Individual Study

This component integrates quantitative and qualitative methods to address the first study objective. A single data collection round was planned for the period November 2019 to May 2020. Data collection was suspended starting in late March 2020 due to the SARS-CoV-2 epidemic.

2.5.1. Quantitative Component


Our quantitative sampling strategy aims to include both a representative sample of PM-JAY eligible households at the district level and a sufficiently representative sample of those eligible households having advanced a claim under PM-JAY by the time of data collection, approximately 14 to 16 months after the scheme launch. Within all the above-mentioned districts, our primary sampling unit (PSU) included villages in rural areas and medium/small urban areas with less than 7500 inhabitants (based on the observations of our pilot surveys, we set the cutoff at 7500 inhabitants to exclude larger urban areas in which administering the survey was not operationally feasible). Sampling adopted a three-step approach. First, information from the 2011 SECC was used to randomly select 50 PSUs that were representative of the entire district with regards to the following district characteristics: number of PM-JAY eligible households, number of PM-JAY eligible individuals, number of PM-JAY claims, share of urban households, share of women, share of children, share of adults, share of elderly, and share of married people. Second, using the selected PSUs and SECC information, 15 households per PSU were randomly selected to reach a total representative sample of 750 eligible households per district. Third, 20 additional PSUs were identified among those with the highest number of claims across all remaining PSUs in a district. In order to ensure a sufficient sample of households with a filed claim under PM-JAY, 20 claim households were selected in each of these 20 PSUs, to reach a total sample of 400 claim households per district. This strategy aimed to reach 1150 households per district, for a total of 18,400 households across the seven states.

Data Collection Tools and Strategies

Data were collected via a household survey containing questions on household socio-demographics, household loans and consumption expenditure, illness reporting (for acute and chronic illnesses and related inpatient hospitalizations), health service use, out-of-pocket spending on health, and PM-JAY awareness and experiences. Given the scheme’s focus on secondary and tertiary care, the survey included questions on inpatient hospitalizations (both surgical and non-surgical) over the prior 12 months. A specific module was built within the survey to assess women’s experiences with the scheme and its role in mediating household decision-making and enhancing women’s empowerment. Data were to be collected by trained interviewers relying on the computer-assisted personal interviewing (CAPI) on digital devices. The survey was translated in all concerned local languages and administered by local interviewer teams perfectly fluent in these languages and recruited directly within the study districts.

Analytical Approach

Descriptive statistics will be used to describe awareness of the scheme and its operational modalities. The scheme effects will be assessed by establishing changes on reported hospitalization rates and OOPE (defined as primary outcomes in line with the scheme focus) by applying a difference-in-difference framework (DID). Using data on the exact empanelment dates of all hospitals in the survey districts and by using the detailed information on hospitalization cases (including date and place of hospitalization) from the household survey, we will be able to compare hospitalization and OOPE of individuals at different points in time (from the date of hospitalization) that are living in districts where many hospitals were empaneled immediately after the start of PM-JAY (treatment group) with districts where the rollout of PM-JAY and the empanelment of hospitals took more time (control group). In order to control for time-invariant state characteristics, such as health infrastructure, that are likely to be correlated with both the hospital empanelment dates in the districts and our primary outcome variables, we will include state dummies into all our specifications. Furthermore, we will include individual and household-specific variables in our specifications.

2.5.2. Qualitative Component


Qualitative data collection also took place across the abovementioned districts, specifically in a sub-set of communities targeted by the quantitative survey, and relied on a combination of focus group discussions (FGDs) among PM-JAY eligible community members, individual in-depth interviews (IDIs) with individuals who had used services under PM-JAY, and key informant interviews (KIIs) among community health workers (CHWs), as those in charge of PM-JAY community dissemination and support activities. We combined FGD and IDI because while the former are useful to elicit knowledge on overall community attitudes, beliefs, and social constructions, the latter are useful to gain in-depth insight into individual views and experiences of PM-JAY [25].
Although settling on a specific sample size is beyond the scope of qualitative research [25], we expected to conduct a minimum of 2 FGD in each district, 1 in a rural and 1 in an urban setting (n = 2 × 16 districts = 32 FGD), with 8–10 participants in each FGD, specifically selecting PM-JAY eligible individuals with or without PM-JAY service use experience. Given the specific cultural context within which our study takes place as well as our wish to maintain a specific focus on gender dynamics throughout the study, FGDs will either be all female or all male. From each FGD, we will select 1–2 information-rich cases, i.e., individuals whose experience with PM-JAY service use appears to be particularly poignant in relation to the underlying research questions (for example those who have experienced high-cost treatments or have lodged grievances under PM-JAY) and follow them up with a request to engage in an additional IDI. We expect to conduct a total of 60–80 IDIs. In addition, we plan to interview at least two CHWs per district as key informants, providing information on the overall scheme rollout and helping us to identify respondents for the FGD and the IDIs.

Data Collection Tools and Strategies

FGDs, IDIs, and KIIs are to be conducted in the local state-specific language, recorded after obtaining written informed consent, transcribed verbatim, and translated into English for analysis. Data collection is to be carried out by trained qualitative research assistants, using the guides briefly described hereafter.
FGD, IDI, and KII discussion guides include themes on health conditions requiring treatment, provider availability in the community including perceptions around their quality, arranging finances for healthcare, household decision-making processes for healthcare and changes therein, awareness of PM-JAY, awareness about eligibility, knowledge about PM-JAY, experiences with utilizing services under PM-JAY (including grievances), and review/assessment of PM-JAY. The IDI guide will also explicitly focus on the illness and service utilization experience and gender-related aspects of the main themes, while the KII guide will also explore PM-JAY-related roles and responsibilities of CHWs.

Analytical Approach

Transcripts from FGD, IDI, and KII will be analyzed using the framework approach [26,27], relying on a mixed deductive and inductive approach [25]. We will develop an initial set of coding categories based on the themes of the interview guides but will allow additional codes and themes to emerge iteratively. We will apply source and analyst triangulation [25]. Data analysis will be conducted using Nvivo Pro 12 QSR International Software (QSR International, Doncaster, Australia) [28].

2.6. Study Component 2: Supply-Side Hospital Evaluation

This component integrates quantitative and qualitative methods to addresses the second study objective. Data collection was carried out during two repeated rounds (the first from February to April 2019 and the second from November 2019 to January 2020).

2.6.1. Quantitative Component


This study component included two different samples, a facility sample and a patient sample. For the facility sample, we identified a non-random sample of 6 empaneled (either in PM-JAY scheme or in a state GFHI) and 6 non-empaneled hospitals in each of 14 of the abovementioned 16 districts (the supply-side hospital evaluation was not conducted in two districts: Gaya (Bihar) and Rampur (Uttar Pradesh). In each district, empaneled and non-empaneled hospitals were balanced (but not matched) on the following characteristics: rural vs. urban setting, hospital size, public vs. private ownership, and medical specialties offered. Information on hospital characteristics were obtained from PM-JAY district nodal officers. The resulting total sample size included 168 hospitals across 14 districts and 7 states. For the patient sample, 6–8 discharged patients previously hospitalized either under PM-JAY or another state-specific insurance scheme were randomly selected from all discharged patients at each sampled hospital at the day of data collection.

Data Collection Tools and Strategies

This study component included three quantitative surveys applied across sampled hospitals: (1). a checklist on hospital infrastructure (i.e., information on facility accessibility, availability of service units and basic equipment, health management and referral structures, safety and monitoring measures, transportation means); (2). a checklist on health service and administrative processes (i.e., information on aspects related to empanelment and accreditation, extent to which empanelment criteria are met, extent to which scheme-specific processes (e.g., trainings, claims settlement, monitoring, reporting, audits, etc.) are met); and (3). exit interviews with previously hospitalized patients (i.e., information on socio-demographic aspects, health insurance coverage, hospitalization and pre-hospitalization aspects (e.g., referral, OOPE, services received, inpatient accommodation, and amenities), personal experience, and perception of hospital stay).
Trained research assistants using CAPI digitalized solutions will carry out data collection using the three tools. The two checklists on infrastructure and service/administrative processes are to be completed by the hospital director or manager with research assistants directly observing selected items and/or processes.

Analytical Approach

Variables obtained by these three quantitative surveys will be examined and combined into composite indicators measuring the following key areas: general availability and readiness of hospital, specific availability and readiness of empaneled secondary level services (i.e., medical, surgical, obstetric, emergency), implementation of scheme-relevant processes, and patient satisfaction with received services. Composite indices will be developed based on a conceptional framework matrix reflecting both the World Health Organization’s Service Availability and Readiness Assessment criteria [29] as well as PM-JAY hospital implementation guidelines [30]. The resulting composite indicators will serve as primary outcome measures in measuring both scheme effects and implementation progress.
Since our sample involves both PM-JAY empaneled and non-empaneled (i.e., control) hospitals within the same study districts, we propose a quasi-experimental approach to impact estimation based on a DID evaluation of hospitals across the two data collection rounds: empaneled hospitals are considered as “treatment” and the remaining as “comparison”. By comparing pre- and post-PM-JAY quality of care outcomes in treatment and comparison hospitals, we can capture the causal impacts of PM-JAY. We may further ensure comparability of treatment (empaneled) and comparison (not-empaneled) hospitals by employing coarsened exact matching [31] methods to control for hospital observable covariates, such as bed strength, human resources, infrastructure, supplies, and others.

2.6.2. Qualitative Component


Qualitative data sources considered for this study component included at least one individual service provider from each empaneled and non-empaneled hospital. These individuals include the main hospital administrator/manager (i.e., the hospital director in most instances) and one key clinical provider (i.e., ideally the provider in charge of general admission of PM-JAY patients, usually a senior general physician). In non-empaneled hospitals, only the main hospital administrator/manager will be interviewed, as their clinical providers are not expected to deal with PM-JAY patients. The resulting qualitative sample will therefore include approximately 250 individuals.

Data Collection Tools and Strategies

Qualitative information is collected through IDI. We will use three different interview guides depending on hospital empanelment status and respondent background. For IDI with administrators/managers at empaneled hospitals, interviews focus on experiences and perceptions of the PM-JAY implementation process in relation to managerial/administrative aspects of providing quality services and resulting suggestions for further improvements in PM-JAY implementation. For IDI with key clinical providers at empaneled hospitals, interviews focus on experiences and perceptions of PM-JAY and other GFHI implementation processes in relation to clinical aspects of providing quality services and resulting suggestions for further improvements in PM-JAY implementation. For IDI with administrators/managers at non-empaneled hospitals, interviews focus on experiences and perceptions of the implementation process in relation to managerial/administrative aspects of providing health care services, challenges or experiences with past empanelment in PM-JAY or other GFHI, and suggestions for these schemes, with an additional focus on why these hospitals did not decide to empanel under PM-JAY.
Data collection will be carried out during two repeated rounds, at the same time as the hospital surveys by trained research assistants.

Analytical Approach

The analytical approach will be the same as for the qualitative component in study component 1.

2.7. Study Component 3: Process Documentation

This study component adopts a qualitative approach to address the third research question by reconstructing and documenting processes related to the ideation, development, and implementation of PM-JAY at the national and state levels. Rooted in a constructivist perspective approach [32], we explored PM-JAY early implementation through the eyes of the people directly involved in the scheme design and rollout.

2.7.1. Sampling

Data for this component was collected in Chhattisgarh, Tamil Nadu, and Uttar Pradesh. These states were purposely selected by scheme implementers to reflect the diversity of PM-JAY implementation modes. In addition, we collected data in the capital city, Delhi, to capture the views of national-level policy makers and implementers. At the state level, we included individuals from the apex scheme implementation body (the State Health Agency or analogous entity within the state); technical support units within the SHA, such as consultant organizations or information technology teams; implementation support agencies, such as insurance companies or third-party administrators; and any external organizations providing independent technical or managerial support in scheme implementation. At the district level, we included district functionaries directly involved in the implementation of PM-JAY. In addition, at the national level, we used the snowball sampling approach to identify respondents from all key stakeholder groups involved in PM-JAY implementation and development, i.e., both closely involved actors (such as government officials and policy makers, international, bi- and multi-lateral agencies, and PM-JAY implementers) and more distantly involved groups (such as academia and civil society).

2.7.2. Data Collection Tools and Strategies

Data collection relies on KIIs conducted with one of three interview guides developed by the study team, with one guide focusing on policy makers, one on state-level functionaries, and one on district implementers. In each of the three guides, we explore the fidelity of implementation to the original design and related adaptations; changes to administrative structures and authority within governing institutions; changes in implementation processes; how stakeholders responded to implementation; and any unintended consequences arising from implementation processes. Data collection in Delhi aimed at capturing national dynamics related to ideation, planning, and implementation; data collection at state and district levels aimed at capturing state-specific implementation realities.
Policy-level interviews were conducted either by the principal investigator and/or by two additional senior researchers (all authors). Interviews at the district level were conducted by a small team of senior qualitative researchers.

2.7.3. Analytical Approach

Analysis of early implementation developments is considered an ongoing process that requires integration of the implementation science perspective [33] into the political economy context [34]. Thus, using thematic analysis, we will describe PM-JAY implementation processes, including achievements and challenges, in relation to their wider socio-political context within which the scheme implementation takes place. This approach entails that at different analytical levels, we aim at understanding how the new scheme is shaped by existing authority structures but at the same time also challenged by these authority structures.

2.8. Ethical Considerations

Ethical approval for the study was obtained from the Ethikkommission of the Medizinische Fakultät Heidelberg, Germany, and from Sigma Institutional Review Board, New Delhi, India via IRB numbers 10057/IRB/19-20 (study component 1) and 1011/IRB/D/18-19 (study components 2 and 3). Written informed consent was sought from all study participants.

2.9. Patient and Public Involvement

The public was not involved in study design or data collection, and will not be involved in analysis.

2.10. Data Sharing

No additional data are available as this is a study protocol and no data are being used. The data produced from this study will not be available for open access, but the corresponding author can be contacted in case of any queries.

3. Discussion

The study design presented in this article represents the first effort to jointly evaluate the implementation processes and early effects of the largest GHFI scheme ever launched in India, PM-JAY. Describing our study protocol serves the purpose of highlighting the efforts channeled towards generating sound scientific evidence in light of real-life evaluation needs, determined by policy considerations and implementation realities. Given PM-JAY’s national reach, it was impossible for our team to control elements related to treatment assignment and rely on a standard experimental design for evaluation. Similarly, decisions on state and district sampling were made a priori, in light of specific policy needs, constraining the team’s methodological choices. Notwithstanding such challenges, we do trust having been able to produce a sound study design, which draws its strengths primarily from the mixed- and multi-method nature of its data collection and analysis effort and from the ability to identify and integrate quasi-experimental design elements in the quantitative analytical strain. We started with the scheme implementation reality and co-created a study design that could accommodate it while at the same time adhering to best research practices.
A further strength of our study design rests in the fact that elements related to process and impact evaluation are embedded within a single study protocol, addressing both demand- and supply-side implementation realities and early effects. This integration of multiple research strains (both from a content and from a methodological perspective) within a single study protocol favors triangulation since it assigns responsibility for a very diverse set of analytical components to a single study team. While data collection activities are fixed, and further constrained by the challenges of the SARS-CoV-2 epidemic at the time of writing this article, reliance on a single team allows analysis to proceed in an emergent manner [35], with process elements being used to elucidate effect results and effect results being used to inspire further investigation into process elements.
Notwithstanding the potential to yield informative results through a very enriching analytical process, we must acknowledge the complexity of our study design and the challenges inherent in leading our combined process and impact evaluation towards a successful end. First, implementing our study design requires reliance on a very diverse team of researchers, with very different content and methodological expertise. It follows that while ultimately enriching, the process of integrating findings across analytical strains will inevitably be difficult and possibly lead to occasional conflict. Second, the breadth of our design, and of the different elements it encompasses, will inevitably add depth to our work but also complexity to our dissemination efforts. The team will have to find ways to present results in a clear and effective manner, drawing from multiple analytical strains to ensure that the broader policy and academic community embrace the complexity of scheme implementation and effects. Given the attention that efforts to expand social health protection worldwide are receiving, our team will need to have the capacity to ensure adequate dissemination well beyond Indian boundaries.
In addition, we must acknowledge the limited external validity of our study design [36]. Not only were states and districts within states purposely selected ex-ante, but India is a vast country characterized by the co-existence of very different realities. Hence, while we trust that our overall approach to sampling and data collection yields a high internal validity [36], we can by no means assert that our findings will provide an accurate picture of PM-JAY implementation processes and early effects for the entire country. In addition, we must acknowledge the inherent bias that arises from the sampling strategies adopted both to identify households for the demand-side component and hospitals for the supply-side component. Prior to the beginning of our study, we pondered multiple alternatives and convened that in the absence of a more recent complete population listing and in the absence of a complete health facility listing (or of resources to undergo one on our own), we had no alternative but to adopt the strategies described earlier in this manuscript. With specific reference to the demand-side component, we must acknowledge that our findings will not be illustrative of urban realities. Further research, relying on different primary sampling frames, is needed to explore implementation realities and early effects in urban settings.

4. Conclusions

The study design presented represents the first effort to jointly evaluate implementation processes and early effects of the largest government-funded health insurance scheme ever launched in India. We hope that having described our study design in detail will provide inspiration for other researchers engaging in analysis of emerging social health protection schemes in India and in the broader region.

Author Contributions

Conceptualization, M.D.A., S.B. (Stephan Brenner), C.S., D.P. (Divya Parmar), S.S., S.Z., S.B. (Sharmishtha Basu) and N.J.; methodology, M.D.A., S.B. (Stephan Brenner), C.S., D.P. (Divya Parmar), S.S., D.P. (Diletta Parisi) and C.W., writing—original draft preparation, M.D.A., S.S., C.S., S.B., D.P. (Diletta Parisi), D.P. (Divya Parmar) and C.W.; writing—review and editing, all authors; project administration, S.M. and R.N.; funding acquisition, M.D.A., S.B. (Stephan Brenner), C.S., D.P. (Divya Parmar), S.S., S.Z., S.B.(Sharmishtha Basu) and N.J. All authors have read and agreed to the published version of the manuscript.


This work was supported by the Indo-German Social Security Programme, Deutsche Gesellschaft für Internationale Zusammenarbeit GmbH (GIZ) with support from the German Federal Ministry for Economic Cooperation and Development (BMZ) and the Bill & Melinda Gates Foundation.


We are grateful to Maria Paola Bertone for her comments on an early protocol draft and to the teams both at IQVIA and at Nielsen who have worked already on the project and will continue to do so in the coming weeks. We are also grateful to all people in India who have so far accepted to share their experiences and views with us: communities, healthcare providers, and policy makers. A special thanks to our project manager Laura Di Lorenzo who keeps our complex team together and ensures communication across stakeholders.

Conflicts of Interest

Three authors (SZ, SB2, NJ) work for the organization that is funding the study, but their contribution towards this work is independent of their policy advising activity and represents exclusively the product of their own views as researchers. The organization and its directors made no attempt to shape the study design. Funding was acquired competitively by the academic team under the leadership of the first author (MDA), with the three authors from the funding organisation becoming involved in study design, tool development, and data collection only at a later stage. The funding agency as such had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.


  1. Forgia, G.; La Nagpal, S. Government-sponsored health insurance in India: Are you covered? 2012; pp. 1–446. Available online: (accessed on 23 October 2020).
  2. International Institute for Population Sciences. India Fact Sheet. Available online: (accessed on 20 October 2020).
  3. Prinja, S.; Chauhan, A.S.; Karan, A.; Kaur, G.; Kumar, R. Impact of Publicly Financed Health Insurance Schemes on Healthcare Utilization and Financial Risk Protection in India: A Systematic Review. PLoS ONE 2017, 12, e0170996. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  4. Sood, N.; Bendavid, E.; Mukherji, A.; Wagner, Z.; Nagpal, S.; Mullen, P. Government health insurance for people below poverty line in India: Quasi-experimental evaluation of insurance and health outcomes. BMJ 2014, 349, g5114. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  5. Health Insurance in India: The Rashtriya Swasthya Bima Yojana Assessing Access for Informal Workers. 2012. Available online: (accessed on 23 October 2020).
  6. Karpagam, S.; Vasan, A.; Seethappa, V. Falling Through the Gaps: Women Accessing Care under Health Insurance Schemes in Karnataka. Indian J. Gend. Stud. 2016, 23, 69–86. Available online: (accessed on 23 October 2020). [CrossRef] [Green Version]
  7. Nandi, S.; Dasgupta, R.; Garg, S.; Sinha, D.; Sahu, S.; Mahobe, R. Uncovering Coverage: Utilisation of the Universal Health Insurance Scheme, Chhattisgarh by Women in Slums of Raipur. Indian J. Gend. Stud. 2016, 23, 43–68. Available online: (accessed on 23 October 2020). [CrossRef]
  8. Nandi, S.; Schneider, H.; Dixit, P. Hospital utilization and out of pocket expenditure in public and private sectors under the universal government health insurance scheme in Chhattisgarh State, India: Lessons for universal health coverage. PLoS ONE 2017, 12, e0187904. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  9. Shaikh, M.; Peters, S.A.; Woodward, M.; Norton, R.; Jha, V. Sex differences in utilisation of hospital care in a state-sponsored health insurance programme providing access to free services in South India. BMJ Glob. Heal. 2018, 3, e000859. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  10. Nagpal, S. Expanding Health Coverage for Vulnerable Groups in India. Available online: (accessed on 23 February 2019).
  11. Angell, B.J.; Prinja, S.; Gupt, A.; Jha, V.; Jan, S. The Ayushman Bharat Pradhan Mantri Jan Arogya Yojana and the path to universal health coverage in India: Overcoming the challenges of stewardship and governance. PLoS Med. 2019, 16, e1002759. Available online: (accessed on 19 July 2019). [CrossRef] [PubMed] [Green Version]
  12. Morton, M.; Nagpal, S.; Sadanandan, R.; Bauhoff, S. India’s Largest Hospital Insurance Program Faces Challenges In Using Claims Data To Measure Quality. Heal. Aff. 2016, 35, 1792–1799. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  13. Indo German Social Security Programme. Process evaluation of India’s health insurance scheme for the poor rashtriya swasthya bima yojana in north east india Paper I. Available online: (accessed on 10 December 2019).
  14. Rajasekhar, D.; Berg, E.; Ghatak, M.; Manjula, R.; Roy, S. Implementing Health Insurance: The Rollout of Rashtriya Swasthya Bima Yojana in Karnataka. Econ. Polit. Wkly 2011, 20, 56–63. [Google Scholar]
  15. Virk, A.K.; Surender, R. Implementing National Health Insurance In India: Lessons For Policymaking And Practice In Low-Income Countries. Br. J. Interdiscip. Stud. 2016, 3, 1–19. Available online: (accessed on 10 December 2019).
  16. Maurya, D.; Ramesh, M. Program design, implementation and performance: The case of social health insurance in India. Heal. Econ. Policy Law 2018, 14, 487–508. [Google Scholar] [CrossRef] [PubMed]
  17. Virk, A.; Atun, R. Towards universal health coverage in India: A historical examination of the genesis of Rashtriya Swasthya Bima Yojana – The health insurance scheme for low-income groups. Public Heal 2015, 129, 810–817. [Google Scholar] [CrossRef] [PubMed]
  18. Nandi, A.; Laxminarayan, R.; Holtzman, E.P.; Malani, A. The need for better evidence to evaluate the health & economic benefits of India′s Rashtriya Swasthya Bima Yojana. Indian J. Med Res. 2015, 142, 383–390. [Google Scholar] [PubMed]
  19. Government of India. Ayushman Bharat—National Health Protection Mission | National Portal of India. Available online: (accessed on 9 July 2018).
  20. Policy & Guidelines Ayushman Bharat. Available online: (accessed on 16 September 2019).
  21. Status of Implementation of PM-JAY Across States Ayushman Bharat I National Health Authority GoI. Available online: (accessed on 3 February 2020).
  22. Levesque, J.-F.; Harris, M.F.; Russell, G. Patient-centred access to health care: Conceptualising access at the interface of health systems and populations. Int. J. Equity Heal. 2013, 12, 18. Available online: (accessed on 17 July 2018). [CrossRef] [PubMed] [Green Version]
  23. Creswell, J.W.; Plano Clark, V.L. Designing and conducting mixed methods research. 457p. Available online: (accessed on 17 July 2018).
  24. Tashakkori, A.; Teddlie, C. Sage handbook of mixed methods in social & behavioral research. 893p. Available online: (accessed on 17 July 2018).
  25. Patton, M.Q. Qualitative research & evaluation methods: Integrating theory and practice. 806p. Available online: (accessed on 23 July 2018).
  26. Gale, N.K.; Heath, G.; Cameron, E.; Rashid, S.; Redwood, S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. Available online: (accessed on 23 July 2018).
  27. Smith, J.; Firth, J. Qualitative data analysis: Application of the framework approach. Nurse Res. 2011, 2, 52–62. Available online: (accessed on 23 July 2018). [CrossRef] [PubMed] [Green Version]
  28. QSR International. NVivo 12.1. Available online: (accessed on 1 August 2019).
  29. World Health Organization. Service Availability and Readiness Assessment (SARA) An Annual Monitoring System for Service Delivery Reference Manual; WHO: Geneva, Switzerland, 2014.
  30. Government of India. Provider Network Under Pm-Rssm Empanelment and Allied Process. 2018. Available online: (accessed on 23 October 2020).
  31. Iacus, S.M.; King, G.; Porro, G. Causal Inference without Balance Checking: Coarsened Exact Matching. Politi- Anal. 2012, 20, 1–24. [Google Scholar] [CrossRef]
  32. Adams, P. Exploring social constructivism: Theories and practicalities. Education 3-13 2006, 34, 243–257. [Google Scholar] [CrossRef]
  33. Brownson, R.C.; Colditz, G.A.; Proctor, E.K. Dissemination and Implementation Research in HealthTranslating Science to Practice; Oxford University Press: Oxford, UK, 2012. [Google Scholar]
  34. DFID. How To Note Political Economy Analysis Contents. 2009. Available online: (accessed on 23 July 2018).
  35. De Allegri, M.; Brenner, S.; Kambala, C.; Mazalale, J.; Muula, A.S.; Chinkhumba, J.; Wilhelm, D.; Lohmann, J. Exploiting the emergent nature of mixed methods designs: Insights from a mixed methods impact evaluation in Malawi. Health Policy Plan. 2020, 35, 102–106. Available online: (accessed on 7 April 2020). [CrossRef] [PubMed]
  36. Lavrakas, P. External Validity. In Encyclopedia of Survey Research Methods; Sage Publications, Inc.: New York, NY, USA, 2013. [Google Scholar]
Figure 1. Conceptual framework, adapted from Levesque et al., 2013 [22].
Figure 1. Conceptual framework, adapted from Levesque et al., 2013 [22].
Ijerph 17 07812 g001
Figure 2. Study components, study objectives, and methods.
Figure 2. Study components, study objectives, and methods.
Ijerph 17 07812 g002
Figure 3. Study design.
Figure 3. Study design.
Ijerph 17 07812 g003
Figure 4. Study states.
Figure 4. Study states.
Ijerph 17 07812 g004
Table 1. Study states, implementation modes, and districts.
Table 1. Study states, implementation modes, and districts.
StateImplementation ModelDistrictsPreviously Implemented GFHI in 2018
BiharTrust modeGaya, Muzaffarpur and PatnaNone
ChhattisgarhMixed mode *Raigarh and BilaspurMukhyamantri Swasthya Bima Yojana
GujaratMixed modeAhmedabad and SuratMukhyamantri Amrutam
KarnatakaTrust modeRaichur and TumkurVajpayee Arograsri Scheme
MeghalayaInsurance modeSouth West Garo Hills and East Khasi Hills Megha Health Insurance Scheme
Tamil NaduMixed modeCoimbatore and SivagangaiChief Minister’s Comprehensive Health Insurance Scheme
Uttar PradeshTrust modeAllahabad, Ghazipur and RampurNone
* At the time of study conceptualization, Chhattisgarh operated as a mixed mode; as of October 2019, it transitioned to trust mode.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Allegri, M.D.; Srivastava, S.; Strupat, C.; Brenner, S.; Parmar, D.; Parisi, D.; Walsh, C.; Mahajan, S.; Neogi, R.; Ziegler, S.; et al. Mixed and Multi-Methods Protocol to Evaluate Implementation Processes and Early Effects of the Pradhan Mantri Jan Arogya Yojana Scheme in Seven Indian States. Int. J. Environ. Res. Public Health 2020, 17, 7812.

AMA Style

Allegri MD, Srivastava S, Strupat C, Brenner S, Parmar D, Parisi D, Walsh C, Mahajan S, Neogi R, Ziegler S, et al. Mixed and Multi-Methods Protocol to Evaluate Implementation Processes and Early Effects of the Pradhan Mantri Jan Arogya Yojana Scheme in Seven Indian States. International Journal of Environmental Research and Public Health. 2020; 17(21):7812.

Chicago/Turabian Style

Allegri, Manuela De, Swati Srivastava, Christoph Strupat, Stephan Brenner, Divya Parmar, Diletta Parisi, Caitlin Walsh, Sahil Mahajan, Rupak Neogi, Susanne Ziegler, and et al. 2020. "Mixed and Multi-Methods Protocol to Evaluate Implementation Processes and Early Effects of the Pradhan Mantri Jan Arogya Yojana Scheme in Seven Indian States" International Journal of Environmental Research and Public Health 17, no. 21: 7812.

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop