Combining Theory-Driven Evaluation and Causal Loop Diagramming for Opening the ‘Black Box’ of an Intervention in the Health Sector: A Case of Performance-Based Financing in Western Uganda

Increased attention on “complexity” in health systems evaluation has resulted in many different methodological responses. Theory-driven evaluations and systems thinking are two such responses that aim for better understanding of the mechanisms underlying given outcomes. Here, we studied the implementation of a performance-based financing intervention by the Belgian Technical Cooperation in Western Uganda to illustrate a methodological strategy of combining these two approaches. We utilized a systems dynamics tool called causal loop diagramming (CLD) to generate hypotheses feeding into a theory-driven evaluation. Semi-structured interviews were conducted with 30 health workers from two districts (Kasese and Kyenjojo) and with 16 key informants. After CLD, we identified three relevant hypotheses: “success to the successful”, “growth and underinvestment”, and “supervision conundrum”. The first hypothesis leads to increasing improvements in performance, as better performance leads to more incentives, which in turn leads to better performance. The latter two hypotheses point to potential bottlenecks. Thus, the proposed methodological strategy was a useful tool for identifying hypotheses that can inform a theory-driven evaluation. The hypotheses are represented in a comprehensible way while highlighting the underlying assumptions, and are more easily falsifiable than hypotheses identified without using CLD.

-Three kinds of incentives: (1) Every interested facility needs to go through a qualification process in which they are scored based on structural measures. "Score > 85%" = the facility is qualified and receives a first incentive (money for equipment and medicines). "65% > score < 85%" = facilities receive a small grant in order to attain 85%. "Score < 65%" = not qualified, no financial support.
(2) Quality bonus = star rating (1 * to 5 *) for each facility based on a structural quality score, each star corresponds to a fixed lump sum grant depending on the administrative level of the health center. (HC III = 1 million UGX per star rating; HC IV = 2 million UGX per star rating).
(3) Quantity bonus = Fixed incentive for each service that was performed and recorded according to the pre-defined quality standards of the MoH.
-Depending on the scores, the amount received per quarter can go up to 20 million UGX for HC III and 40 million UGX for HC IV.
-Incentives are paid to the facility, 25% of the funds recceived can be used for performance-based incentives for the health workers.
-Quarterly paid after a reporting and verification process of about 6 weeks. -Incentives are paid on a bank account of the facility, yet the facility needs the authorization of the district accounting officer to access it. Every facility receives monthly and yearly targets for different services from the MoH. In most cases these are hung out in the different departments or in the waiting room. Also, each facility has a Quality Improvement team, which also puts forward working points. However, these were not always very active before the project. NGOs also often have targets in specific areas (e.g., drug store management).

Service and quality measures
-The pre-qualification measures correspond to infrastructure, equipment and human resources requirements.
-The quarterly qualification tool contains cross cutting issues of quality of service like hygiene and working environment, availability of medicines and consumables, staffing levels, use of guidelines and other management tools, implementation of performance improvement strategies, etc.
-Quality indicators and measures were drawn from the national guidelines, hence are per definition owned by the country.
-The timeframe of the measures is 3 months.
Facilities are supposed to report in the HMIS books of the MoH and make a monthly report which is send to the district. Most HC III do not have computers for the records office nor a specific records assistant. The HC IV and some HC III do have a computer and a records assistant. However, the hard copies are still needed which creates double work.

Monitoring and verification system
-The facilities first perform a self-assessment of their own performance, after which they are being visited by the extended-district health management team (E-DHMT = DHMT + representative from the concurrent medical bureau) for verification.
-Verifiers are not necessarily medical doctors.
-Performed each quarter.
-Quantity measures are drawn from the registers.
-No extra costs are envisaged, yet the district receives incentives that can be used to cover the costs of the visits (e.g., car fuel).
The MoH is both purchaser and provider. Oversight is mainly done by the district health office, the sub-district and the local government. PNFP facilities are also answerable to their respective religious medical bureau which is the owner of the PNFP facilities.

Split of functions
-The split of functions is limited. The MoH/MoFPED is envisaged to be the purchaser although in the project the BTC is the natural fund holder/purchaser. The provision is being done by the facilities under the aegis of the medical bureaus. The health coordinators of the latter together with the DHMT are responsible for the verification. To avoid collision, the coordinators of the medical bureaus will not be present during the verification visits of their own facilities. PNFP facilities have more autonomy than public facilities. Whereas services at public facilities are free of charge and these facilities thus receive all their funds from the government through fixed line budgets, the PNFPs receive only a small part of their budget from the government with fixed budget lines. They have full autonomy on the use of the received user fees and free donations. Funds from NGOs are mostly earmarked, not giving any autonomy to the facility (e.g., Baylor Uganda).

Autonomy
-Facilities cannot use the budget for infrastructural investments or salary top ups with no link to performance.
-Facilities have to use the funds as fixed in the business plan, which they compose themselves with support from the district.
-The qualification bonus can only be used for medicines and small equipment in order to bridge the first quarter in which no funds are received yet. Only a fixed part of the received funds (25%) can be used for performance-based incentives for the health workers.
-In order to retrieve money from their bank account facilities have to justify to and get permission from the district accountancy officer. Information board often have statistics on the performance of the health facility, however, these are not always accurate or up-to-date. Accountability is mainly towards the health unit management committee (HUMC) and the district.
Accountability arrangements -Facilities are obliged to communicate their results, exact prices and the money received to the community via their information boards.
-Performance reports are send to the district which aggregates the data and sends it through to the national level. The community is involved through their representatives in the HUMC. The UCMB has its own patient satisfaction surveys.

Community involvement
-Community involvement is very limited. No new tasks were assigned to the community. The community is represented in the HUMC via their local leaders. The HUMC has to underwrite the business plan. No incentives are given to the community leaders, nor are their incentives based on the participation of the community.
-Patient-satisfaction surveys will be performed but incentives are not based on them, they are purely informative. The activity of the HUMC differed strongly across facilities. However, no explicit business or strategic plan was used in most of the facilities.
Planning arrangements -The business plan is meant to set out how the facility aims to reach their goals using the expected PBF funds. It is focused on the PBF indicators and the three month timeframe.
-Misappropriation or misuse of funds is being sanctioned -Business plan is approved by the HUMC and the district health office All the facilities of the medical bureaus go through a light accreditation process. Workshops and meetings exist with mainly the in-charges attending them. The districts are seen to be lacking the capacities to really support the facilities in there functioning. Health workers strongly appreciate the supervision received and often stated that they would like to welcome the supervisors more often. The user fees at the PNFP facilities were seen as important barriers to access to care. Since services at public facilities are free, many patients divert to them instead of the PNFP facilities. This lead to a very low number of patients in the PNFP facilities.
Ancillary components -Certification: every interested facility needs to go through a pre-qualification process in which it performs a self-assessment after which this assessment is being verified. Facilities with a score of 85% and above are allowed in the project. This is like a certification process.
-The in-charges of the facilities receive workshops on the specificities of the project which they have to communicate to their staff.
-The district PBF focal persons have quarterly meetings during which the performance of the different districts and facilities are being discussed and solutions are discussed.
-Support supervision to the facilities from the district is being incentivized.
-Facilities are obliged to lower the user fees below a maximum. Patients should receive all care, lab tests and medication for this fee. It thus entails a shift from fee-for-service to case-based payment.