Monitoring Methods for Systems-Strengthening Activities Toward Sustainable Water and Sanitation Services in Low-Income Settings

Abstract

To address the complex challenge of sustaining basic water and sanitation services in low income settings, international organizations and local and national government entities are beginning to design and implement interventions explicitly aimed at addressing system weaknesses. Often referred to as “systems approaches,” these interventions seek to understand, engage with, and positively influence the network of actors and the interacting factors that deliver services. As WASH sector assistance and support activities shift toward systems approaches, many associated intermediate results and desired outcomes become less quantifiable than those of more traditional WASH activities. This paper reviews systems approaches, evaluation methodologies, and several applications in East Africa, at varying geographic scales. Early findings from the application of outcome mapping and system-wide assessments within the USAID-funded Sustainable WASH Systems Learning Partnership (SWS) indicate the importance of including both within an overall monitoring approach to support systems strengthening of water and sanitation services. The views and opinions expressed in this paper are those of the author(s) and not necessarily the views and opinions of the United States Agency for International Development, or the U.S. Government.

1. Introduction

Over the past 50 years, billions of dollars have been invested in the water, sanitation, and hygiene (WASH) sector by national and regional governments in low- and middle-income countries, as well as by international donors and non-governmental organizations. These investments largely prioritized the installation of new water and sanitation infrastructure under the assumption that local communities and regional governments would be capable of managing operations, maintenance, and service delivery [1,2]. Despite this investment, infrastructure failure rates remain high sector-wide, with electrical pumps, hand pumps, piped water systems, latrines, and other WASH infrastructure falling into disrepair far sooner than their design life. The assumption that local entities would maintain infrastructure and sustain services did not hold true, with local funding and accountability for service delivery falling short of what was required for sustained services, resulting in a high degree of service disruptions [3,4,5,6,7]. As infrastructure for water services fail, the cost of water delivery increases drastically, as water must then be provided through less cost-efficient or effective means.

In 2015, recognizing the disconnect between infrastructure investment and sustainable services, the top-line goals for the WASH sector shifted from the Millennium development goals (MDGs), which largely focused on increased access to infrastructure, to the sustainable development goals (SDGs), which focus on sustained access to adequate services [8]. To achieve sustainable services as prioritized by the SDGs, sector stakeholders, including national and local governments and WASH sector donors and implementers, are looking beyond the construction and provision of infrastructure to the systems—the dynamic network of factors and actors that support and sustain functional services (see Figure 1 below). In doing so, there is increasing recognition of the complexity of the systems that deliver and sustain services, as well as the challenges associated with strengthening their capacity.

To address the complex challenge of sustaining basic services and benefit from the full lifecycle value of the associated infrastructure, sector stakeholders are beginning to design and implement interventions explicitly aimed at addressing system weaknesses [9]. Often referred to as “systems approaches,” these interventions seek to understand, engage with, and positively influence (i.e., strengthen the functions of) the network of actors and the interacting factors that deliver services, theoretically leading to improved service delivery and sustainability. Such an approach (1) seeks to understand the complexity, interactions, and interdependencies between actors and factors in a deliberate, rigorous manner; (2) acts based on this understanding; and (3) regularly adapts to feedback and changing conditions [10].

In addition to the shifts in WASH programming toward greater inclusion of systems approaches, national governments are also rethinking and redesigning sector frameworks, policies, and monitoring indicators and approaches, in order to better align country-wide efforts toward sustainability. Programs like the Ethiopia ONE WASH National Programme [11], the currently in-draft National Framework for Operation and Maintenance of Rural Water Infrastructure in Uganda [12], and the Kenya [13] represent meaningful shifts in national policies toward greater emphasis on sustaining existing infrastructure and improving services sustainability.

As WASH sector assistance and support activities shift toward systems approaches, many associated intermediate results and desired outcomes become less quantifiable than those of more traditional infrastructure-building WASH activities. While traditional measurements of an activity’s successes and impacts focus on the recipients served (e.g., “number of persons with improved access”) or the quality of the service (e.g., “liters/capita produced”) [14], new metrics are required to monitor and evaluate the impacts or outcomes of systems approaches. Existing indicator frameworks and metrics, such as the SDG Goal 6 indicators, fail to capture the changes in a system and its strength. As development partners increasingly change their approaches to improving services, the associated monitoring frameworks, as well as the overarching priorities of such frameworks, must evolve as well in order to effectively monitor, evaluate, and learn from systems approaches.

2. Components of a Systems Approach

2.1. General Theory of Change for a Systems Approach

In response to growing recognition of the pivotal role local systems play in supporting development outcomes, the United States Agency for International Development (USAID) developed the Local Systems Framework, which provides guidance on effective perspectives through which local systems can be understood, as well as the importance of strengthening local systems in order to sustain outcomes. The Local Systems Framework defines sustainability as the “ability of a local system to produce desired outcomes over time.” Within the WASH sector, sustainability translates to functional, safe, and reliable services [15]. As such, in order to achieve sustainable services, an approach that seeks to understand and engage with the local systems that deliver such services is needed.

In order to develop an effective monitoring strategy for systems-based interventions, an overall theory of change is necessary. This is true for development activities broadly, including WASH activities. Once established, an overall theory of change provides the basis of monitoring, indicating both the intermediate results for evaluation and associated activities to be monitored [16]. Differing rubrics exist regarding how to segment and understand systems, as well as how to engage with and seek to change them. One such approach is USAID’s “5Rs Framework,” which highlights the importance of Results, Roles, Relationships, Rules, and Resources within a local system, as well as the ways in which these components drive sustainability [17].

Recently, the USAID-funded Sustainable WASH Systems Learning Partnership (SWS) developed a general theory of change governing how a systems approach to water and/or sanitation services can theoretically result in increased service sustainability. This theory of change is provided in Figure 2 below, followed by a brief description.

This includes the following broad steps:

• Systems mapping and analysis: By mapping the system, including both the network of actors and the systems of interacting factors, key stakeholders will better understand what is driving outcomes and identify weaknesses and potential leverage points as entry points for systems-strengthening interventions.
• Systems strengthening and engagement: Based on the identified weaknesses, gaps, or disconnects within the existing system, interventions are designed and deployed to specifically address them and strengthen the overall function of the system.
• Service delivery impacts: In the near term, the resulting impacts on the system as a whole, or, in the longer term, the impacts on the sustainability of the service itself, can be observed, through both quantitative and qualitative monitoring.

2.2. Considering Systems Complexity

A variety of methods have been applied to monitor and evaluate WASH interventions, ranging from routine, implementer-collected monitoring data to independent randomized, controlled trials. However, many of the most widely applied indicator sets, such as those used by the WHO/UNICEF Joint Monitoring Programm, neglect the full complexity of systems engagement and change [14,18]. While some implementing entities have begun using system-wide assessments (e.g., sector scorecards or building blocks assessments), when applied alone these assessments capture a simplified, static view of the system within predefined categories that can fail to fully capture or adapt to emergent changes within the system [19].

Innovative monitoring methods and indicators can capture change within complex systems [20]. Often, these methods are more flexible in nature and can capture the shifts that traditional WASH indicators fail to identify: Changes in actor behaviors, shifts in network structures and communication channels, and qualitative changes that occur within major systems components [21].

In order to be effective, such methods must have a wide enough view to capture the full system and actor network, and be flexible enough to capture emergent results, outcomes, and changes. Without flexible methods, emergent or unpredicted findings can be missed, which can hinder effective programmatic adaptation or decisions, especially in complex contexts such as those that support a local service [22].

2.2.1. Incremental and Flexible Methods

Given that the changes in actor networks or factor strength can be hard to predict from project onset, monitoring requires flexibility [20,23]. Systems monitoring tools must be able to accommodate changing technical areas of focus, data collection timing, and even the data collected. Incrementally applied monitoring approaches that can accept emergent prioritized technical areas are therefore more effective at capturing the effectiveness of systems change approaches [24]. Examples of such monitoring approaches include outcome harvesting and most-significant change, both of which first focus on emergent or identified outcomes and then seek to identify ways in which interventions may have contributed to their realization [25].

Additionally, systems, as well as related policies, institutions, and organizational practices, often change slowly. Meaningful changes in the overall system often cannot be fully realized within a typical 5-year project cycle. Given this limitation, monitoring should focus on tracking stepwise progress toward a broader goal. This might include capturing more modest, incremental contributions achieved during a program cycle. By applying iterative monitoring techniques and setting realistic boundaries that differentiate between the factors a project intends to address and those outside of its purview, the impacts of an activity on a system’s strength can be captured, and subsequent actions to further progress can be designed more effectively.

2.2.2. Monitoring the Full System of Actor

When considering an overall monitoring approach for systems activities, a suite of methods that consider both factors and actors will help ensure that systems change and impacts are captured and that programmatic shifts or changes can be fully considered. Given the complexity of the network of actors involved in WASH service delivery—rural water services often involve multiple tiers of local government, a variety of technical ministries, the private sector, civil society, and both national and international non-governmental organizations—shifts in behaviors and changes in organizational relationships can have widespread system effects. By considering actors and their behaviors, as well as the relevant technical factors, systems changes can be effectively monitored, and interventions adapted accordingly.

2.3. Understanding and Setting Effective Boundaries

Setting effective boundaries around local systems helps to focus interventions, and in turn monitoring activities, on the critical factors, actors, and dynamics that contribute to, or hinder, the delivery of a given WASH service. Setting boundaries that are too broad makes identifying critical factors or actors challenging, while setting boundaries that are too confined excludes potentially key or influential system components [25]. In order to properly understand and set systems boundaries, it is critical to consider those boundaries that define what can be controlled, what can be influenced, and what cannot be influenced yet is of concern to the local system.

Within Figure 3, the “Sphere of Control” refers to the direct boundaries of system-strengthening activities or interventions, including the actors and factors directly engaged (alternatively referred to as boundary partners) [26]. Illustrative examples might include recipients of training (actor) or an increase in information sharing resulting from the creation of a database (factor).

The “Sphere of Influence” refers to areas that are influenced by the outputs and outcomes of the activities or interventions within the Sphere of Control. These can include changes that cannot be achieved through a direct intervention or that require multiple interventions and greater time to achieve. Changes in behaviors (e.g., decision-making) often fall within the Sphere of Influence. Illustrative examples might include improved coordination between local and regional governments on issues such as budget allocation and regulatory enforcement resulting from increased communication and better information management (both an improvement in the actor network and strengthened financial factors).

Finally, the “Sphere of Concern” encompasses all factors or actors that might influence a service but fall outside of an activity’s Sphere of Influence. Illustrative examples include changes in national-level policy and regulation.

Monitoring and evaluation methodologies for systems approaches must capture both (1) the oft-intangible, qualitative changes in stakeholder behaviors and actions within the sphere of influence and (2) changes to the system itself, and the services it delivers, within the Sphere of Concern. These are the two pillars required of a monitoring approach to systems approaches, as well as the resulting system change.

2.4. WASH Systems Monitoring Methods

The Local Systems Framework states that “it is unlikely [that]… a single monitoring method will be sufficient to capture system change” [15]. This paper recommends a three-pronged approach that includes:

• Qualitative monitoring methods, focused primarily on the process of systems change;
• Analytical monitoring and analysis of the overall structure of the system and actor networks;
• Indicators or proxy indicators that adequately capture impacts and results of activities, while also considering the complex nature of such shifts and impacts.

In alignment with the Local Systems Framework, SWS is applying two multifaceted approaches for monitoring systems approaches and associated systems change, both of which use qualitative and quantitative methods. These approaches include:

• Implementation Science: Using an implementation science framework to iteratively evaluate a water service intervention, we use time series analysis to link measures of water service delivery to intervention activities. This method is being applied in the Afar region of Ethiopia and in northern Kenya;
• Outcome Mapping: Outcome mapping is a process of setting desired progress markers for key actors and monitoring their achievement through the creation and analysis of regular project journaling. Through this, connections between individual actions, discussions, or interventions, as well as resulting impacts on broader systems components, can be explored, improving the understanding of how systems approaches work and what impacts can be realized through their application. This is being applied in Kitui County in Kenya, in two regions and two small towns in Ethiopia, and in four districts in Uganda;
• System-Wide Assessments: System-wide assessments include those that explore system factors, as well as map and assess actor networks. System-wide assessments regularly evaluate broad shifts in the system over time, providing an overall indication of how systems are changing concurrently with an intervention.

2.4.1. Implementation Science

Responsive and iterative system monitoring methods are consistent with core methods in implementation science, an emerging field that seeks to advance the effectiveness of activities and interventions in a range of sectors [27]. In the context of environmental health, implementation science seeks to narrow the “know-do” gap between a given intervention and observed results under real-world conditions [28], ideally facilitating the development and delivery of better programming that has a higher likelihood of achieving the desired impact. A range of tools, techniques, and analytical platforms are available for integration in advanced monitoring and evaluation and implementation science, representing smart uses of feedback in helping activities deliver on their promise. Importantly, unlike the traditional randomized clinical trial, this approach allows information from formative evaluation to be utilized during the conduct of the study to optimize implementation and create a framework for assessing the impact of evolving implementation strategies.

For example, we are applying an interrupted-time series (ITS) experimental design to evaluate whether providing the Afar Regional Water Bureau with near-time data from approximately 150 instrumented boreholes in conjunction with training and capacity building activities conducted through USAID-funded activities results in improved water service delivery. The boreholes included in this study are located in the Afar region of Ethiopia [29]. These sites were selected by the Afar Regional Water Bureau, which is the governing institution for water service delivery in the region.

The ITS design is primarily the combination of two data sets: Qualitative monitoring of the systems-strengthening activities, and quantitative analysis of the functionality data delivered by the real-time instrumentation. The ITS design is widely regarded as the most rigorous of the experimental designs in which randomization is infeasible, as is the case with the current effort in Afar. ITS designs require collecting data on the outcome of interest (borehole functionality) at multiple time points both before and after an intervention to test whether the intervention has an effect significantly greater than the baseline (pre-intervention) secular trend [30]. Although most commonly used in the context of single time series, the ITS design is readily adapted to situations involving multiple sites and/or successive interventions.

Figure 4 below illustrates the ITS study design concept. In this application, measures of borehole pump functionality are taken continuously using sensor data. Periodic, well-documented intervention events, such as a new monitoring platform, a new operations and maintenance protocol, or new budget allocations, are presumed to influence borehole functionality. Meanwhile, other variations and changes in borehole functionality may be observed, attributable to seasonal influences on water system use or to other non-intervention-related changes such as other governance or community activities. With an ITS study design, we are able to monitor our outcome continuously, while having statistically significant attribution of intervention events to any observed changes in the mean level or slope (rate of change) of borehole function. While this study design is fundamentally similar to a simple pre-post analysis, the statistical strength is achieved through the continuous measure of the outcome variable, with the well-documented timing of the intervention events.

The ITS analysis is being conducted over a several-year period. The outcome of interest in ITS study is borehole functional time, which is operationalized as the count of functional borehole days per period (i.e., week, month, quarter) and collected on a continuous basis during the study period via installed sensors. Data on functional time is aggregated into weekly time intervals, yielding a total of at least 52 weeks of time series data for each borehole. Mixed-effects piecewise (segmented) regression is used to model the change in the trajectory of functional time following implementation of the intervention, with borehole and woreda random effects to account for the nested structure of the data, and rainfall as a covariant. Further, data collected as part of the implementation evaluation are being used to examine how borehole and woreda-level factors contribute to variation in intervention effectiveness.

2.4.2. Outcome Mapping

Outcome mapping is a qualitative methodology that focuses on measuring changes in the behavior and relationships among individuals, groups, and organizations with whom a project works most closely [26,31]. Qualitative, behavior-centric monitoring helps to capture the incremental, or more nuanced, changes in system function that might elude siloed, category-based monitoring systems such as a system-wide scorecard or building blocks assessments [19]. When category-based, system-wide monitoring is applied on its own, it tends to focus only on what a system looks like rather than how it functions (e.g., a policy exists but is not being implemented), overlooking the interconnections and interdependencies that drive performance and encouraging a static view of the system through fixed indicators that do not allow room for capturing emergent changes [19]. However, when used in tandem, outcome mapping seeks to compliment system-wide monitoring tools by looking at changes in function of target actors and linking their changes in behaviors to factors or actors.

In order to apply outcome mapping, a set of desired changes for actors in the system are defined and regularly tracked for achievement using a project journal to capture changes as they unfold. This process allows a project team to monitor its progress in influencing the behaviors, relationships, and actions of local people and organizations that drive systems change.

Under SWS, outcome mapping is being applied for ten different multi-stakeholder platforms, each of whom is implementing systems-strengthening activities. These platforms comprise six districts in Ethiopia and Uganda focused on rural water service delivery, two small towns in Ethiopia focused on sanitation service delivery, and one county in Kenya focused on rural water service delivery via small piped networks.

For each multi-stakeholder platform, the team first develops an overall goal statement describing actors’ ideal behavior, as well as up to 15 progress markers describing a progression of changed behaviors leading to the vision statement. The goal statements and subsequent progress markers can seek to influence stakeholder behaviors within a broad range of possible outcomes, from aligning local actors with national frameworks and policies, such as the Ethiopia ONE WASH National Programm [11], to the functioning of the multi-stakeholder platform itself. This is a simplification of the original outcome mapping methodology, which recommends defining a unique outcome challenge and progress markers for each stakeholder type. The revised methodology places greater emphasis on the collective action approaches used within the multi-stakeholder platforms and reduces the data collection burden imposed on project staff. See Figure 5 below for an illustration of both an overall goal statement and a set of associated progress markers.

Under SWS, field teams regularly self-record their observations against the predefined progress markers set at the start of the project. Observations include any qualitative information available to the project team, such as public remarks by decision-makers at meetings (recorded in the minutes), references to news reports or public speeches, notes from interviews with stakeholders, or personal observations from team members. This type of qualitative data gathering provides frequent and relevant feedback to teams that is more immediately usable than more traditional forms of monitoring. See Figure 6 below for an illustrative reporting example against WASH-focused performance markers.

Finally, on a semi-annual basis, partner teams summarize their observations for each progress marker and assign a low, medium, or high rating to assess progress. In order to mitigate against reporter or collector bias, data are independently reviewed and cross-referenced with other project documentation, such as sector reports, interview transcriptions, and other information that might corroborate recorded observations. Mitigating biases is discussed further under the Initial Findings and Considerations section.

2.4.3. System-Wide Assessments

As a complement to monitoring specific progress markers through outcome mapping, we are applying two system-wide assessments to regularly evaluate broad shifts in the overall system: One that evaluates the key factor pillars of service delivery, and another that maps and analyzes the actor networks.

System-Wide Factor Assessment

There are a number of system-wide factor assessments currently being applied within the WASH sector [32,33,34,35]. Within SWS, the system-wide factor assessment applied includes all key components of a system that impact sustainability, beyond just those that the project is targeting. It is structured around five main components of a system that delivers WASH services: Financial, institutional, environmental, technical, and social (FIETS) [36]. Since this tool tracks large-scale changes in the system, attribution to specific project activities or partners is unlikely, as there is increased uncertainty about specific contributions to any observed results. However, a system-wide assessment allows project teams to see a more complete picture of change and make informed programming decisions. Additionally, unlike the ITS approach, the system-wide assessment seeks to evaluate impacts on the likelihood of WASH services being sustained by assessing the strength and status of key systems components.

Within the SWS activity, we customized the system-wide assessment for the two sub-sectors under the project (rural water and small-town sanitation). In developing the assessment, we reviewed 12 existing frameworks with diverse sets of indicators related to systems components from the WASH sector. We then selected two frameworks—the IRC Building Blocks Assessment (national and district) and the World Bank Water and Sanitation Program (WSP) Enabling Environment Assessment—for direct comparison. We looked at different aspects of sustainability based on the FIETS framework. Throughout the process of developing the assessment, we drew from relevant indicators in the Building Blocks Assessment and WSP Enabling Environment Assessment and further refined and adapted these indicators following consultations with key study staff.

During the consultation process, SWS partners provided input toward defining what each aspect of sustainability would look like for each aspect project. Based on these definitions, the team then developed an assessment scoring system using low, medium, and high criteria for each aspect of sustainability. Figure 7 below provides an example of a full assessment vision for a system-wide sustainability scorecard, specifically for sustainable rural sanitation services.

Partner teams complete the assessment annually for each district or small town within the study. They select a rating of low, medium, or high for each aspect of sustainability based on the pre-defined criteria and describe how they arrived at that rating with supporting data and links to reference materials. SWS then calculates an overall score composed of an unweighted average of the indicators that can be tracked for changes over time. Figure 8 below provides an example of the scoring rubric within a given category, in this case the Institutional category for sustainable rural sanitation services.

In addition to the system-wide assessments, study partner teams are conducting multiple analyses for each case study. The system-wide assessment is not meant to replace these analyses; rather, it provides a structure for synthesizing and summarizing available data and partners’ observations.

Social Network Analysis

Social network analyses (SNA) are approaches to developing maps of stakeholder networks and their interactions and analyzing this network for opportunities and constraints. A subset of SNA, organizational network analysis (ONA), focuses specifically on organizations as the actors. These network approaches can be used in multiple ways within a WASH program cycle, including (1) aiding in activity design through by identifying critical partners or stakeholders, and the network within which they operate, (2) as a monitoring or evaluation tool that captures shifts in network structures and functions, either pre-/post-implementation or iteratively through an activity, and (3) as an intervention themselves, by involving relevant stakeholders in network map development and analysis, which in turn leads to broader systems understanding by stakeholders [37].

These primary steps (see Figure 9 below) can be summarized as follows:

• Network Parameter Definition: Overall network analysis criteria and parameters, such as the overall network boundaries, actor attributes and connection types explored, etc. are defined before data are collected to ensure clear study inputs and outputs;
• Census/Data Collection: Information is collected on each actor and relationship, primarily through standardized interviews, often supplemented by other documentation on actor relationships, such as data sharing or meeting attendee lists and minutes;
• Analysis and Reporting: Based on the information collected, network maps are created and analyzed for constraints and/or opportunities that can be addressed through interventions and/or measured over time. Illustrative constraint/opportunity examples include disconnects between actor groups or clusters, limited centrality of decision-makers, or poor information-sharing among relevant actors.

Once the above steps are completed, a validation of the network map, analysis, and findings is typically held with activity participants to provide context and validate findings.

Under SWS, network analysis is being applied over time in ten localized geographies in four countries. The geopolitical boundaries range from small towns and rural districts to a nation-wide sector working group. It is being applied at both baseline and endline, and in some instances at midline, in order to capture shifts that have occurred concurrently with interventions and activities. Given its application at these varying points within the partnership, network analysis is (1) informing the design of activities and interventions, particularly those around information sharing (e.g., database development), (2) monitoring the impacts of such interventions by capturing shifts that occur within the network of relevant actors, and (3) building local stakeholder awareness of the role networks and actors play in delivering a specific service (e.g., the range of actors involved in maintaining a clean and functional shared latrine for small towns in Ethiopia).

In addition to varying geographic applications, SWS is applying different approaches to network analysis, including methodologies that vary significantly in terms of level of effort required, data collection processes, and analysis performed, with the goal of understanding the resource-output tradeoffs among methods so that WASH sector implementers can choose the method most appropriate, given their analytical needs and resources available.

Figure 10 below illustrates how network analysis can provide a pre- and post-intervention measure of how an actor network functions. Key attributes, such as density, distance, and centrality, are proxy measures for understanding how connected actors are to each other and how reliant a given network might be on a limited set of actors. In Figure 10, the illustrative “after” map shows a network that is more interconnected and less reliant on a select number of central actors when compared with the “before” map [23]. The “after” map shows an increase in the number of ties (or direct connections) among actors and an increase in the overall density (or the number of actual connections versus the number of total possible connections, as a percentage) of the network, indicating that the network shifted toward greater interconnectedness [38].

Under SWS, midline applications were recently completed in Ethiopia, for both district-scale water services and small-town sanitation services. As a summary, the midlines indicated that overall network connectivity and coordination increased, with particular increases in information sharing, which was a focus in many of the target geographies. However, some of the non-governmental actors that were previously identified as critical were not being as engaged as desired. Such findings at midline will support adaptive management and design of interventions and activities, particularly with respect to how partner and stakeholder engagements are designed [39].

3. Discussion

Early findings from the application of both outcome mapping and system-wide assessments indicate the importance of their inclusion within an overall monitoring approach.

Incorporating broader system shifts into localized programming: In Ethiopia, changes in the system-wide factor analysis (as assessed using the rural water sustainability scorecards at the district scale annually) show a weakening system within the broader Sphere of Concern for rural water services. Weakening financial and regulatory components nationally were noted, and the field programs integrated this into their work by prioritizing the development of a master plan: A long-term investment-planning document that can be effectively used for fundraising both for operational and maintenance costs and for capital expenditures. Concurrently through outcome mapping, positive behavioral changes were captured at the district level, including improved public-private actor coordination for operations and maintenance provision, which was a desired result of the collective action approach of the field implementer team. Midline network analysis also indicated areas of focus for the latter half of the program, including the inclusion of key civic and non-governmental entities in decision-making and planning activities.

Emergent, incremental progress demonstrates success and builds support: In Uganda, district-level partners noted the service delivery benefits of the incremental, qualitative nature of outcome mapping. The team’s regular creation and analysis of process documentation (in the form of after-meeting reports and regular activity journaling) allowed incremental progress to be shared team-wide and activities to be modified accordingly in a more time-effective manner. The ability to identify and share quick successes has been critical to increasing stakeholder support for, and engagement with, key activities, such as the development of a district master plan. In general, outcome mapping in Uganda is being identified as potentially more effective than more traditional semiannual (or less frequent) analysis and reporting methods, which are more challenging to consider within the administration of service delivery, given their infrequency.

Resources requirements: Qualitative and systems-based monitoring approaches are new to many implementers of WASH programming. As such, their application requires significant up-front investment and resources, including training for data collection or reporting staff, the creation of new monitoring platforms, and ongoing support to field staff to ensure monitoring is being performed properly. Under SWS, the creation of effective journaling templates for data collection, the completion of regular outcome mapping reporting, and other monitoring activities required significant support by centralized SWS staff during the early years of the partnership. While systems-based monitoring approaches can be very effective at supporting WASH activities, the resources required for their creation and application should be considered at activity inception, especially if they represent shifts in how staff have performed monitoring to-date.

Mitigating bias: Qualitative information can often be subject to collector or reporter bias, especially in the case of first-hand reporting on progress by an implementing partner or local stakeholder (e.g., a local government official reporting on community progress). SWS applies multi-layered reviews by third parties to seek to identify, clarify, or address any areas that might be subject to such biases. For example, semiannual outcome mapping reporting by field-base partners is reviewed for validity and proper justifying documentation by researchers at the University of Colorado Boulder. Any inputs that appear to be potentially subject to collector or reporter biases are discussed and clarified with reporting partners. In general, when integrating qualitative monitoring into water and sanitation programming, it is critical to understand its susceptibility to bias and put in place measures to mitigate it.

The monitoring methods discussed in this paper will continue to be applied for the remaining 18 months under the SWS partnership, after which greater insights into their use in WASH programming more broadly will be disseminated.