Improving Understanding and Management of Uncertainty in Science-Informed Collaborative Policy Processes

Abstract

Decision making in natural resource management must deal with uncertainty. This can be very challenging when there is high uncertainty in the system being managed, particularly when participatory processes are used that potentially involve a wide range of stakeholders with competing interests and values, and where technical resources are limited. In the first phase of the research, a formal review of how uncertainty was managed in a collaborative and community-centred policy process to set water quality and water quantity limits in the Selwyn Waihora catchment in New Zealand. Three recommendations for improvement were identified: (i) increase the transparency of the nature and level of uncertainty considered, (ii) expand the types or sources of uncertainty considered, and (iii) apply a systemic and systematic approach to identifying and prioritising uncertainties. In the second phase, this led to the development of a five-stage conceptual framework incorporating a number of steps and tools designed to facilitate understanding, communicating, and managing uncertainty. The framework was tested and refined using the data from the original Selwyn Waihora policy process. This paper describes our framework, which was found to support understanding, managing, and communicating uncertainty in collaborative processes tasked with developing new environmental policy, despite time and resource constraints.

1. Introduction

Governments need to work with their citizens and other stakeholders to find solutions to complex global and domestic challenges [1]. Decisions, whether made by individuals, members of communities, or government on society’s behalf, must deal with uncertainty [2,3]. As the issues become more complex, different dimensions of uncertainty become more apparent [2]. Sustainable natural resource management (NRM) is a complex real-world problem where some “facts are uncertain, values are in dispute, stakes are high and decisions are urgent” [4]. As managing uncertainty well is critical to making good policy and decisions, robust uncertainty processes are necessary to support sustainable NRM policy development [5]. These uncertainties may relate to the science used to model and understand the biophysical relationships, or to the human–stakeholder relationships, or to the policy and planning frameworks, e.g., in [2,6].

Participation is a key element of contemporary approaches to both NRM [7] and sustainability challenges more broadly [8], and it is recognised as an important factor in creating adaptive policy [9]. In New Zealand, this is manifested in the recent popularity of participatory and collaborative environmental decision making [10,11]. Increasingly in participatory processes, the linear model of science-led environmental decision making is being challenged, as alternative forms of knowledge, such as local and indigenous knowledge, are incorporated [12,13,14], and stakeholders are more involved in the production of knowledge [7]. A parallel and interrelated theme in NRM is the steady evolution towards using integrated environmental modelling, a holistic approach to understanding environmental issues and problems, and for informing subsequent decisions and policies [15,16].

The shift towards integrated modelling and collaborative policy-making processes has implications for understanding, communicating, and managing uncertainty. Model assessment of uncertainty is often done in a final step. However, in order to integrate model results into the collaborative NRM process and increase the effectiveness of knowledge production, uncertainty analysis needs to be an ongoing theme throughout the modelling process [17,18], not just occurring at the end. The shift means that the problems are likely to be framed from a wider range of perspectives and world views [15,19,20] than would be the case if only framed by scientists.

Bijlsma et al. [19] compared stakeholder-based policy development with expert-based policy development and found opposing approaches to managing uncertainties during policy development. While the experts tended to respond to uncertainty by reducing the problem’s scope, the local stakeholders responded by broadening the scope to include all of their important stakeholder criteria. A wider framing brings in a greater number of uncertainties and unknowns because a greater number of aspects are considered. Furthermore, the concerns of the stakeholders are generally not confined to information that can be supplied by single disciplines, but require a greater level of interdisciplinarity and integration [3,15], thus generating new uncertainties.

Uncertainty is interpreted differently by different people [17]. The diverse groups involved in participatory modelling and collaborative policy making may have different norms and expectations regarding the appropriate characterisation of uncertainty and what might constitute reliable evidence [21]. This may also vary within each group. The groups or individuals may use very different language, thus creating uncertainty in the form of ambiguity [18]. This diversity also creates a greater need for good visualisation, communication, and translation tools to enable a wide range of people to understand the modelling and uncertainty, and thus participate in decision making [22].

Similarly, the trend towards integrated environmental modelling has implications for managing uncertainty. Baustert et al. [16] note that such modelling can create additional sources or types of uncertainty that relate to coupling more than one model. They also highlight the lack of guidance in dealing with the heterogeneous uncertainties that might arise from the various models being integrated.

Another characteristic with implications for managing uncertainty is that collaborative decision-making processes are often constrained in either time or resources, or both, due to the need to follow prescribed processes and timelines. Deadlines can be tight and inflexible for legal, political, or other externally driven reasons and may require modellers to make pragmatic choices, e.g., to use readily available data and algorithms despite their known limitations. Requiring modellers to undertake an additional uncertainty analysis yet still meet deadlines can be very challenging.

There are numerous existing frameworks in the literature to assist with modelling and/or managing uncertainty. Voinov et al.’s [18] description of the components of the participatory modelling process is useful, although not focused on the identification and management of uncertainty. The proposed framework and guidance for uncertainty in the environmental modelling process by Refsgaard et al. [17] is helpful, but the additional complexity of participatory modelling and participatory decision making increases the types and sources of uncertainty that are relevant. The NUSAP system proposed by Funtowicz and Ravetz [4] captures both quantitative and qualitative dimensions of uncertainty, and aims to provide an analysis and diagnosis of uncertainty in the knowledge base of complex policy problems. However, it does not seem to deal with integrating multiple models, assessments, or indicators. Updated guidance on uncertainty written by the Group of Chief Scientific Advisors [23] is very comprehensive and describes several approaches for representing and communicating uncertainty, including NUSAP and the Netherlands Environmental Assessment Agency (PBL) guidance [24].

Baustert et al. [16] reviewed 15 uncertainty frameworks and commented on the sources of uncertainty that each one covers. They highlighted four challenges: inconsistent terminology, characterisation of multiple sources of uncertainty, treatment of uncertainty, and communication of uncertainty. Ascough et al. [25] and Skinner et al. [26] also reviewed the various uncertainty typologies in the literature, and went on to derive classifications distinguishing uncertainty in knowledge (data, model, process understanding), decision making, human, language, and natural variability. Brugnach et al. [27] explain how, in addition to deficiencies in information, modelling-related uncertainty arises from the way the information is interpreted and framed. They list the five causes of uncertainty most relevant to modelling as error in empirical observations, complex dynamics, ambiguity and conflicting knowledge, ignorance, and values and beliefs. They note how each of these causes of uncertainty can affect the data and parameter values, model structure, and modelling process (framing). Another classification refers to known knowns, unknown knowns, known unknowns and unknown unknowns. These various classifications are very useful in highlighting the wide range of sources of uncertainty that should be considered, and provide guidance for how these might be explored.

Voinov et al. [18] note that more research is needed into pragmatic hybrid approaches that allow resource managers and scientists to integrate uncertainty analysis throughout the whole environmental policy-making process, even in a time- and resource-constrained situation. The contribution of this paper responds to this call for pragmatic approaches. We have developed a framework that guides a set of practical steps to understand, communicate, and manage uncertainty more explicitly and transparently in time- and resource-constrained collaborative and integrative environmental policy-making processes. This paper documents our use of a past real-world case study to develop and test this framework and its associated steps and tools.

2. Case Study Background

Between 2011 and 2014, Environment Canterbury (the regional authority responsible for environmental management in the Canterbury region of New Zealand’s South Island) ran a collaborative and community-centred policy process (Figure 1) to set water quality and quantity limits in a regional plan for the Selwyn Waihora catchment. The requirement to do this was prescribed in the non-statutory Canterbury Water Management Strategy [28] and the National Policy Statement for Freshwater [29].

In this collaborative process, the Selwyn Waihora Zone Committee, made up of representatives of the community, the indigenous Māori community, and government, was responsible for recommending outcomes and water quality and quantity limits to Environment Canterbury. The limits specifically considered impacts on a range of environmental, economic, social, and cultural values. These outcomes and limits were subsequently finalised as statutory and enforceable through a Resource Management Act (RMA) regional plan hearing process. (The RMA is the overarching legal framework for environmental management in New Zealand (https://www.mfe.govt.nz/rma/ (accessed on 12 May 2022).) An interdisciplinary technical team was established to support the Zone Committee and Environment Canterbury in their task.

The Zone Committee described their social, cultural, environmental, and economic aspirations for the catchment in the form of desired first- and second-order (primary and secondary) outcomes (phase 1, Figure 1) [30]. The technical team developed indicators based on these outcomes, against which attainment of outcomes could be assessed under different management options or scenarios. Due to time and resource constraints, a decision was made to model the indicators by connecting mostly pre-existing models into an assemblage. A series of exploratory and anticipatory scenarios was developed by the Zone Committee, and the technical team developed models to predict the catchment state (phase 2, Figure 1) under each of the scenarios, and used the indicator predictions to inform the likelihood of meeting the community outcomes (phase 3, Figure 1). This technical information was integrated, translated, and communicated back to the Zone Committee and wider community to help inform their discussions (phase 4, Figure 1), and to reach a negotiated agreement on limits for the catchment (phases 5–6, Figure 1). See Robson [31] for further background information on this process.

The main communication tool used to describe and convey uncertainty was an expression of likelihood of the desired outcomes being met (at first- and second-order levels) for different scenarios [31]. This was presented as a matrix with five categories of likelihood, ranging from highly unlikely to almost certainly (Figure 2). The underlying assessment of uncertainty included data and model uncertainty (based on peer review and expert knowledge), model integration (such as consistency of modelling assumptions and compatibility of model outputs for use in other models), and a general consideration of uncertainty (lack of data or knowledge). Sense testing with science stakeholders and decision stakeholders was used at multiple stages.

3. Methods

All of the authors of this paper were in the technical team that supported and informed the Zone Committee in their determination of environmental limits. The high levels of uncertainty in the technical information were a challenge for the collaborative process. Thus, in 2016, the authors began to review the methods and tools that had been used to manage and communicate uncertainty in the supporting technical process. We then embarked on a formal review of the strengths and weaknesses of the approaches used to account for uncertainty, assessed using the project evaluation questions from the Integration and Integration Science framework [3] (p. 99). This framework provides a systematic way to describe and evaluate case-based research on complex real-world problems and comprises three domains and five questions, where the second domain covers understanding and managing diverse unknowns. A summary of the review process is shown in the Section 4, Part 1.

In response to this review, we developed a five-stage approach to understanding, communicating, and managing uncertainty that would be feasible within a time- and resource-constrained process (Section 4, Part 2). This drew upon the sources of uncertainty identified by Brugnach et al. [27], Skinner et al. [26] and Walker et al. [32].

We tested and refined the detail of this approach by applying it to increasingly complex examples within the case study, using the same data and modelling as was used in the original case study. In each iterative example, where we found a step or tool within the approach that did not work or was too complicated or time-consuming using real data, the approach detail was refined. Some aspects of the third and final experiment are presented in this paper (Section 4, Part 3).

Note that the scope of this paper is the methods for managing uncertainty within a collaborative policy-making process.

4. Results

4.1. Part 1: Critical Review of How Uncertainty Was Managed

As a result of the critical review using the domain-two evaluation questions from Bammer [3], we identified a number of aspects where the management of uncertainty could have been improved.

• Two sources of uncertainty (model and data) were adequately dealt with on an individual model or assessment basis, but other sources of uncertainty (such as systems processes, human, language, extrapolation, and decision [26,31]) were not specifically identified or transparently managed.
• There was no systematic approach to prioritising where to focus efforts on managing uncertainty [33], for example, by identifying the most important and uncertain factors [5].
• The decision to use Zone Committee first- and second-order outcomes to scope the technical work both increased the number of relevant unknowns and created uncertainties [34,35]. For example, some community outcomes went beyond the available indicators, e.g., [15], and uncertainties arose from the language used in the community outcomes [26,27].
• There were important uncertainties that were legitimately ‘banished’ [3] for the duration of the project that may be able to be reduced for a subsequent policy-making process, but the further investigation needed in these areas was not adequately discussed.
• The likelihood matrix was successful in helping the stakeholders to appreciate that assessments were uncertain, but it lacked transparency in what uncertainties were considered. Descriptions of some data and model uncertainty were included in the documentation of the modelling components, but it is not clear to what extent these and uncertainties from other sources [26,32] were taken into account in determining the likelihood of achieving first- and second-order outcomes in the matrices.
• The aggregation of indicators of the likelihood of achieving first- and second-order outcomes was largely implicit [31] (p. 21); for example, the technical experts implicitly weighted the relative importance of individual indicators in contributing to an outcome. This weighting was tested within the technical team, but was not open to wider scrutiny or an uncertainty assessment. This implicit weighting of variables may be vulnerable to issues with consistency and differences in values [27].

We considered the opportunities raised by the critical review in the context of likely time and resource constraints, and identified three areas where future similar processes could be improved:

• Increase the clarity of what uncertainty is included, at what level, and its nature;
• Expand the sources of uncertainty considered;
• Apply a systemic and systematic approach to identifying and prioritising uncertainties.

4.2. Part 2: Conceptual Framework

In response to the identified opportunities above, we devised a conceptual framework to understand, communicate, and manage uncertainty in collaborative policy-making processes. This framework reflects the key aspects of the case study process (Figure 1). These include the definition of the indicators (which are proxies for the range of community outcomes), and the use of models or assessments by a technical team to estimate the state of the set of indicators under different management options/scenarios. The likelihood information was then used by the collaborative group to inform policy and plan development.

The five-stage-conceptual framework is shown in Figure 3 and described below.

• Consider and account for the adequacy of the indicators chosen to describe the desired outcomes.
• Use the best available knowledge and expert judgements to determine data and model uncertainty that underlies the measurements or model estimates of state.
• Convert the information from stages 1 and 2 into likelihoods of achieving an outcome under the various management options/scenarios of interest.
• Undertake a ‘wind tunnel’ analysis, where external contextual uncertainties with respect to the priority outcomes being met are considered.
• Determine appropriate strategies to manage the uncertainties identified in steps 1–4 to support policy implementation and in preparation for informing the next policy-making cycle.

The five-stage conceptual framework structure and identified opportunities guided our exploration of the specific steps and tools for improving the understanding, management, and communication of uncertainty. Additional sources of uncertainty, beyond those of data and modelling, were included. Of particular interest were the uncertainties associated with the collaborative and participatory policy-making process before modelling, and during and after decision making. We also undertook a more complete and transparent exploration and prioritisation of uncertainties in the data and modelling phase.

The steps (and tools) within each stage follow, with examples provided in Results Part 3.

4.2.1. Stage 1—Indicator Adequacy

User-Generated Desired Outcome Statements

The first phase of the collaborative policy-making process is to establish the desired community outcomes including economic, environmental, and cultural outcomes (phase 1, Figure 1). These outcome statements are often high-level, so the start of the process for managing uncertainty is to ask the Zone Committee to describe, in narrative or quantitative terms, what the catchment looks like if each outcome is realised. It can be useful to break high-level outcomes into secondary outcomes. The exercise and the resultant descriptions help reduce the human uncertainty [26] of members of a group having different views on what the outcomes mean. It also helps manage the variability in these views over the course of the policy-making process.

Concept Maps and Indicators

The technical team assesses these outcome statements and descriptions and, along with other knowledge holders (as necessary), creates concept maps [36], also called influence diagrams, to represent the various factors and dynamic processes that influence each first-order outcome. The concept maps are used to bring together disciplinary expertise and other sources of knowledge [37], such as local, historical, cultural, and anecdotal. The process of generating the concept maps enables the technical team to explore differences in terminology, and in their framing and understanding of the system being represented [27].

From these concept maps, the technical team proposes a suite of indicators to describe the desired outcomes. When the indicators (and secondary outcomes) are shown on the concept map, they can make areas of unknowns more explicit to the Zone Committee. They can also highlight areas where there is no indicator (these should be documented and taken into account in stage 3). The indicator suite is then agreed by both the Zone Committee and the technical team.

Each indicator is then described with respect to the range of potential values that will or will not achieve the associated outcome (yes; no; uncertain). This might be specified by referring to defined absolute values or categories, or relative to a state at specified time. These categories may be able to be quite clearly defined, or they may have some uncertainty (e.g., due to lack of knowledge or value differences between Zone Committee members). Based on these descriptions, each indicator is classified as either categorical or numeric; absolute or relative; and “fuzzy” (imprecise) or not. Figure 4 shows a visual representation of these six types (combinations of categorical or numerical, absolute or relative, and fuzzy or not) that can be developed with the stakeholders. (Note that it was hard to think of examples of absolute/relative indicators, so this was not included).

Many indicators are likely to be fuzzy due to epistemic uncertainty or value-based disagreement about the thresholds of when a state is acceptable. The relative indicators might also be quite vague. For example, the Zone Committee might have expressed an outcome of increased employment, with an indicator being an increase in the number of farmers and farm workers, but without specifying the magnitude of the desired increase. Similarly, an outcome of an improvement in water colour might not go as far as specifying the desired colour. The absolute category is useful for legally specified objectives where there is no ambiguity (e.g., in New Zealand there is a drinking-water standard of 11.3 mg N/L [38] and a categorical suitability for recreation standard [39]).

Table 1. Examples of indicator types.

	Absolute	Fuzzy Absolute	Fuzzy Relative
Categorical	Suitability for contact recreation standard	Satisfaction with ability to collect food	Lake water colour
Numeric	Nitrate concentration in drinking-water zones	Water clarity index	Number of farm workers

gives an example of each of the six indicator types.

Indicator–Outcome Objects

Tables and graphs are drawn up that relate indicator values to whether the desired outcome is met. These are referred to as indicator–outcome objects (IOO). This description of the relationship between the indicator and outcome encapsulates the adequacy of the indicator to represent the desired outcome, as well as the uncertainty in the thresholds at which the indicator is acceptable with respect to the outcome. The IOOs can capture the human uncertainty [26] where different members of the Zone Committee have differing views on what is acceptable by requiring a negotiated decision from the group. The IOO thus serves as a boundary object [40] between the different members of the Zone Committee and the technical team, thereby reducing language ambiguity uncertainty [41].

Proportional Weightings

Based on these concept maps and the chosen indicators, a proportional weighting is agreed that reflects the relative importance of the indicators that make up the second-order outcomes. This step is repeated to determine the proportional weighting of the second-order outcomes in describing the first-order outcomes. The agreed weightings might reflect a range of value judgements for the outcomes [27].

4.2.2. Stage 2—Data and Model Uncertainty

Prioritising Important and Uncertain Factors and Relationships

The outcome concept maps produced in stage 1 are used to systemically and systematically consider the different sources of data and model uncertainty and to undertake a prioritisation exercise. With the same group of technical experts and knowledge holders who contributed to each outcome concept map, nodes (i.e., factors) are first characterised in terms of how important they are in influencing the factor they feed into, vis-à-vis other contributing factors. This assessment of importance is at the systems level, rather than considering the models and data used. Second, the nodes and their connecting relationships are assessed for how uncertain the data and the modelling (or assessment) are, respectively. This step considers the uncertainty of the data quantifying or describing a factor and the uncertainty of the model or relationship between factors. It also considers the sensitivity in the model to the contributing factors. It is possible to see a very uncertain factor feeding into a less uncertain factor, for example, due to a lack of sensitivity to that very uncertain factor, or perhaps because there are other lines of supporting evidence.

The experts should consider a range of uncertainties including the limitations of available data and parameter values, missing data, randomness, the appropriateness of the model structure, inclusion or otherwise of key processes, evidence of predictive success, and the likelihood of extrapolation validity in their assessment of the uncertainty.

Nodes and lines in the concept maps (e.g., Figure 5) are colour-coded by the group into high (magenta), medium (blue), and low (grey) importance and uncertainty, and a rationale is recorded to support the categorisation. In the uncertainty map, node text colour represents data uncertainty and line colour represents uncertainty in the relationship or model. A suggested strategy when time and resources are limited is to focus on recording the rationale for the most important and uncertain (high/high) factors. We elected to identify the nodes for further analysis by assigning 3, 2, or 1 to each of the node’s importance and data and relationship uncertainty rankings (high, moderate, and low, respectively), and summing the values for importance and maximum amount of data and relationship uncertainty. All nodes with a combined value of 4, 5, or 6 were colour-coded yellow, orange, and brown, respectively (Figure 5). The colour-coded diagrams and the rationale assigned by the group form an important part of the documentation of uncertainty, and are a way to prioritise further analysis. Communication between the different disciplines is essential for the group to arrive at a reasonably consistent assessment across the whole system.

Representing Data and Model Uncertainty on IOOs

An indicator value or state might be qualitatively assessed or predicted using a quantitative model, or it might be measured directly if assessing the current or historical state. In all cases, there is a degree of data uncertainty, perhaps due to poor quality measurements or poor data inputs into the model, and a degree of model uncertainty (e.g., due to model structural error or choice of summarising method in the case of data measurements; e.g., mean, 95th percentile). A formal uncertainty analysis of the data and model uncertainty (e.g., by one of the methods described by Matott et al. [42]) is preferable where resources and time permit. The simpler approach described here is designed to identify and characterise uncertainty in a relatively short time using expert knowledge and taking a top-down approach that focuses on the more important sources of uncertainty.

The measured or estimated state for each indicator, as determined by the technical team, is marked on the relevant IOO (with an asterisk). The next step is to draw the possible distribution of each state, given the key data and model uncertainties described in the relevant concept map, onto the IOO (Figure 6). Where a model has been used to assess or estimate an indicator value (state), the group seeks to determine the distribution of possible values as best they can. The objective is to quantify the likelihood or probability of meeting, being uncertain, or not meeting the relevant outcome (i.e., the red, orange, and green categories on the IOO). Options for this include a Monte Carlo analysis, a Bayesian approach, or a simpler simulation or interval arithmetic exercise. In the latter, mutually plausible and consistent data input and expert estimates of model errors for the most important and uncertain factors (i.e., the high/high factors from the previous step) are used to explore the possible range or distribution in model output values. Other converging or diverging lines of evidence can also be considered. The resulting distribution is plotted on the IOO either as a probability density function or as class probabilities. Documenting the assumptions and rationale is essential to make this assessment transparent and to allow the analysis to be updated as more information becomes available.

4.2.3. Stage 3—Outcome Likelihood

Determining Scenario Likelihood of Meeting First- and Second-Order Outcomes

This stage brings together the results from stages 1 (inadequacy of indicator) and 2 (data and model uncertainty). We have adopted the likelihood categories (

Table 2. Likelihood classes.

Narrative Description	Probability Class	Description	Colour Code
Very likely	90–100%	Likely to occur even in extreme conditions
Likely	67–90%	Expected to occur in normal conditions
About as likely as not	33–67%	About an equal chance of occurring as not
Unlikely	10–33%	Not expected to occur in normal conditions
Very unlikely	0–10%	Not likely to occur even in extreme conditions

) suggested by MfE [33]. These are a simplification of the Intergovernmental Panel on Climate Change (IPCC) scale of likelihood. Each indicator is reviewed by the technical team as to the distribution of meeting an outcome or not and converted into a likelihood estimate, as described in

Table 3. The key for characterising the likelihood of whether an outcome is met (N = probability outcome is not met and Y = probability outcome is met).

Very likely	If Y ≥ 0.9
Very unlikely	If N ≥ 0.9
Likely	If Y ≥ 0.67 and N < 0.33
Unlikely	If N ≥ 0.67 and Y < 0.33
About as likely as not	the rest

.

Translating Figure 6a into likelihood according to Table 3 results in a likelihood assessment of ‘Likely’. Figure 6b translates into ‘About as likely as not’.

Many of the first- and second-order outcomes are described by multiple indicators. These are combined based on the proportional weightings developed in stage 1. The end result is a likelihood matrix that represents all of the indicators for each first- and second-order outcome, and the likelihood of the outcome being met under the different scenarios that were part of the policy-making process. The matrix format facilitates understanding of the trade-offs between different, often conflicting outcomes.

4.2.4. Stage 4—Wind Tunnel Assessments

The first three stages are undertaken for all scenarios considered by the Zone Committee. Once the Zone Committee is close to agreeing on their proposed solution as the basis for new policy or a plan, the underpinning assumptions are identified by the Zone Committee and stress tested. This is to determine their robustness for delivering the outcomes across a range of external context factors that were not considered in the scenario testing phase (e.g., climate change or a significant change in economic factors). This is analogous to verifying a near-final engineering solution in a wind tunnel to test whether the design will hold up in a range of conditions [5]. A set of wind tunnel mini-scenarios are identified. For each wind tunnel condition, the technical group should review the concept maps and IOOs to judge whether there are any changes in the highly important and uncertain factors, or estimated state, and thus whether there should be any revision of the outcome likelihood classes in the matrix. Any changes are documented and shared with the Zone Committee, who decide whether any changes need to be made to the policy and plan direction.

4.2.5. Stage 5—Uncertainty Strategies

The uncertainty analysis captured in stages 1–4 will help the collaborative decision-making group in the policy-making process. Stage 5 identifies uncertainties that were excluded in stages 1–4, plus any key uncertainties in underpinning assumptions that could be reducible in the implementation phase of the plan. Once identified, methods to reduce the uncertainties are proposed. For example, a phased implementation timeline might permit the measurement of important unknowns and trigger changes to the rules.

4.3. Results Part 3: Case Study Test

In the case study, seven relevant first-order outcomes and 19 second-order outcomes were articulated by the Zone Committee, and an initial series of five management scenarios were defined and explored. These ranged from environmentally conservative through to significant land-use intensification, and included various types of mitigations and restoration initiatives. Subsequently, a final ‘solutions package’ was arrived at after the Zone Committee had deliberated on the learnings from the five initial scenarios. This package aimed to achieve a sustainable balance between the outcomes. Our five-stage framework was tested on three first-order outcomes of increasing complexity. The final and most complex outcome tested was ‘(Lake) Te Waihora is a healthy ecosystem’. This first-order outcome is made up of six second-order outcomes (

Table 4. List of second-order outcomes and associated indicators under the first-order outcome of ‘Te Waihora is a healthy ecosystem’. A weighting of 2 reflects indicators that were agreed to be twice as important, 1 equally important, and 0.5 half as important.

Second-Order Outcomes	Proportional Weightings to First-Order Outcome	Indicators	Proportional Weightings to Second- Order Outcome
There are healthy and extensive macrophyte beds (to 1960 extent)	1	% cover macrophyte beds	1
Food gathering on and around the lake is improved	1	Customary fish stocks	1
		Commercial fish stocks (risk of commercial quota not being supported)	1
		Trophic Lake Index (TLI)	1
		% cover macrophyte beds	1
		Cyanobacterial and/or other toxic blooms	1
		Other contaminants	n/a
		Satisfaction with ability to gather food	1
Fish populations and diversity have increased	0.5	Lake opening/closing regime; impact on fish passage and recruitment. At least 1 opening in spring and at least 1 opening in autumn.	2
		Trophic Lake Index (TLI)	1
		Cyanobacterial and/or other toxic blooms	1
		Fishing pressure (take)	1
Ecological health is acceptable	2	Trophic Lake Index (TLI)	1
		Lake nitrate-N concentrations	1
		Cyanobacterial and or other toxic blooms	1
		Biodiversity – loss of species expected to be there	1
		Biodiversity – loss of diverse habitat	1
		Other contaminants	n/a
Recreational opportunities are improved	1	Cyanobacterial and/or other toxic blooms	1
		Water safe for contact recreation	1
		Recreational fish stocks trout, eel, flounder and whitebait	1
		Water colour in Te Waihora	0.5
		Water clarity in Te Waihora – mid-lake	0.5
		Bird populations and diversity	n/a
Visual appearance is improved	0.5	Water colour in Te Waihora	1
		Water clarity in Te Waihora – mid-lake	0.5
		Water clarity in Te Waihora – lake edges	1

). For the purposes of this experiment, the wording of the original primary and second-order outcomes [31] (p. 4) was modified slightly while retaining the original intent (Table 4). The original narrative descriptions [31] (pp. 53, 55, 57, 60) were used. All of the scenarios and second-order outcomes from the original study were tested. For brevity, only the results for one of the second-order outcomes, ‘There are healthy and extensive macrophyte beds (to 1960 extent)’ under the ‘current’ state and final ‘Solutions Package’ scenarios, are presented below.

4.3.1. Stage 1—Indicator Adequacy

Concept Maps and Indicators

The full suite of indicators for the first-order outcome and the six second-order outcomes and their proportional weightings are shown in Table 4. Figure 7 shows a simplified version of the full concept map.

Indicator–Outcome Objects

IOOs were created for each of the indicators in Table 4. These would normally be developed with the Zone Committee, but in this study were developed through the analysis of the original case study data and discussion between the authors. Some indicators were used in more than one second-order outcome, but with different IOOs. The second-order outcome of macrophyte beds (presented here as an example) has just one indicator, which is a fuzzy absolute numeric. Figure 8 is the IOO for this indicator. It indicates the numerical extent of macrophyte beds that is considered to achieve the desired outcome, that which will not achieve it, and an in-between zone where it is not agreed that the extent will either achieve the outcome or not.

4.3.2. Stage 2—Data and Model Uncertainty

Prioritising Important and Uncertain Factors and Relationships

Based on the concept map, the authors assessed the relative importance of the incoming nodes to each node, starting with the central node representing the first-order outcome of a healthy lake Te Waihora. Similarly, an assessment of the uncertainty of each node (data uncertainty) and each relationship (line) between nodes (model uncertainty) was made.

Figure 9 shows the part of the concept map relevant to the example second-order outcome. It visually represents the very important and uncertain factors/relationships in magenta, the moderately important/uncertain factors/relationships in blue, and the unimportant and certain factors/relationships in grey. The rationale behind each high and moderate classification was recorded in a table (e.g.,

Table 5. Record of the rationale behind the assessments of importance and uncertainty for the example nodes in Figure 9.

Data/State Uncertainty
Lake total phosphorus (total P)		M	Driven by lake inputs and from the lake bed itself during resuspension in windy conditions. There are reasonable measurements of the current lake P load and the models are considered to be reasonably well developed. In the Solution Package the application of alum to reduce lake P levels was assumed, and there is some uncertainty about its efficacy.
Lake total nitrogen (total N)		M	Driven by lake inputs. Models generate estimates of relative changes from the current load for each scenario where the lake inputs are specified. There are reasonable measurements of the current lake N load and the models are considered to be reasonably well developed
Chlorophyll a		M	Driven by total N, total P and lake clarity
Trophic Level Index (TLI)		M	TLI3 is used in this lake, which is derived from total N, total P and Chl-a
Lake opening regime		L	Solutions Package assumption
Re-establishment of macrophyte beds		L	Solutions Package assumption
Wave protection		L	Solutions Package assumption
Relationship/model uncertainty
Total N, total P, Chl-a	TLI	L	Defined calculation
TLI	Macrophyte extent	L	Macrophyte beds are unlikely to flourish in this lake with elevated TLI (Norton et al. 2014)
Opening regime	Macrophyte extent	M	The potential success of the re-establishment of macrophytes is affected by lake level. There is uncertainty about the response of macrophytes to a managed opening regime
Re-establishment of beds	Macrophyte extent	L	Re-introduction of macrophytes in the lakes requires re-establishment programme
Protection	Macrophyte extent	L	Macrophyte bed establishment is known to be hindered by wind and wave action
Importance
Lake total P		M	Equally important in calculating TLI
Lake total N		M	Equally important in calculating TLI
Chlorophyll a		M	Equally important in calculating TLI
TLI		M	Good water quality moderately important to the survival of the macrophytes
Opening regime		H	Macrophyte survival requires avoiding periods of low water level as they dry out.
Re-establishment of beds		H	Critical for achieving the outcome as there are no macrophytes currently
Wave protection		M	Moderately important to prevent breakup of macrophyte bed during establishment

). Note that the assessment of important factors relates specifically to this lake in this context (scenario), and the assessment of uncertainty additionally relates to the specific models and data used to produce the indicator estimates during the case study process.

Representing Data and Model Uncertainty on IOOs

The indicator values (for the Solutions Package scenario, as originally determined by the case study technical team) were recorded onto the relevant IOO (with an asterisk). We then considered the most important and uncertain factors (Figure 9c) in order to determine the possible distribution of each indicator value based on the data and model uncertainties.

The simple assessment of the uncertainty of the total N and P loads to the lake is given in Appendix A as an example. From the estimated ranges of uncertainty of total N (55% less up to 102% more) and total P (50% less and up to 30% more), we can calculate a best and worst case for TLI.

• Worst case: +102% N and +30% P gives a TLI of approximately 7.5.
• Best case: −55% N and −50% P gives a TLI 5.7 of approximately 5.7.

This expert-estimated distribution is then reflected in the relevant IOOs, and, in the case of a numeric indicator, converted into probabilities of each class of outcome achieved (yes/no/uncertain). With respect to the relationship between TLI and macrophyte beds, we note that, historically, there have been extensive macrophyte beds, mostly confined to the lake margins and embayments of Lake Waihora. Currently, macrophyte beds are virtually absent. Macrophyte beds were periodically abundant from 1900 to the 1960s, but have not returned in significant cover since a large storm in 1968. Some small areas of natural re-establishment have occurred on occasions (1986, 2011), but these have not persisted due to lake levels, salt water intrusion, and water quality. A potential improvement of TLI 6 is expected to create more favourable conditions for macrophytes, but the beds would probably remain absent unless further interventions, such as lake-level control and a re-establishment programme, are employed. These additional measures were assumed to occur in the Solutions Package scenario, but there is still uncertainty about their combined efficacy. Therefore, the macrophyte bed extent second-order outcome was assessed by experts as 70% (yes), 20% (uncertain), and 10% (no), as shown in Figure 9.

4.3.3. Stage 3—Outcome Likelihood

Determining Likelihood

In stage 3, the indicators were combined by taking a weighted average of the probabilities of each class of outcome achieved (yes/no/uncertain) and converting them to a likelihood category, as per the rules in Table 3. The resulting outcome likelihood matrix for the current and the Solutions Package scenarios is presented in Figure 10. This allows the Zone Committee to quickly see which outcomes might not be achieved given the various uncertainties that were considered. Sustainable limits could then be set based on the scenarios with an acceptable balance and certainty of achieved outcomes.

4.3.4. Stage 4—Wind Tunnel Assessments

In stage 4, several wind tunnel scenarios were tested. One of the scenarios examined the assumptions underpinning several important aspects of the agreed Solutions Package. These aspects were the availability of investment funds for the various assumed mitigations and restoration activities (e.g., installing a level control weir, developing a wave barrier system, and undertaking a macrophyte reinstatement programme) and cultural acceptance of some of the assumed interventions in the lake (e.g., the use of alum to reduce internal phosphorus loading). Expert opinion was used to infer what would happen if investment funds were not made available for these mitigations. The concept diagrams were updated and the results of considering the most important and uncertain factors are added to the relevant IOOs.

Using the same example of TLI as in stage 2, the wind tunnel test makes no difference to the estimations of N losses, so the range remains at −55% to +102%. For P, in the original Solutions Package scenario, the combination of stream and on-farm mitigations and alum application was conservatively assumed to reduce P loads in the lake by half. The best case is a 0% change to the estimated P load, where the efficacy of the stream and on-farm mitigations might make up for the loss of the alum. The worst case is an increase of 160%, assuming the stream, on-farm, and alum interventions are ineffective (so the reduction of 50% does not happen), plus the uncertainty documented in stage 2 about land management practices. This would change the potential range of TLI from approximately 7.2 in the best case to above TLI 7.5 in the worst case. In both the best and the worst case, the unfavourable TLI and the lack of critical interventions would mean it is highly unlikely that the macrophyte beds will re-establish. The results of this wind tunnel analysis are added to the IOO and the likelihood matrix (

Table 6. The likelihood matrix for the healthy lake outcome under the current and Solutions Package scenarios (stage 3) and one of several wind tunnel analyses from stage 4. Colour coding is described in Figure 2.

First-Order Outcomes	Scenarios
	Current	Solutions Package	Wind Tunnel (No Investment)
There are healthy and extensive macrophyte beds (to 1960 extent)
Fish populations and food gathering on and around the lake is improved
Ecological health is acceptable (ie meets bottom lines)
Recreation opportunities are improved
Water clarity is improved

). This explicitly shows that the investment funds are very important to achieving the desired outcomes. This type of analysis highlights key external uncertainties that would need to be managed or mitigated to increase the likelihood of the policy delivering the desired outcomes.

4.3.5. Stage 5—Uncertainty Strategies

Stage 5 identifies the uncertainties underpinning key assumptions, along with uncertainties that were excluded in stages 1–4, and considers which of these are feasible to reduce in the implementation phase of the plan. For the example given here, key assumptions in the modelling that would benefit from additional research might include research to calibrate nutrient loss modelling, groundwater investigation to increase knowledge of attenuation losses, feasibility studies on re-establishing macrophyte beds, and constructing a lake-level control weir. Other uncertainties that were excluded from consideration in stages 1–4 include the impact on the marine environment and explicit modelling of climate change scenarios. If these excluded uncertainties are judged important for informing the next iteration of the policy, this early identification could provide an opportunity to conduct the relevant investigations.

5. Discussion

5.1. Improving Process of Managing Uncertainty in Time and Resource-Constrained Policy-Processes

The Selwyn Waihora process was the first in a series of collaborative processes by Environment Canterbury to set sustainable environmental limits [31]. A critical reflection of how uncertainty was managed in the Selwyn Waihora process revealed three opportunities to better identify and manage uncertainty in similarly time- and resource-constrained policy-making processes.

The first was to provide more transparency in terms of what uncertainty has been considered. The stage 1–2 colour-coded concept maps (importance and uncertainty) and associated rationale tables were critical in documenting and maintaining a record of the considered uncertainties. The step of creating the IOOs helped draw out and clarify the scientific assumptions as well as the tacit knowledge of the decision-making group as to whether the indicators fully captured the desired outcomes. The likelihood matrices had already proved to be an excellent tool for communicating uncertainty. Our stage 3 steps and tools made the derivation of this matrix, including the aggregation of the indicators, much more transparent and objective.

The second opportunity was to expand the sources of uncertainty considered. The original process considered data and model uncertainty (now captured in the concept maps and IOOs). New sources (as a result of this study) included the uncertainty due incomplete coverage or mismatches between the narrative outcome and the indicators (stage 1), and uncertainty due to external factors via the wind tunnels (stage 4). The latter allowed political and technical issues behind the scenario assumptions to be explored. Furthermore, it was useful to separate out the likelihood of a scenario meeting the outcomes from the underpinning assumptions of the scenario and whether it would happen (e.g., if a planned investment was made). The concept map assists by allowing the framing and boundaries of the problem to be examined, along with ignorance and ambiguity in the difference sources of knowledge. Stage 5 is not strictly a new source of uncertainty, but it was important in the overall limit-setting process to consider how the identified uncertainties might be managed and reduced (e.g., through monitoring critical assumptions). In this exercise, we drew upon various uncertainty classifications to extend the uncertainty sources, i.e., Ascough et al. [25], Skinner et al. [26], Brugnach et al. [27], and Walker et al. [32]. It might be useful to more explicitly work with one classification in any future exercise to help a decision-making group better understand the different sources.

The third opportunity was having a systematic approach to identifying and prioritising the uncertainties. Use of the concept maps (stage 1) enabled a more comprehensive analysis that synthesised multiple knowledges into a single coherent understanding. This facilitated identification of gaps in this understanding. The framework enabled a consistent synthesis, where experts from different fields had to assess and compare the magnitude of uncertainties across the entire domain (stage 2). Effort was then focused on the uncertainties deemed to be the most relevant. The approach could also help to identify any potential conceptual flaws in the scenarios before the modelling effort.

We note that our quantitative assessments of uncertainty in this case study (stage 2) were informal and not very robust. However, Hamel and Bryant [43] suggest that a simple analysis of the ranges or bounds of many uncertainties can provide a solution in the face of significant uncertainty and resource constraints. One way to improve the assessments might have been to adopt an expert elicitation approach [43], although the number of available experts is very small. In an associated project, Etheridge et al. [44] used the Sheffield Elicitation Framework to derive uncertainty estimates of leached nitrate under different land uses. Other, more quantitative approaches, such as a Monte Carlo simulation exercise, would be prohibitive in terms of the required resources.

5.2. Supporting Participation, Collaboration, and Integration

More participatory and collaborative policy making has signalled a move away from linear, science-led approaches towards the greater inclusion of other knowledge sources and knowledge systems [7,12,45]. An increasing interest in integrated modelling to reflect a more holistic approach to understanding environmental challenges also signals the inclusion of more fields of knowledge [16]. These are important trends in sustainable NRM, so it is interesting to reflect on whether our framework is likely to support the inclusion of other sources of knowledge and a greater understanding of interconnections.

The proposed uncertainty framework is based on the steps of the collaborative policy-making process described in the case study, and therefore will reflect any inherent benefits (e.g., inclusivity) or problems (e.g., lack of representation) present in that policy-making process. However, there are several ways that the uncertainty framework additionally supports the inclusion of other sources of knowledge. To illustrate this, we consider Brandt et al.’s [8] three types of knowledge necessary for tackling sustainability challenges: system knowledge, target knowledge, and transformation knowledge. These cover what we know about the problem, what is desirable and plausible to achieve, and how can that change be achieved, respectively.

Our uncertainty framework supports the inclusion of diverse sources in each of these types of knowledge. The uncertainty framework is values-led (target knowledge, [8]). The knowledge of both the technical and decision-making groups are required when developing the IOOs (i.e., boundary objects [40] for elucidating the relationship between values and indicators, stage 1). The development of the concept maps in stage 1 [37] are an example of developing systems knowledge [8]. The maps bring together disciplinary expertise and other sources of knowledge. Although they are led by the technical team, they are the product of collaboration and include diverse knowledge such as local, historical, cultural, and anecdotal. The use of the concept maps (stage 1) and the likelihood matrix (stage 3) to indicate areas of uncertainty is also anticipated to help manage the power relationships and expectations between the technical and decision-making groups, by clearly showing the limits to knowledge and expertise, and by demonstrating that the burden of understanding and responding to uncertainty must be shared between the technical and decision-making groups [35]. Transformation knowledge [8] is also included in the uncertainty framework through the stage 4 wind tunnel assessments, where both the technical and decision-making groups identify critical underpinning assumptions of the proposed solutions and external factors that could impinge on these assumptions and test to see whether the proposed solution is robust.

The proposed uncertainty framework is likely to support the increased understanding of interconnections through the development and use of the systems-based concept maps (stage 1). These maps describe the various factors and dynamic processes that affect the community values and the integrative nature of these values. By their nature, they require dialogue between different disciplines to understand the interconnections.

5.3. Supporting Decision Making in Time- and Resource-Constrained Policy Processes

The complexity of environmental sustainability problems and the importance of considering uncertainty for good policy and decision making [2,5] serve as an imperative for the better understanding, management, and communication of uncertainty. There are many existing frameworks for managing uncertainty (e.g., [4,17,18]), and many more robust approaches are available for quantifying uncertainty (e.g., Monte Carlo analyses). However, our proposed framework responds to the need for simple and pragmatic approaches that allow resource managers and scientists to integrate uncertainty analysis throughout the whole environmental policy-making process in the face of significant uncertainty and resource constraints. Our proposed five-stage framework (and the associated steps and tools) provides a forum for a constructive discussion between all parties on the various unknowns, their significance, and how they might be managed, and in doing so, it also supports participatory and collaborative policy making and integrative environmental modelling. We estimate that our approach to including uncertainty, as used in this case study, would have required an additional 200 h of effort across the technical team. At approximately 5% of the total technical effort, we consider this to be a manageable level of effort in the resource-constrained context of the collaborative limit-setting processes being undertaken in New Zealand.

6. Conclusions

In summary, our framework is a new pragmatic hybrid approach to identifying, assessing, and communicating uncertainty in an environmental policy-making process, where the objective is to set sustainable limits for agriculture that achieve acceptable social, environmental, economic, and cultural outcomes. While not as robust as a fully-fledged uncertainty analysis, our rather less-intensive approach and five-stage framework did support the identification, analysis, and documentation of key uncertainties. A key strength was making the identification and consideration of the uncertainties explicit, systematic, and transparent. Our contribution, developed for situations where resources are constrained, helps to identify and focus effort on the main sources of uncertainty. The proposed five-stage framework (and the associated steps and tools) provides a forum for a constructive discussion between parties on uncertainties, their significance, and how they might be managed. In doing so, it also supports participatory and collaborative policy making and integrative environmental modelling by providing multiple opportunities to include diverse knowledge sources, as well as building an understanding of interconnections across disciplinary fields. Our approach is designed to be conducted by those already involved in the technical team and decision-making group, and only requires a limited additional increase in effort (approx. 5%). As such, we expect that it can be used in low or limited capacity settings.