- freely available
Sustainability 2013, 5(3), 955-979; doi:10.3390/su5030955
Published: 4 March 2013
Abstract: There is increasing interest in long-term plans that can adapt to changing situations under conditions of deep uncertainty. We argue that a sustainable plan should not only achieve economic, environmental, and social objectives, but should be robust and able to be adapted over time to (unforeseen) future conditions. Large numbers of papers dealing with robustness and adaptive plans have begun to appear, but the literature is fragmented. The papers appear in disparate journals, and deal with a wide variety of policy domains. This paper (1) describes and compares a family of related conceptual approaches to designing a sustainable plan, and (2) describes several computational tools supporting these approaches. The conceptual approaches all have their roots in an approach to long-term planning called Assumption-Based Planning. Guiding principles for the design of a sustainable adaptive plan are: explore a wide variety of relevant uncertainties, connect short-term targets to long-term goals over time, commit to short-term actions while keeping options open, and continuously monitor the world and take actions if necessary. A key computational tool across the conceptual approaches is a fast, simple (policy analysis) model that is used to make large numbers of runs, in order to explore the full range of uncertainties and to identify situations in which the plan would fail.
“It is not the strongest of the species that survive, nor the most intelligent, but the ones most responsive to change.”
Translated into long-term planning terms, this well-known quote of Darwin suggests that sustainable (strong) plans should be adaptive plans to survive changes. (In this paper, we use the terms “plans” and “policies” interchangeably. Plans are usually made in the private sector, while policies apply mainly to the public sector.) The World Commission on Environment and Development (1987), in a report often referred to as the “Brundtland report” after its main author, introduced a well-known definition of sustainable development: “development that meets the needs of the present without compromising the ability of future generations to meet their own needs” . In practice, this definition of sustainability has often been summarized as meeting economic, environmental, and social objectives now and in the future. Given the uncertain changing conditions many decisionmakers are facing nowadays, a sustainable plan is not only one that is able to achieve objectives related to society, economy, and environment, but a sustainable plan should also be robust, meaning that it performs satisfactorily under a wide variety of futures, and adaptive, meaning that it can be adapted to changing (unforeseen) future conditions . The question of how to design such plans is rarely addressed in the sustainability literature.
A major challenge in designing sustainable plans is the requirement to accept, understand, and manage uncertainty, since:
not all uncertainties about the future can be eliminated;
ignoring uncertainty could mean that we limit our ability to take corrective action in the future and end up in situations that could have been avoided; and
ignoring uncertainty can result in missed chances and opportunities, and lead to unsustainable plans.
Most of the traditional applied scientific work in the engineering, social, and natural sciences has been built on the supposition that the uncertainties result from a lack of information, which “has led to an emphasis on uncertainty reduction through ever-increasing information seeking and processing” , or from random variation, which has concentrated efforts on stochastic processes and statistical analysis. However, most of the important strategic planning problems currently faced by decisionmakers are characterized by uncertainties about the future that cannot be reduced by gathering more information and are not statistical in nature . The uncertainties are unknowable at the present time, but will be reduced over time. They can involve uncertainties about all aspects of a long-term strategic planning problem—external developments, the appropriate (future) system model, and the valuation of the model outcomes by (future) stakeholders. Such situations have been characterized as having “deep uncertainty”—defined as “the condition in which analysts do not know or the parties to a decision cannot agree upon (1) the appropriate models to describe interactions among a system’s variables, (2) the probability distributions to represent uncertainty about key parameters in the models, and/or (3) how to value the desirability of alternative outcomes” [4,5]. This implies that one can (incompletely) enumerate multiple possibilities for the system model, the probability distributions, and sets of values, without being able or willing to rank order the possibilities in terms of how likely or plausible they are judged to be .
Although policy analysts and strategic planners are aware that they are facing deep uncertainty, most of them still develop plans based on the assumption that the future can be predicted. They develop a static “optimal” plan using a single “most likely” future, often based on the extrapolation of trends, or a static “robust” plan that will produce acceptable outcomes in a small number of hypothesized future worlds [5,7]. However, if the future turns out to be different from the hypothesized future(s), the plan is likely to fail. McInerney et al.  liken this to “dancing on the tip of a needle”. Furthermore, the world is continuously changing, so the conditions planners need to deal with are continuously changing. Therefore, plans need to be adapted to meet these changing conditions. But, it is rare that such adaptation has been planned for in advance.
Any single guess about the future is likely to prove wrong. The performance of plans optimized for a most likely future can deteriorate very quickly due to small deviations from the most likely future, let alone in the face of surprise. Even analyzing a well-crafted handful of scenarios will miss most of the future’s richness and provides no systematic means to examine their implications [9,10,11]. This is particularly true for methods based on detailed models. Such models that look far into the future should raise troubling questions about their assumptions and their validity in the minds of both the model builders and the consumers of their output. Yet the root of the problem lies not in the models themselves, but in the way in which they are used. Too often, analysts ask “what will happen?”, thus trapping themselves in a losing game of prediction, instead of the question they really would like to have answered: “Given that one cannot predict, which actions available today are likely to serve best in the future?” Broadly speaking, although there are differences in definitions, and ambiguities in meanings, the literature offers four (overlapping, not mutually exclusive) ways for dealing with deep uncertainty in making sustainable plans :
resistance: plan for the worst possible case or future situation
resilience: whatever happens in the future, make sure that the system can recover quickly
static robustness: aim at reducing vulnerability in the largest possible range of conditions
dynamic robustness (or flexibility): plan to change over time, in case conditions change
The first approach is likely to be very costly and might not produce a plan that works well because of surprises, or what some call “Black Swans” . The second approach accepts short-term pain (negative system performance), but focuses on recovery. The third and fourth approaches do not use models to produce forecasts. Instead of determining the best predictive model and solving for the plan that is optimal (but fragilely dependent on assumptions), in the face of deep uncertainty it may be wiser to seek among the alternatives those actions that are most robust—that achieve a reasonable level of goodness across the myriad models and assumptions consistent with known facts. This is the heart of any robust decision method. A robust plan is defined to be one that yields outcomes that are deemed to be satisfactory according to some selected assessment criteria across a wide range of future plausible states of the world . This is in contrast to an optimal plan that may achieve the best results among all possible plans but carries no guarantee of doing so beyond a narrowly defined set of circumstances. A plan based on the concept of robustness is also closer to the actual reasoning process employed by senior planners and executive decisionmakers. As shown by Lempert and Collins , analytic approaches that seek robust plans are often appropriate when uncertainty is deep and a rich array of options is available to decisionmakers.
This paper deals with static and dynamic robustness to reach a sustainable plan. A plan that can adapt to changing conditions is well suited to situations involving deep uncertainty. An adaptive plan is developed in light of the multiplicity of plausible futures that lie ahead, and is designed to be changed over time as new information becomes available. Thus, changes become part of a larger, recognized process and are not forced to be made repeatedly on an ad-hoc basis. Planners, through monitoring and corrective actions, keep the system headed toward the original goals.
There is a definite appetite for adaptive approaches. However, although the concept of adaptive policies can be traced back to 1927, when John Dewey  proposed that “policies be treated as experiments, with the aim of promoting continual learning and adaptation in response to experience over time”, a recent literature review conducted at the International Institute for Sustainable Development found that the literature relating directly to the topic of adaptive policies is limited , and a typology of approaches has yet to emerge. A useful division of approaches suggested by Burton  is based on whether the adaptation is planned for in advance and is externally initiated or emerges from changes within the system (see, below). That is, the approaches can be divided into:
Planned adaptation, which is the result of deliberate decisions, based on an awareness that conditions might change or have changed and that action is required to return to, maintain, or achieve a desired state.
Autonomous adaptation, which is adaptation that is not a planned external response to a situation, but is an internal system reaction due to changes within the system. (This is sometimes referred to as resilience.)
As shown in Table 1, Burton  has three additional dimensions with which he categorizes (climate) adaptation approaches: (1) whether the actions taken are anticipatory, concurrent, or reactive; (2) whether the temporal scope is short term or long term; and (3) whether the spatial scope is localized or widespread.
This paper deals with approaches to planned adaptation that are both anticipatory and reactive, that are long term, and that can be both localized and widespread. A plan that embodies these ideas allows for its adaptation over time to meet changing circumstances, and can thus be considered a sustainable plan. In Section 2, we present an overview of planned adaptation approaches for designing sustainable plans. In Section 3, we discuss the computational support tools for adaptation planning that have been combined with these planned adaptation approaches. These tools and approaches are often interconnected. Some of these methods could even be categorized both as an approach and as a computational tool. From the two discussions in Section 2 and Section 3, we identify common principles for designing adaptive plans and for providing computational support to the design process, which are presented in Section 4. Section 5 presents our conclusions.
|Adaptation based on:||Type of adaptation|
|In relation to climatic stimulus||e.g., unmanaged natural systems||e.g., public agencies|
|Timing of actions||Reactive||Concurrent||Anticipatory|
|From observed modification||During||Prior modification|
|Temporal scope||Short term||Long term|
|Adjustment, instantaneous, autonomous||Adaptation, cumulative, policy|
2. Approaches for Designing Sustainable Plans
A variety of different approaches for designing robust plans under deep uncertainty have been developed, but few produce dynamic robust plans. Traditional scenario planning assesses the performance of alternative static plans under different hypothetical futures (e.g., [19,20]). Given a static plan, Dewar et al.  used signposts to monitor the need for actions to either shape the future or to reduce the plan’s vulnerability to uncertain future developments. They called this “Assumption-Based Planning” (ABP). This was a first step towards adaptive planning. In contrast to static robust plans, adaptive planning defines contingency plans and specified conditions, called signposts and triggers, under which the plan should be reconsidered and revised . In this section we briefly introduce ABP, and then provide an overview of several adaptive planning approaches that are directly related to the principles underlying ABP—one that produces a static robust plan, and others that produce dynamic robust plans.
2.1. Assumption-Based Planning
Assumption-Based Planning (ABP) [21,23] was developed at the RAND Corporation in the late 1980s as a tool for improving the adaptability and robustness of existing plans, not as a tool for creating plans. It was designed to make a plan more resistant to significant change, and to help an organization to identify when to adapt the plan. ABP was specifically developed in response to shortcomings of the traditional scenario planning approach. The scenario planning approach  identifies a future world with high plausibility and finds a plan that would work well in that world. ABP turns this approach upside down. It begins by assuming that there is a proposed plan, or that there is a plan already in operation. It then tries to protect the plan from failing, by examining each of the underlying assumptions, and seeing what would happen to the plan if that assumption were not to be true.
ABP identifies the assumptions upon which the success of the plan most heavily rests (the “load-bearing” assumptions) and the assumptions that are most vulnerable to being overturned by future events. Assumptions that are both load-bearing and vulnerable are the most likely to produce nasty surprises as the plan unfolds. To deal with potential surprises, ABP produces three things: signposts, shaping actions, and hedging actions. A signpost is an event or threshold that, if detected, signifies that a vulnerable assumption is being broken or is dangerously weak, and that some action should be taken. A shaping action is an action that is intended to help protect an uncertain assumption—to control the future as much as possible. A hedging action prepares for the possibility that an assumption will fail, despite the shaping actions. (Insurance is a classic hedging action.) Figure 1 illustrates the five steps in ABP.
ABP was first used to help the U.S. Army with a long-range planning exercise 30 years in the future . It has subsequently been applied to U.S. Navy, Air Force, and Marines planning, and has been used by planners in at least two militaries outside the United States. It has also been used to improve plans for public enterprises, ranging from a small nonprofit organization to a large water district, and has been used to test plans in higher education and in private businesses .
2.2. Robust Decision Making
Robust Decision Making (RDM) is an approach together with a set of model-based methods and tools that supports decisionmaking under deep uncertainty and is used to produce a static robust plan. Relating RDM to Table 1, its approach to adaptation is planned, anticipatory, long term, and widespread. It has been developed over the last 15 years, primarily by researchers associated with the RAND Corporation. The RDM framework uses multiple views of the future to support a thorough investigation of modeling results that helps to identify a static plan that (1) is robust (i.e., it performs “well enough” across a broad range of plausible futures, but may not perform optimally in any single future), (2) avoids most situations in which the plan would fail to meet its goals, and (3) makes clear the remaining vulnerabilities of the plan (i.e., conditions under which the plan would fail to meet its goals) [5,25].
Scoping—determine the scope of the analysis by identifying exogenous uncertainties, policy options, key relationships, and performance metrics; construct a simulation model that relates actions to consequences.
Simulation—identify a candidate policy to evaluate and run it against an ensemble of scenarios;
Scenario discovery—identify vulnerabilities of the candidate policy (i.e., which combinations of exogenous uncertainties, and in which ranges, cause the policy to fail to meet the goals);
Adaptation—identify hedging actions (modifying existing policies or defining new ones) to address these vulnerabilities. Repeat steps 2 and 3 for additional candidate policies;
Display—Plot expected outcomes of all policies over probabilities of vulnerable scenarios, and choose the most robust plan for implementation.
RDM has been applied to strategic planning problems in a diverse set of fields, including economic policy , climate change [5,29], flood risk management , sea level rise , energy resource development , and water resources management [25,33,34]. RDM is often used in combination with computational support of the Scenario Discovery method (see Section 3).
2.3. Adaptive Policymaking
Walker et al.  specified a generic, structured approach for designing dynamic robust plans, called Adaptive Policymaking (APM). This approach was specifically developed to support the implementation of long-term plans despite the presence of uncertainties. The adaptive approach makes adaptation over time explicit at the outset of plan formulation. Thus, the inevitable changes become part of a larger, recognized process and are not forced to be made repeatedly on an ad hoc basis. Planners, through monitoring and corrective actions, would try to keep the system headed toward the original goals. McCray et al.  describe it succinctly as keeping plans “yoked to an evolving knowledge base”.
APM occurs in two phases: (1) the design phase, in which the dynamic adaptive plan, monitoring program, and various pre- and post-implementation actions are designed, and (2) the implementation phase, in which the plan and the monitoring program are implemented and adaptive actions are taken, if necessary. Once the basic dynamic adaptive plan is established, the plan is implemented, and monitoring commences (Figure 2). The process is not necessarily linear, since the monitoring is continuous and may lead to different actions taken at various points over time. The framework was introduced by Walker et al. and has been revised and clarified over time [36,37]. The approach has sometimes been called “Dynamic Adaptive Planning” (DAP) in the literature.
In Step I, the existing conditions of a system are analyzed and the objectives for future development are specified. In Step II, the way in which these objectives are to be achieved is specified by assembling a basic plan. This basic plan is made more robust (i.e., the chance that it will meet the objectives across a range of plausible futures is increased) through four types of actions (Step III): mitigating actions (actions to reduce the likely adverse effects of a plan); hedging actions (actions to spread or reduce the uncertain adverse effects of a plan); seizing actions (actions taken to seize likely available opportunities); and shaping actions (actions taken to reduce failure or enhance success). Even with the actions taken in Step III, there is still the need to monitor the plan’s performance and to take action if necessary. This is called contingency planning (Step IV). Signposts specify information that should be tracked in order to determine whether the plan is meeting the conditions for its success. In addition, critical values of signpost variables (triggers) beyond which additional actions should be implemented are specified. There are four different types of actions that can be triggered by a signpost, which are specified in Step V: defensive actions (actions taken to clarify the basic plan, preserve its benefits, or meet outside challenges in response to specific triggers that leave the basic plan unchanged); corrective actions (adjustments to the basic plan); capitalizing actions (actions to take advantage of opportunities that can improve the performance of the basic plan); and a reassessment of the plan (initiated when the analysis and assumptions critical to the plan’s success have clearly lost validity). The Step III actions are anticipatory and concurrent; the Step IV actions are reactive.
APM has been applied for strategic planning for airports , expansion of the port of Rotterdam , flood risk management in the Netherlands in light of climate change , policies with respect to the implementation of innovative urban transport infrastructures , congestion road pricing , intelligent speed adaptation , magnetically levitated (Maglev) rail transport , and energy transitions .
2.4. Adaptation Tipping Points and Adaptation Pathways
There is a need to include time and feedbacks in an analysis for adaptive policies, since an endpoint in the future is not only determined by what we have experienced in the past and envision for the future, but also by what we will experience on our way to the future , how we respond to changes over time, and how our vision for the future changes over time . Considering time in the development of a plan, results in a plan that can adapt over time as conditions change. A first step towards including time was the use of signposts, triggers, and contingency actions, such as is done in APM.
Both the Adaptation Tipping Point (ATP) and Adaptation Pathways (AP) approaches consider the timing of actions explicitly in their approach. The ATP approach  was developed in response to a request from Dutch water managers arising out of their experiences in the development of the National Water Agreement in 2003 , which required that they update their plans in response to the release of new climate scenarios [48,49]. They desired a planning approach in which the final plan was less dependent on the climate scenarios available at the time of producing the plan. Instead of asking “what if scenario x occurs?”, the Adaptation Tipping Point approach focuses on “under what conditions will a given plan fail”, which is analogous to the question that is asked in Assumption-Based Planning. These are conditions under which the magnitude of external change is such that the current management strategy can no longer meet its objectives and new actions are needed to achieve the objectives. When this occurs in time (and even if it occurs) is unknown—it is dependent on the scenario. Consequently, under the ATP approach, when new climate scenarios are presented, only the timing of actions needed to protect the plan from failing need to be updated to reflect situations in the new scenarios.
The Adaptation Pathways approach [2,45] (also referred to as the “route-map” approach or “decision pathways” approach) is a logical extension of the ATP approach, since the reaching of a tipping point requires new actions to be implemented to achieve the objectives. As a result, a pathway emerges. Although the AP approach was developed with a focus on water management, it is a generic approach that can be applied to other long-term strategic planning problems. The approach encourages decisionmakers to think about “what if” situations and their outcomes, and to make decisions over time to adapt while maintaining flexibility with respect to making future changes . The approach aims at building flexibility into the overall adaptation strategy (rather than into the individual actions) by sequencing the implementation of actions over time in such a way that the system is adapted over time to changing climate, social, economic conditions, etc., and options are left open to deal with a range of plausible future conditions.
The AP approach produces an overview of alternative routes into the future. An effective way of communicating this is shown in Figure 3. Similar to a Metro map (see, for example, ), the adaptation pathways map presents different routes to get to a desired point in the future. All routes presented meet pre-specified minimum performance levels, such as a safety norm. They can, thus, be considered as “different ways leading to Rome” (as is true of different routes to a specified destination on the Metro). Also, the moment of an ATP (“terminal station”) and the available actions at all potential ATPs are shown (via “transfer stations”). Due to constraints on actions, some routes are available only in some of the scenarios (dashed lines). Actions that add little to the performance of a route are translucent. With the map, it is possible to identify opportunities, no-regret actions, lock-ins, and the timing of actions, in order to support decisionmaking in a changing environment. That is, the adaptation map can be used to prepare a plan for actions to be taken immediately (anticipatory and concurrent), and for preparations that need to be made in order to be able to implement an action in the future in case conditions change (reactive).
The ATP approach was first applied to water management in the Netherlands in 2011. Although the ATP approach is relatively new, it has been used in several other studies, such as flood risk management in the city of Rotterdam , nature restoration in the Rhine basin and wine production in Italy , risk management in New Zealand , and risk management in the Elbe basin . The adaptation pathways approach was extensively tested using a hypothetical case called “the Waas” [45,56,57,58]. Practical examples of the ATP and AP approach include the UK Thames Barrier project [50,59,60] and the Dutch Delta Programme [50,59]. For New York City, an approach is being developed called “flexible adaptation pathways” , which shares a strong family resemblance with both the AP approach and Holling’s work on adaptive management [62,63]. Both ATP and AP approaches have also been used to explore socially robust strategies using the Perspectives method [45,64].
2.5. Dynamic Adaptive Policy Pathways (DAPP)
The Dynamic Adaptive Policy Pathways (DAPP) approach  combines the work on Adaptive Policymaking with the work on Adaptation Tipping Points and Adaptation Pathways. Figure 4 shows the overall approach.
As in the other approaches, the DAPP approach begins with the identification of objectives, constraints, and uncertainties that are relevant for decisionmaking. The uncertainties are then used to generate an ensemble of plausible futures. These futures are compared with the objectives to see if problems arise or if opportunities occur. This determines if and when (reactive) policy actions are needed. To assemble a rich set of possible actions, the approach distinguishes among four types of actions, which are defined in the same way as in APM: shaping actions, mitigating actions, hedging actions, and seizing actions . In subsequent steps, these actions are used as the basic building blocks for the assembly of adaptation pathways. The performance of each of the actions and pathways is assessed in light of the defined objectives to determine its adaptation tipping point. Once a set of actions seems adequate, potential pathways (a sequence of actions) can be constructed, and subsequently one or more preferred pathways can be selected as input for a dynamic robust plan. The aim of this plan is to keep the preferred pathway open as long as possible. For this purpose, contingency actions are specified and a trigger for each contingency action is specified and monitored. This approach is being tested on a fictitious case , and is being applied in a real case involving the Lower Rhine Delta of the Netherlands .
3. Computational Support Tools for Designing Sustainable Plans
Often, analysts use simulation models to quantitatively explore various futures (e.g., [67,68,69]). Within this school of computational scenario-based approaches, it is common to use a small set of (two to four) scenarios for one or two projection years to design (static) robust plans (plans that will do well no matter which of the scenarios actually occurs) (e.g., [70,71]). In the context of supporting adaptive planning, this computational scenario-based approach can be further extended. In this section, we discuss several aspects that are of relevance for the use of computational scenario-based approaches for supporting the design of sustainable plans using the approaches described in Section 2.
3.1. Fast and Simple Policy Models
Models come in many shapes and sizes. A useful distinction can be made between policy models and scientific models. Scientific or engineering models aim at obtaining a better understanding of a well-defined clearly demarcated system. The better the match between the model and the real world, the better the model is considered to be (a close match implies a valid model). Policy models give policymakers insights into their (future) problem situation on which they can base their decisions . Policy models serve as laboratory environments, to test alternative policies, and compare their performance without actually having to implement them in the real world to see how they would perform. The main purpose of models to support the development of sustainable plans is similar to that of a policy model—to assess a large number of alternative actions under different possible futures, and to design adaptation pathways. In this case, the main purpose of the model is not to provide the solution, but to provide information supporting decisionmakers.
Given the fact that models for developing sustainable plans require the evaluation of alternative actions over a wide variety of plausible futures, an additional requirement of these models is that they have a relatively short runtime. There are various ways in which such models can be developed, including meta-modeling and the use of modeling approaches that result in relatively fast models. Meta-models are models of models, intended to mimic the behavior of larger, more complex models. Such models are also known as “low resolution models”, “repro models”, or “response surfaces”. There are two ideal types of meta-models: statistical and theory based. Often, however, actual meta-models are a mixture of these ideal types, motivated by phenomenological considerations [73,74].
Designing a policy model is a balancing act between model completeness (in terms of considering all policy-relevant components), model credibility (in terms of physical detail and validity), and flexibility and calculation time of a simulation. To make the model manageable, simplifications in time scale, spatial scale, and processes are needed. A complex model can subsequently be used to obtain more detailed information about the performance of the most promising options resulting from the policy exploration.
Another requirement for policy models follows from the need to cover a wide variety of outcome indicators. We need models that not only focus on a part of the system, but include enough of the system relationships to be able to estimate all relevant outcome indicators. Therefore, these models should be integrated impact assessment models. An example of such a policy model that has been used in demonstrating the AP approach is the integrated assessment meta-model of the Waas case study . This model is what Welsh et al.  call a “new generation model”. They argue, with respect to water management, that “with the increasing complexity of water management, sectorial applications, such as separate groundwater and surface water models, are becoming outdated and that water managers are increasingly looking for new generation tools that allow integration across domains to assist in their decisionmaking processes for short-term operations and long-term planning; not only to meet current needs, but those of the future as well”.
3.2. Exploratory Modeling and Analysis (EMA)
Exploratory Modeling and Analysis (EMA) is a research methodology that uses computational experiments to explore uncertainties in the context, the system model, and different perspectives [76,77,78,79]. EMA begins by acknowledging the fact that a validatable predictive long-term policy model cannot be built [80,81]. It then asks the question “in that case, how can we still use our model?” . As noted above, in situations with deep uncertainty, relying on a “best estimate” model to predict system behavior can result in the choice of a very poor plan. Therefore, rather than attempting to predict system behavior, EMA aims to analyze and reason about the system’s behavior , for example by using several plausible models of the system.
EMA supports this process of researching a broad range of assumptions and circumstances. In particular, EMA involves exploring a wide variety of scenarios, alternative model structures, and alternative value systems. The exploration is carried out using computational experiments. A computational experiment is a single run with a given model structure and a given parameterization of that structure. It reveals how the real world would behave if the various hypotheses presented by the structure and the parameterization were correct. By exploring a large number of these hypotheses, one can get insights into how the system would behave under a large variety of assumptions. In published examples the number of hypotheses vary, but are between roughly 10  and 60 .
To support the exploration of these hypotheses, data mining techniques for analysis and visualization are employed. EMA aims to “cover the space” of possibilities, which can be described as the space being created by the uncertainty surrounding the many variables. Because each model run is treated as a deterministic hypothesis about the system of interest, EMA does not require the a priori assignment of likelihood or probability to uncertainty variables.
In EMA, relatively fast and simple computer models of the system are applied. Because EMA aims to cover the whole space of possibilities, it is usually necessary to make huge numbers of computer runs (thousands to hundreds of thousands). With traditional best estimate models this would take too much time. With fast and simple policy models, one can cover the entire uncertainty space, and then drill down into more detail where initial results suggest interesting system behavior (e.g., the boundary between a plan’s success and failure). EMA has been used to support applications of APM and DAPP.
3.3. Scenario Discovery
Scenario Discovery is a model driven approach based on the intuitive logic school  that builds on earlier work on EMA [76,85]. It starts from an ensemble of model runs that is analyzed in order to identify runs that are of particular interest. Next, these runs of interest are analyzed to reveal the combinations of factors responsible for generating them. These combinations of factors are called “scenarios”. (In this case, “scenarios” does not have its traditional meaning as a set of plausible future external contexts in which a plan might need to function.) Results of interest can be identified based on the performance of candidate plans, but other criteria can also be used. Scenario Discovery has recently been extended to cope with dynamics over time  and multiple outcomes of interest .
In many applications of Scenario Discovery, the runs of interest are determined based on the failure to meet pre-specified objectives. The mindset behind Scenario Discovery is one of the most important defining differences between traditional ex-ante policy analysis and the analysis underlying the design of robust plans. In the former, scenarios would be specified and plans would be evaluated on how well they performed across the scenarios. The “best” (static robust) plan would the one that performed the best across all of the scenarios. In contrast, Scenario Discovery is performed to identify the scenarios in which a plan would perform poorly. These scenarios highlight the vulnerabilities of the plan. Then, actions are specified to protect the plan from failing. APM, RDM, and DAPP all can use Scenario Discovery to identify combinations of external events or situations that would lead to the failure of the plan being investigated (i.e., the “perishing” of the plan).
Scenario Discovery has been applied to various cases. Bryant and Lempert  apply it to renewable energy in the United States. Groves and Lempert  report on an application for water resource management in California. Kwakkel et al.  apply dynamic Scenario Discovery to material scarcity. Gerst et al.  apply multi-dimensional Scenario Discovery to an energy transition case.
3.4. Robust Optimization
Optimization is a very popular tool for supporting decisionmaking. Optimization can be defined as trying to find the best solution among a set of possible alternatives without violating certain constraints. It is mostly employed for predictive purposes, where the aim is to identify a single best estimate solution. However, under deep uncertainty, this predictive approach cannot be used for decisionmaking, since usually an optimum solution does not exist [87,88]. Robust optimization aims to overcome this difficulty. Robust optimization methods aim at finding optimal outcomes in the presence of uncertainty about input parameters [89,90,91,92].
Robustness can be operationalized in a wide variety of ways. Rosenhead et al.  understand robustness as flexibility—that is, as leaving options open. Other ways of operationalizing robustness include Wald’s minimax criterion, which chooses the decision alternative that minimizes the maximum risk ; minimax regret , which results in choosing the solution with the least maximum regret ; and various forms of satisficing , such as risk discounting, and certainty equivalents . EMA can be combined with robust optimization in a number of ways: robustness can be defined as the first order derivative of the objective function ; as a reasonable performance over a wide range of plausible futures [15,43]; as regret [5,96]; and as sacrificing a small amount of optimal performance in order to be less sensitive to violated assumptions . This last definition bears a large similarity to the local robustness model employed in Info-Gap decision theory [27,97].
Along a different dimension, a distinction can be made between single-objective optimization and multi-objective optimization. In the single objective optimization case, multiple objectives are combined, drawing on Multi-Criteria Decision Analysis (MCDA) approaches . In the multi-objective case, one aims at identifying the Pareto frontier, leaving discussions about tradeoffs among the various objectives out of the optimization.
Examples of the application of robust optimization in support of adaptive policymaking can be found in Hamarat et al.  on energy transitions, and Kwakkel et al.  on long-term airport planning. Kwakkel et al.  combine robust optimization and Dynamic Adaptive Policy Pathways for the development of a long-term water management adaptation strategy in response to climate change and socio-economic uncertainty.
An information gap is defined as the disparity between what is known and what needs to be known in order to make a reliable and responsible decision. Info-Gap decision theory is a non-probabilistic decision theory that seeks to optimize robustness to failure—or opportunity for windfall—under “severe uncertainty” [97,99]. The theory underlying Info-Gap was initially developed by Ben-Haim in the context of the reliability of mechanical systems. [97,99] Increasingly, it is also being applied to support the development of robust plans. Info-Gap starts with a set of available actions and evaluates the actions computationally. It can, therefore, be considered as a computational support tool (although it could also be categorized as an approach for robust decisionmaking). In contrast to the other approaches considered in this paper, unforeseen events (Black Swans) are not incorporated: Info-Gap addresses modeled uncertainty, not unexpected uncertainty.
Info-Gap analysis uses a non-probabilistic model of the uncertainty encountered in the decision problem, a policy model to evaluate the effects of alternative actions given the uncertainty, and a specification of minimum performance requirements that an action should satisfy . In contrast to Scenario Discovery, where actions can be developed iteratively, Info-Gap assumes that the actions are known prior to the analysis . It then proceeds to evaluate how large the uncertainty should become before a given action fails to meet its specified performance requirements. In this analysis, both the robustness and the “opportuneness” of strategies is evaluated. Robustness is the minimum performance above the threshold for a given level of uncertainty. Opportuneness is the maximum level of performance above the threshold for a given level of uncertainty . The main results from an Info-Gap analysis are a visualization of the robustness and opportuneness of different actions as a function of the level of uncertainty. Recently, Hall et al.  presented a quantitative comparison of RDM and Info-Gap. They identified many similarities, such as the multiple plausible representations of uncertainty and the use of quantified system models.
Info-Gap theory has been applied in a range of practical applications, including the reduction of greenhouse gasses , flood risk [100,101], biological conservation , and water resources planning .
In this paper, we have provided an overview of one family of approaches to adaptive planning—those that have their roots conceptually in Assumption-Based Planning—and tools used to support these approaches computationally. Comparing the approaches, we note both commonalities and differences. All approaches represent uncertainties with sets of multiple plausible futures instead of probabilities over future states of the world. The way Exploratory Modeling and Analysis is used in combination with Adaptive Policymaking explicitly includes uncertainties arising from simulating the real world in a system model. What is considered a “plausible” future is subject to different interpretations, and depends on one’s expectations about the future and understanding of the system. Moreover, what is considered acceptable performance of a policy depends on people’s values. A concept called “Perspectives” has been used to describe people’s dynamic view of the value of water and how water should be managed [64,70,104,105]. Recently, the Scientiﬁc Council for Government Policy in the Netherlands requested attention for including such uncertainties in scenario studies . Adaptation Tipping Points and Adaptation Pathways have been used in combination with the concept of Perspectives to explore uncertainties arising from different values and expectations about the future [45,64,107]. The use of Perspectives not only enables the identification of physically robust pathways, but also socially robust pathways.
All approaches aim at enhancing a plan by keeping it from failing. All approaches include (some more explicit then others) the question: what could make a plan fail? For example, in defining adaptation tipping points, this question is rephrased into: under what conditions does a plan perform unacceptably? Adaptive Policymaking explicitly distinguishes different types of actions to keep a policy from failing. Such actions can result from the last three steps in Robust Decision Making. Both Adaptive Policymaking and Robust Decision Making have similar first steps, and go back one step in policymaking compared to the Assumption-Based Planning approach. They do not assume that a plan exists, but begin by first designing the plan to be examined. Robust Decision Making differs from Adaptive Policymaking, Adaptation Pathways, and Dynamic Adaptive Policy Pathways in not explicitly considering the dynamic adaptation of the plan over time, while the other three place increasing emphasis on this aspect. That is, the last three approaches produce dynamic robust plans (covering anticipatory, concurrent, and reactive adaptation), while Robust Decision Making produces a static robust plan (focusing on anticipatory adaptation).
Looking at the family of adaptive planning approaches, we note several recurring “principles” for planned adaptation. The essential idea of planned adaptation is that planners facing deep uncertainty create a shared strategic vision of the future, explore possible adaptation strategies and pathways, commit to short-term actions, while keeping long-term actions open, and prepare a framework (including in some cases a monitoring system, triggers, and contingency actions) that guides future actions. Implicit in this is that planners accept the irreducible character of the uncertainties about the future and aim to reduce uncertainty about the expected performance of their plans. So, planners have to accept—and in a sense embrace—uncertainty, rather than spending large amounts of time and effort on trying to reduce it, and waiting to take action until the uncertainties have been resolved.
In order to develop dynamic robust plans, planners have to explore a large variety of futures to assess what actions can be used to achieve their objectives despite how the future unfolds. This is realized by connecting long-term objectives with short-term targets. It necessitates the analysis of actions over time, rather than looking only at future states of the world. Adaptivity and the avoidance of lock-ins are necessary conditions to achieving the objectives no matter how the future unfolds. Adaptivity includes the pre-specification of contingency actions and the monitoring of developments over time in order to activate the contingency actions if and when needed. It is conceivable that, despite the thorough exploration of the future, events or developments occur that have not been considered or are so extreme that a reassessment is needed. The conditions for this can be explicitly made part of the plan. Adaptation can, therefore, keep a plan alive and prevent the policymaker’s vision from perishing.
Figure 5 is a map of the discussed approaches for developing adaptive policies according to the level of uncertainty that they address and the type of adaptation concept being used by the approach. Static means that timing is not explicitly considered. Static robust means that the adaptation is primarily anticipatory in character. Dynamic means that the adaptation can be anticipatory, concurrent, and reactive. The level of uncertainty specifies the degree of uncertainty. It can range from low, well characterized uncertainty, to deep uncertainty, and even recognized ignorance [4,6,108].
Assumption-Based Planning covers static robustness, since it analyzes the critical assumptions of an existing plan, but does not continue to be used to cope with changes in the world. Robust Decision Making no longer requires an existing plan, but still aims at developing a static plan rather than a dynamic plan. Adaptive Policymaking makes clear the importance of monitoring and adapting to changes over time to prevent the basic (static) plan from failing. Adaption Tipping Points can be categorized as static, since it specifies the conditions and time frame under which a new or an additional management strategy is needed. It is most useful in helping to identify vulnerabilities in the system and where actions may needed to be taken first. More information about timing is added in the Adaptation Pathways concept, producing a more dynamic approach. Dynamic Adaptive Policy Pathways includes both the pathway idea and the contingency planning concepts from Adaptive Policymaking. Both of these approaches provide support to identifying options and vulnerabilities of a plan over time, and are, therefore, well suited for situations that are apt to undergo relatively frequent large changes. In practice, of course, there is a great deal of overlap among the approaches, and the differences may not be as clear as shown in Figure 5.
The various planning approaches can be supported by computer models. This does not imply that other techniques, such as qualitative scenarios, backcasting , Delphi , etc., do not also have their place in the adaptive planner’s toolbox. Especially in developing countries, data to feed computational tools can be scarce. In the literature, the approaches we have discussed are most often combined with computational techniques to support the development of adaptive plans. This is clearest for Robust Decision Making, which is almost completely dependent on computer runs. However, all approaches emphasize and require relatively fast and simple policy models. This is motivated by the fact that there is a need to explore a wide variety of uncertain futures and actions. Scenario Discovery, Robust Optimization, Info-Gap, and fast and simple policy models work together well. The reliance on fast simple models is consistent with the adage that it is better to be roughly right than precisely wrong. For the types of problems we are addressing, the level of uncertainty typically swamps the increased precision of more detailed models. This does not imply that more detailed models do not have their place in the adaptive planning process. Once there is an emerging consensus on a dynamic robust plan, more detailed models can be used to further detail this plan. Through Scenario Discovery, the crucial scenarios can be identified in order to maximize the effective use of the computational resources.
5. Concluding Remarks
“Monitor and adapt” is gradually becoming preferred to “predict and act” as the strategy for long-term planning in the face of deep uncertainty—i.e., as a means to design sustainable plans that are able to achieve economic, environmental, and social objectives for a long-term uncertain future. To this end, a sustainable plan should be robust, meaning that it performs satisfactorily under a wide variety of futures and can be adapted over time to (unforeseen) future conditions. There are different approaches for developing sustainable plans, but with some similar elements. To date, few formal comparisons of these approaches exist. We have provided a first attempt at comparison, focusing on tools and approaches that have their roots in Assumption-Based Planning and that can be used to design a sustainable plan under deep uncertainty. Further work needs to be done on the systematic comparison of approaches and computational tools to assist decisionmakers and planners in choosing among and employing these approaches more effectively. Recently, Hall et al.  made a start to address these issues by applying Robust Decision Making and Info-Gap on the same case and comparing the results.
Based on our work, we have found that key principles in developing long-term sustainable plans are:
Explore a wide variety of relevant uncertainties in a dynamic way. That is, uncertainties in natural variability, external changes, and policy responses need to be explored over time. For example, climate change may affect precipitation, resulting in floods and droughts, which may initiate a policy response (e.g., dams and dikes) that may affect the water system, initiate urban developments, and influence future policy options.
Connect short-term targets and long-term goals.
Commit to short-term actions while keeping options open for the future.
There is evidence that such policies are efficacious  and cost-beneficial . As a means for computational support, fast and simple policy models are used in order to explore a wide variety of uncertainties and actions in a dynamic way. For such an analysis, computational experiments, such as in Exploratory Modeling and Analysis and Scenario Discovery, are used. These approaches are beginning to be used in practice (e.g., the Thames Estuary in the UK, the Rhine-Meuse delta in the Netherlands, the Port of Los Angeles and New York City in the USA). However, more needs to be done to bridge the gap between theory and practice in order to produce plans that adapt and lead to survival, not perishing.
Conflict of Interest
The authors declare no conflict of interest.
- World Commission on Environment and Development. Our Common Future (the Bruntland Report); Oxford University Press: Oxford, UK, 1987.
- Haasnoot, M.; Middelkoop, H.; van Beek, E.; van Deursen, W.P.A. A method to develop sustainable water management strategies for an uncertain future. Sust. Develop. 2011, 19, 369–381, doi:10.1002/sd.438.
- McDaniel, R.R.; Driebe, D.J. Uncertainty and Suprise in Complex Systems: Questions on Working the Unexpected; Springer: Berlin, Germany, 2005.
- Walker, W.E.; Lempert, R.; Kwakkel, J.H. Deep Uncertainty. In Encyclopedia of Oper. Res. and Management Science, 3rd; Gass, S., Fu, M., Eds.; Springer: Berlin, Germany, 2013.
- Lempert, R.J.; Popper, S.; Bankes, S. Shaping the Next One Hundred Years: New Methods for Quantitative, Long Term Policy Analysis; Report MR-1626-RPC; RAND: Santa Monica, CA, USA, 2003.
- Kwakkel, J.H.; Walker, W.E.; Marchau, V.A.W.J. Classifying and communicating uncertainties in model-based policy analysis. Int. J. Technol. Policy Manage. 2010, 10, 299–315, doi:10.1504/IJTPM.2010.036918.
- Dessai, S.; Hulme, M. Assessing the robustness of adaptation decisions to climate change uncertainties: A case study on water resources management in the East of England. Global Environ. Change 2007, 17, 59–72.
- McInerney, D.; Lempert, R.; Keller, K. What are robust strategies in the face of uncertain climate threshold responses. Climate Change 2012, 112, 547–568, doi:10.1007/s10584-011-0377-1.
- Annema, J.A.; de Jong, M. The History of the Transport Future: Evaluating Dutch Transport Scenarios of the Past. Transport Rev. 2011, 31, 341–356, doi:10.1080/01441647.2010.527440.
- Dessai, S.; Hulme, M. How Do UK Climate Scenarios Compare with Recent Observations? Atmos. Sci. Lett. 2008, 9, 189–195, doi:10.1002/asl.197.
- Goodwin, P.; Wright, G. The limits of forecasting methods in anticipating rare events. Techno. Forecasting Soc. Change 2010, 77, 355–368, doi:10.1016/j.techfore.2009.10.008.
- Van Drunen, M.; Leusink, A.; Lasage, R. Towards a Climate-Proof Netherlands. In Water Management in 2020 and Beyond; Biswas, A.K., Tortajada, C., Izquierdo, R., Eds.; Springer: Berlin/Heidelberg, Germany, 2009.
- Taleb, N.N. The Black Swan; Random House: New York, NY, USA, 2007.
- Rosenhead, J.; Mingers, J. Rational Analysis of a Problematic World Revisted: Problem Structuring Methods for Complexity, Uncertainty, and Conflict, 2nd ed.; Wiley and Sons: Chichester, UK, 2001.
- Lempert, R.J.; Collins, M. Managing the risk of uncertain threshold response: Comparison of robust, optimum, and precautionary approaches. Risk Analysis 2007, 24, 1009–1026, doi:10.1111/j.1539-6924.2007.00940.x.
- Dewey, J. The Public and its Problems; Holt and Company: New York, NY, USA, 1927.
- IISD. Designing Policies in a World of Uncertainty, Change and Surprise—Adaptive Policy-Making for Agriculture and Water Resources in the Face of Climate Change—Phase I Research Report; International Institute for Sustainable Development: Winnipeg, Canada, 2006.
- Burton, I. Climate Change and the Adaptation Deficit. In The Earthscan Reader on Adaptation to Climate Change; Schipper, E.L.F., Burton, I., Eds.; Earthscan: London, UK, 2009.
- Van der Heijden, K. Scenarios: The Art of Strategic Conversation; John Wiley & Sons: Chichester, England, 1996.
- Carter, T.R.; Jones, R.N.; Lu, X.; Bhadwal, S.; Conde, C.; Mearns, L.O.; O’Neill, B.C.; Rounsevell, M.D.A.; Zurek, M.B. New Assessment Methods and the Characterisation of Future Conditions. In Climate Change 2007: Impacts, Adaptation and Vulnerability. Contribution of Working Group II to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change; Parry, M.L., Canziani, O.F., Palutikof, J.P., van der Linden, P.J., Hanson, C.E., Eds.; Cambridge University Press: Cambridge, UK, 2007.
- Dewar, J.A.; Builder, C.H.; Hix, W.M.; Levin, M.H. Assumption-Based Planning: A Planning Tool for Very Uncertain Times; Report MR-114-A; RAND: Santa Monica, 1993.
- Walker, W.E.; Rahman, S.A.; Cave, J. Adaptive policies, policy analysis, and policymaking. Eur. J. Oper. Res. 2001, 128, 282–289, doi:10.1016/S0377-2217(00)00071-0.
- Dewar, J.A. Assumption-Based Planning: A Tool for Reducing Avoidable Surprises; Cambridge University Press: Cambridge, UK, 2002.
- Dewar, J.A.; Levin, M.H. Assumption-Based Planning for Army 21; R-4172-A; RAND: Santa Monica, CA, USA, 1992.
- Groves, D.G.; Lempert, R.J. A new analytic method for finding policy-relevant Scenarios. Global Environ. Change 2007, 17, 73–85, doi:10.1016/j.gloenvcha.2006.11.006.
- Keefe, R. Reconsidering California Transport Policies: Reducing Greenhouse Gas Emissions in an Uncertain Future. Ph.D. dissertation, Pardee RAND Graduate School, 2012.
- Hall, J.W.; Lempert, R.J.; Keller, A.; Hackbarth, A.; Mijere, C.; McInerney, D. Robust climate policies under uncertainty: A comparison of robust decision making and info-gap methods. Risk Analysis 2012, 32, 16527–11672.
- Seong, S.; Popper, S.; Zheng, K. Strategic Choices in Science and Technology: Korea in the Era of a Rsing China; MG-320-KISTEP; RAND: Santa Monica, CA, 2005.
- Lempert, R.J.; Schlesinger, M.E.; Bankes, S.C. When we don’t know the cost or the benefits: Adaptive strategies for abating climate change. Climatic Change 1996, 33, 235–274, doi:10.1007/BF00140248.
- Fischbach, J.R. Managing New Orleans Flood Risk in an Uncertain Future using Non-Structural Risk Mitigation. Ph.D. dissertation, RAND Graduate School, 2010.
- Lempert, R.J.; Sriver, R.L.; Keller, K. Characterizing Uncertain Sea Level Rise Projections to Support Investment Decisions; California Energy Commission: Sacramento, CA, USA, 2012.
- Popper, S.; Griffin, J.; Berrebi, C.; Light, T.; Min, E.Y. Natural Gas and Israel’s Energy Future: A Strategic Analysis Under Conditions of Deep Uncertainty; TR-747-YSNFF; RAND: Santa Monica, CA, USA, 2009.
- Lempert, R.J.; Groves, D.G. Identifying and evaluating robust adaptive policy responses to climate change for water management agencies in the American west. Technol. Forecast. Soc. 2010, 77, 960–974, doi:10.1016/j.techfore.2010.04.007.
- Groves, D.G. New Methods for Identifying Robust Long-Term Water Resources Management Strategies for California. Ph.D. dissertation, Pardee RAND Graduate School, Santa Monica, CA, USA, 2005.
- McCray, L.E.; Oye, K.A.; Petersen, A.C. Planned adaptation in risk regulation: an initial survey of US environmental, health, and safety regulation. Technol. Forecast. Soc. 2010, 77, 951–959, doi:10.1016/j.techfore.2009.12.001.
- Kwakkel, J.H.; Walker, W.E.; Marchau, V.A.W.J. Adaptive airport strategic planning. Eur J. Trans. Infr. Res. 2010, 10, 227–250.
- Walker, W.E.; Marchau, V.A.W.J.; Kwakkel, J.H. Uncertainty in the Framework of Policy Analysis. In Public Policy Analysis: New Developments; Thissen, W.A.H., Walker, W.E., Eds.; Springer: Berlin, Germany, 2013.
- Taneja, P.; Walker, W.E.; Ligteringen, H.; van Schuylenburg, M.; van der Plas, R. Implications of an uncertain future for port planning. Maritime Policy Manage. 2010, 37, 221–245, doi:10.1080/03088831003700637.
- Rahman, S.A.; Walker, W.E.; Marchau, V. Coping with Uncertainties about Climate Change in Infrastructure Planning—An Adaptive Policymaking Approach; Ecorys: Rotterdam, The Netherlands, 2008.
- Marchau, V.A.W.J.; Walker, W.E.; van Duin, R. An adaptive approach to implementing innovative urban transport solutions. Transport Policy 2008, 15, 405–412, doi:10.1016/j.tranpol.2008.12.002.
- Marchau, V.A.W.J.; Walker, W.E.; van Wee, G.P. Dynamic adaptive transport policies for handling deep uncertainty. Technol. Forecast. Soc. 2010, 77, 940–950, doi:10.1016/j.techfore.2010.04.006.
- Agusdinata, B.; van Der Pas, J.W.G.M.; Walker, W.E.; Marchau, V.A.W.J. Multi-criteria analysis for evaluating the impacts of intelligent speed adaptation. J. Adv. Transport. 2009, 43, 413–454, doi:10.1002/atr.5670430402.
- Hamarat, C.; Kwakkel, J.H.; Pruyt, E. Adaptive robust design under deep uncertainty. Technol. Forecast. Soc. 2012, doi:10.1016/j.techfore.2012.10.004.
- Yohe, G.W. Imbedding Dynamic Responses with Imperfect Information into Static Portraits of the Regional Impact of Climate Change. In Presented at International Workshop on the Natural Resource and Economic Implications of Global Climate Change, Interlaken, Switzerland, 5–9 November 1990.
- Haasnoot, M.; Middelkoop, H.; Offermans, A.; van Beek, E.; van Deursen, W.P.A. Exploring pathways for sustainable water management in river deltas in a changing environment. Climatic Change 2012, 115, 795–819, doi:10.1007/s10584-012-0444-2.
- Kwadijk, J.C.J.; Haasnoot, M.; Mulder, J.P.M.; Hoogvliet, M.M.C.; Jeuken, A.B.M.; van der Krogt, R.A.A.; van Oostrom, N.G.C.; Schelfhout, H.A.; van Velzen, E.H.; van Waveren, H.; et al. Using Adaptation Tipping Points to Prepare for Climate Change and Sea Level Rise: A Case Study in the Netherlands. Wiley Interdisciplinary Reviews: Climate Change 2010, 1, 729–740.
- Ministerie van Verkeer en Waterstaat Nationaal Bestuursakkord Water. Available online: http://www.ipo.nl/files/8913/5722/8547/nbw.pdf (accessed on 4 February 2013).
- Van den Hurk, B.; Tank, A.K.; Lenderink, G.; van Ulden, A.; van Oldenborgh, G.J.; Katsman, C.; van den Brink, H.; Keller, F.; Bessembinder, J.; Burgers, G.; et al. New climate change scenarios for the Netherlands. Water Sci. Technol. 2007, 56, 27–33.
- Katsman, C.; Hazeleger, W.; Drijfhout, S.; van Oldenborgh, G.J.; Burgers, G. Climate scenarios of sea level rise for the northeast atlantic ocean: A study including the effects of ocean dynamics and gravity changes induced by ice melt. Climatic Change 2008, 91, 351–374, doi:10.1007/s10584-008-9442-9.
- Jeuken, A.; Reeder, T. Short-term decision making and long-term strategies: How to adapt to uncertain climate change. Water Governance 2011, 1, 29–35.
- Washington Metropolitan Area Transit Authority Interactive Map. Available online: http://www.wmata.com/rail/maps/map.cfm (accessed on 4 February 2013).
- Gersonius, B.; Nasruddin, F.; Ashley, R.; Jeuken, A.; Pathirana, A.; Zevenbergen, C. Developing the evidence base for mainstreaming adaptation of stormwater systems to climate change. Water Res. 2012, 6824–6835.
- Werners, S.; Swart, R.; van Slobbe, E.; Bölscher, T.; Pfenninger, S.; Trombi, G.; Moriondo, M. Turning Points. In Climate Change Adaptation, The Governance of Adaptation—An International Symposium, Amsterdam, The Netherlands, 22–23 March, 2007; The Governance of Adaptation: Amsterdam, The Netherlands, 2007.
- Lawrence, J.; Manning, M. Developing Adaptive Risk Management for Our Changing Climate; A Report of Workshop Outcomes under an Envirolink Grant; The New Zealand Climate Change Research Institute, Victoria University of Wellington: Wellington, New Zealand, 2012.
- Schulte-Rentrop, A.; Rudolph, E. A Sensitivity Study of Storm Surges Under the Conditions of Climate Change in the Elbe Estuary. In Climate Change and Disaster Risk Management; Leal Filho, W., Ed.; Springer: Berlin/Heidelberg, Germany, 2013.
- Kwakkel, J.H.; Haasnoot, M.; Walker, W.E.; Wijermans, N. Developing Dynamic Adaptive Policy Pathways: A Computer-Assisted Approach to Developing Robust Adaptation Strategies for a Deeply Uncertain World. In Proceedings of Climate Impact Research & Response Coordination for a Larger Europe (CIRCLE 2) Workshop on Uncertainty and Climate Change Adaptation, Lisbon, Portugal, 8–9 November 2012.
- Kwakkel, J.H.; Haasnoot, M. Computer Assisted Dynamic Adaptive Policy Design for Sustainable Water Management in River Deltas in a Changing Environment. In International Congress on Environmental Modelling and Software Managing Resources of a Limited Planet, Sixth Biennial Meeting; Seppelt, R., Voinov, A.A., Lange, S., Bankamp, D., Eds.; Sixth Biennial Meeting: Leipzig, Germany, 2012.
- Wijermans, N.; Haasnoot, M.; Offermans, A.; Middelkoop, H. Modelling policy as social uncertainty, integrated modelling and pathway explorationunpublished work.
- Ministry of Transport. The 2011 Delta Programme: Working on the delta—Investing in a Safe and Attractive Netherlands, Now and in the Future. 2010. Available online: http://www.deltacommissaris.nl/english/Images/Deltaprogramma_ENG1_tcm310-286802.pdf (accessed on 4 February 2013).
- Wilby, R.L.; Keenan, R. Adapting to flood risk under climate change. Prog. Phys. Geog. 2012, 36, 348–378, doi:10.1177/0309133312438908.
- Rosenzweig, C.; Solecki, W.D.; Blake, R.; Bowman, M.; Faris, C.; Gornitz, V.; Horton, R.; Jacob, K.; Le Blanc, A.; Leichenko, R.; et al. Developing coastal adaptation to climate change in the New York city infrastructure-shed: Process, approach, tools, and strategies. Climatic Change 2011, 106, 93–127.
- Holling, C.S. Adaptive Environmental Assessment and Management; John Wiley & Sons: New York, New York, USA, 1978.
- Lee, K. Compass and Gyroscope: Integrating Science and Politics for the Environment; Island Press: Washington, DC, USA, 1993.
- Offermans, A.; Haasnoot, M.; Valkering, P. A method to explore social response for sustainable water management strategies under changing conditions. Sustain. Dev. 2011, 19, 312–324, doi:10.1002/sd.439.
- Haasnoot, M.; Kwakkel, J.H.; Walker, W.E.; Ter Maat, J. Dynamic adaptive policy pathways: A new method for crafting robust decisions for a deeply uncertain world. Global. Environ. Change 2003, doi:10.1016/j.gloenvcha.2012.12.006.
- Haasnoot, M.; van Deursen, W.P.A.; Middelkoop, H.; van Beek, E.; Wijermans, N. An Integrated Assessment Metamodel for Developing Adaptation Pathways for the Rhine Delta in the Netherlands. In International Congress on Environmental Modeling and Software: Managing Resources of a Limited Planet; Seppelt, R., Voinov, A., Lange, S., Bankamp, D., Eds.; International Congress on Environmental Modeling and Software: Leipzig, Germany, 2012.
- Morgan, M.G.; Dowlatabadi, H. Learning from integrated assessment of climate change. Climatic Change 1996, 34, 337–368, doi:10.1007/BF00139297.
- Rotmans, J.; De Vries, B.J.M. Perspectives on Global Change: The TARGETS Approach; Cambridge University Press: Cambridge, UK, 1997.
- Van Asselt, M.B.A. Perspectives on Uncertainty and Risk; Kluwer Academic Publishers: Dordrecht, the Netherlands, 2000.
- Middelkoop, H.; van Asselt, M.B.A.; Vanʼt Klooster, S.A.; van Deursen, W.P.A.; Kwadijk, J.C.J.; Buiteveld, H. Perspectives on flood management in the Rhine and Meuse rivers. River Res. Appl. 2004, 20, 327–342.
- Van Asselt, M.B.A.; Rotmans, J. Uncertainty in integrated assessment modelling. From positivism to pluralism. Climate Change 2002, 54, 75–105, doi:10.1023/A:1015783803445.
- Walker, W.E.; van Daalen, C. System Models for Policy Analysis. In Public Policy Analysis: New Development; Thissen, W.A.H., Walker, W.E., Eds.; Springer: Berlin, Germany, 2013.
- Davis, P.K.; Bigelow, J.H. Experiments in Multiresolution Modeling (MRM); RAND: Santa Monica, CA, USA, 1998.
- Van Grol, R.; Walker, W.E.; Rahman, S.A.; de Jong, G. Using a Metamodel to Analyze Sustainable Transport Policies for Europe: The SUMMA Project’s Fast Simple Model. In EURO 2006, 21st European Conference on Operational Research; Iceland, 2006.
- Welsh, W.D.; Vaze, J.; Dutta, D.; Rassam, D.; Rahman, J.M.; Jolly, I.D.; Wallbrink, P.; Podger, G.M.; Bethune, M.; et al. An integrated modelling framework for regulated river systems. Environ. Modell. Softw. 2012, 39, 81–102.
- Bankes, S.C. Exploratory modeling for policy analysis. Oper. Res. 1993, 4, 435–449.
- Bankes, S.C.; Walker, W.E.; Kwakkel, J.H. Exploratory Modeling and Analysis. In Encyclopedia of Oper. Res. and Management Science, 3rd; Gass, S., Fu, M.C., Eds.; Springer: Berlin, Germany, 2013.
- Agusdinata, D.B. Exploratory Modeling and Analysis: A Promising Method to deal with Deep Uncertainty. Ph.D. thesis, Delft University of Technology, Delft, the Netherlands, 2008.
- Kwakkel, J.H.; Pruyt, E. Exploratory modeling and analysis, an approach for model-based foresight under deep uncertainty. Technol. Forecast. Soc. 2012, doi:10.1016/j.techfore.2012.10.005.
- Hodges, J. S. Six (or so) Things you can do with a bad model. Oper. Res. 1991, 39, 355–365.
- Hodges, J.S.; Dewar, J.A. Is it You or Your Model Talking? A Framework for Model Validation; R-4114-AF/A/OSD; RAND: Santa Monica, CA, USA, 1992.
- Kwakkel, J.H.; Walker, W.E.; Marchau, V.A.W.J. From Predictive Modeling to Exploratory Modeling: How to use Non- Predictive Models for Decisionmaking under Deep Uncertainty. In Proceedings of Uncertainty and Robustness in Planning and Decision Making, Coimbra, Portugal, 15–17 April 2010.
- Bryant, B.P.; Lempert, R.J. Thinking inside the box: A participatory computer-assisted approach to scenario discovery. Technol. Forecast. Soc. 2010, 77, 34–49, doi:10.1016/j.techfore.2009.08.002.
- Kwakkel, J.H.; Auping, W.; Pruyt, E. Dynamic scenario discovery under deep uncertainty: The future of copper. Technol. Forecast. Soc. 2012, doi:10.1016/j.techfore.2012.09.012.
- Lempert, R.J.; Bryant, B.P.; Bankes, S. Comparing Algorithms for Scenario Discovery; WR-557-NSF; RAND: Santa Monica, CA, USA, 2008.
- Gerst, M.D.; Wang, P.; Borsuk, M.E. Discovering plausible energy and economic futures under global change using multidimensional scenario discovery. Environ. Model. Softw. 2012, doi:10.1016/j.envsoft.2012.09.001.
- Bankes, S.C. The Use of Complexity for Policy Exploration. In The SAGE Handbook of Complexity and Management; Allen, P., Maguire, S., McKelvey, B., Eds.; SAGE Publications Ltd.: London, UK, 2011.
- Rosenhead, J.; Elton, M.; Gupta, S.K. Robustness and optimality as criteria for strategic decisions. Oper. Res. Q. 1973, 23, 413–431, doi:10.1057/jors.1972.72.
- Kouvalis, P.; Yu, G. Robust Discrete Optimization and its Application; Kluwer Academic Publisher: Dordrecht, The Netherlands, 1997.
- Ben-Tal, A.; Nemirovski, A. Robust convex optimization. Mathematics Oper. Res. 1998, 23, 769–805.
- Ben-Tal, A.; Nemirovski, A. Robust solutions of linear programming problems contaminated with uncertain data. Math. Program. 2000, 88, 411–424, doi:10.1007/PL00011380.
- Bertsimas, D.; Sim, M. The price of robustness. Oper. Res. 2004, 52, 35–53.
- Wald, A. Statistical decision functions which minimize the maximum risk. Ann. Math. 1945, 46, 265–280, doi:10.2307/1969022.
- Savage, L.T. The theory of statistical decisions. J.A.S.A. 1951, 46, 55–67, doi:10.1080/01621459.1951.10500768.
- Simon, H.A. Rational choice and the structure of the environment. Psychol. Rev. 1956, 63, 129–138, doi:10.1037/h0042769.
- Kwakkel, J.H.; Walker, W.E.; Marchau, V.A.W.J. Assessing the efficacy of adaptive airport strategic planning: Results from computational experiments. Environ. Plann. B. 2012, 39, 533–550, doi:10.1068/b37151.
- Ben Haim, Y. Information-Gap Decision Theory: Decision Under Severe Uncertainty; Academic Press: London, UK, 2001.
- Deb, K.; Pratap, A.; Agarwal, S.; Meyarivan, T. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE T. Evol. Comput. 2002, 6, 182–197, doi:10.1109/4235.996017.
- Ben Haim, Y. Information-Gap Decision Theory: Decision Under Severe Uncertainty, 2nd ed.; Academic Press: London, UK, 2006.
- Hall, J.W.; Solomatine, D. A framework for uncertainty analysis in food risk management decisions. J.R.B.M. 2008, 6, 85–98.
- Hall, J.W.; Harvey, H. Decision Making under Severe Uncertainties for Flood Risk Management: A Case Study of Info-Gap Robustness Analysis. In Proceedings of 8th International Conference on Hydroinformatics 2009, Concepcion, Chili, 12–16 January 2009.
- Levy, J.K.; Hipel, K.W.; Kilgour, M. Using environmental indicators to quantify the robustness of policy alternatives to uncertainty. Ecol. Model. 2000, 130, 79–86, doi:10.1016/S0304-3800(00)00226-X.
- Korteling, B.; Dessai, S.; Kapelan, Z. Unsing information-gap decision theory for water resources planning under severe uncertainty. Water Resour. Manag. 2012, 27s, 1149–1172.
- Van Asselt, M.B.A.; Rotmans, J. Uncertainties in Perspective. In Perspectives on Global Change: The TARGETS Approach; Rotmans, J., de Vries, B., Eds.; Cambridge University Press: Cambridge, UK, 1997.
- Hoekstra, A.Y. Perspectives on Water: An Integrated Model-Based Exploration of the Future. Ph.D. thesis, Delft University of Technology, Delft, The Netherlands, 1998.
- Van Asselt, M.B.A.; vanʼt Klooster, S.A.; van Notten, P.W.F. Foresight in Action: Developing Policy-Oriented Scenarios; Earthscan: London, UK, 2010.
- Offermans, A. Perspectives Method: Towards Socially Robust River Management. University of Maastricht: Maastricht, The Netherlands, 2012.
- Walker, W.; Harremoës, P.; Rotmans, J.; Van der Sluijs, J.; van Asselt, M.; Janssen, P.; von Krauss, M. Defining uncertainty: A conceptual basis for uncertainty management in model-based decision support. Integr. Assessment 2003, 4, 5–17, doi:10.1076/iaij.126.96.36.19966.
- Holmbert, J.; Robert, K.H. Backcasting from Non-overlapping Sustainability Principles—a Framework for Strategic Planning. Int. J. Sustain. Dev. World Ecology 2000, 7, 293–308.
- Lindstone, H.A.; Turoff, M. The Delphi Method: Techniques and Applications; Addison-Wesley: Boston, MA, USA, 1975.
- Yzer, J.R.; Walker, W.E.; Marchau, V.A.W.J.; Kwakkel, J.H. Dynamic adaptive policies: A way to improve the cost-benefit performance of megaprojects? Environ. Plann. B.. under review
© 2013 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).