Next Article in Journal
Environmental Information—Explanatory Factors for Information Behavior
Next Article in Special Issue
T21-Ohio, a System Dynamics Approach to Policy Assessment for Sustainable Development: A Waste to Profit Case Study
Previous Article in Journal / Special Issue
System Dynamics Modeling of the Massachusetts SREC Market
Article

Participatory System Dynamics Modeling for Sustainable Environmental Management: Observations from Four Cases

School of Environmental and Public Affairs, 4505 Maryland Parkway, Las Vegas, NV 89154-4030, USA
Sustainability 2010, 2(9), 2762-2784; https://doi.org/10.3390/su2092762
Received: 30 July 2010 / Revised: 19 August 2010 / Accepted: 20 August 2010 / Published: 2 September 2010

Abstract

Sustainable environmental management requires a decision support approach that accounts for dynamic connections between social and ecological systems, integrates stakeholder deliberation with scientific analysis, incorporates diverse stakeholder knowledge, and fosters relationships among stakeholders that can accommodate changing information and changing social and environmental conditions. Participatory system dynamics modeling provides such a framework. It supports stakeholder learning about the system and the perspectives of other stakeholders, and can help build social capital among stakeholders. Four cases of participatory system dynamics modeling, which range from no to full participant involvement in model development, support the idea that greater social capital development results from greater participation in model development, but also suggest that even the simplest use of simulation models in a group fosters stakeholder learning about the system through surprise and discovery. To maximize the learning value of simulation models, it is important to allow enough time for debriefing the “aha!” moments that lead to curiosity about system behavior. To maximize social capital development, it is important to build enough time into the problem structuring and model conceptualization phases for stakeholders to articulate their mental models and examine those of other participants.
Keywords: system dynamics; participatory modeling; group model building system dynamics; participatory modeling; group model building

1. Introduction

In the thirty or so years since the concept of ‘sustainable development’ became part of environmental management discussions, many people have sought to clarify the goal of sustainability and find a path toward the goal. Instead of reaching consensus on a definition, however, we have developed a better understanding of the complexities of sustainability. Appreciation of the scientific uncertainties, multiple and often conflicting stakeholder goals and values, and interconnected environmental and social dynamics that characterize environmental issues has led to calls for management frameworks that facilitate deliberation among stakeholders as well as scientific analysis in support of decision-making [1,2].
Participatory system dynamics modeling offers a strong framework for analytic-deliberation in sustainable environmental management. It is particularly useful for facilitating participant learning about the system, promoting social learning among participants and building social capital. I use the term participatory system dynamics to include any approach for engaging stakeholders in problem analysis using system dynamics methods, including approaches such as group model building (e.g., [3,4]) and mediated modeling [5]. I consider participatory system dynamics modeling to include any use of a system dynamics model to structure group analysis of a problem, whether it is a conceptual or fully operational model, and whether or not the model users are involved in model development.
To make the case for participatory system dynamics modeling as a framework for sustainable environmental management, I first examine what is needed to support sustainability decisions and describe the characteristics of participatory system dynamics modeling that make it well suited in this context. I then present four cases in which system dynamics models were used for stakeholder involvement in environmental resource management. In all cases, participants used a simulation model for decision analysis. In two cases, participants were significantly involved in model development. The applications include water resource management, transportation-related air quality, relationships between land use and air quality, and waste management. Model use led to stakeholder learning in all cases, and to the development of social capital in at least one case. They show the importance of debriefing participant experiences for system learning and of allowing enough time for social learning in building social capital.

2. Decision Support Needs for Sustainable Environmental Management

Environmental management generally involves making decisions about some kind of human activity with the goal of affecting one or more environmental characteristics. In the broadest sense, it is a process for taking some action to move from an undesired actual or expected environmental state to a more desirable state. It includes several standard management steps: defining the problem or discrepancy between the actual and desired states, identifying potential courses of action, evaluating the effects of the different options, choosing and then implementing an alternative. Each of these presents its own challenges. Defining the problem is rarely straightforward because environmental management decisions almost always involve multiple stakeholders with different views and interests. Complex feedback relationships between human activity and environmental systems make it difficult to understand the potential consequences of decisions. Conflicts among stakeholders may obstruct implementation. Solving environmental problems can be as much about defining the problems to be solved as finding technical solutions, or negotiating between acceptable rather than optimal strategies.
A growing number of experts in environmental decision-making argue that because environmental decisions often require subjective judgment—involving tradeoffs, conflicting values, and decisions that have to be made with incomplete or uncertain information—deliberation among stakeholders should be central to environmental decisions, with scientific analysis directed by and in support of deliberation [2,6]. Stern ([6], p. 977) points out that: “… reasonable people disagree about which information is most needed to understand the choices facing them, about how best to get it, and about how to interpret the information that is available. People want accurate, objective information, but for many policy problems, science alone cannot resolve factual disputes among reasonable and well-informed people.” There is rarely an optimal solution when people assign different weight to different goals. Voinov and Gaddis [7] note that, as decision-making processes become more constrained by feasible options and time horizons, the consequences of wrong decisions will become more dramatic. They argue that “scientific activities must be reinforced with local knowledge and iterative participatory interactions to derive solutions that are well-understood, politically feasible, and scientifically sound ([7], p. 198).” Beierle and Cayford ([1], p. 75) further argue for making participation central to decision-making: “Rather than seeing policy decisions as fundamentally technical with some need for public input, we should see many more decisions as fundamentally public with the need for some technical input.” When participation is integrated well with technical information, education, and analysis, deliberation and technical analysis can reinforce each other.
Stakeholder participation is promoted on normative, substantive and instrumental grounds [1,8,9]. Normative arguments promote participation as a democratic right, substantive rationales say that public participation improves the quality of decisions, and instrumental or pragmatic justifications argue that participation increases the likelihood that decisions can be implemented. Participation is thought to increase the legitimacy of decisions, reduce conflict among stakeholders, and build trust and long-term, ongoing relationships among participants that improve the social context of future decisions. Recent reviews of the history and experience with public participation in environmental decision-making support these claims, but caution that outcomes depend greatly on the process [9,10]. Dietz and Stern [9] recommend that to best enhance the quality and legitimacy of decisions, and build the capacity of participants, processes should be inclusive, integrative, and include both deliberation and analysis. Specifically, participatory processes should ensure transparency of information and analysis, pay attention to facts and values, make assumptions and uncertainties explicit, engage in collaborative inquiry with stakeholders, and allow iteration to accommodate new information ([9], p. 3).
Questions of sustainability, with an implied long-term planning horizon, amplify all the challenges of environmental management. They fall into the category of “messy” [11] or “wicked” [1] managerial problems, in which there are large differences of opinion on the problem or even on whether there is a problem, where there is no clear solution or perhaps no solution at all. Costanza and Patten [12] argue that sustainability can only be assessed after the fact, that is, we can only see if something has persisted or survived at some point in the future. They suggest we think, therefore, of sustainability as a problem of prediction rather than definition. In managing for sustainability, the issue is to predict which approaches are likely to result in the persistence or survivability of the thing in question. They also note, however, that this approach also requires that decisionmakers clarify what they want to sustain and for how long. These are questions of value and have to be answered by stakeholders.
In addition to a way to structure messy problems and cope with uncertainty, some believe moving toward sustainable management will require fundamental shifts in our views of how the world works, better appreciation of the effects of individual and collective behavior, and development of stronger social capital. Farley and Costanza [13] and others note that we have to live within the carrying capacity of a finite planet, and that carrying capacity changes. They propose that rapid technological change, population growth and consumption mean that moving toward sustainability will require a shift in worldview from a world that is predictable, with abundant resources, to one that is ecological complex, indeterminate and not so predictable [13]. Beddoe et al. [14] argue we need to move from the current “empty” world culture to one that recognizes the world is “full.” The current worldview, institutions, and technology is based on an “empty world” perspective, developed from a time when the world was relatively empty of human beings and resources were abundant relative to human demands. This fostered a consumer economy. Now we live in a world relatively “full” with humans, where resources are scarce relative to human demands. With greater human demands on resources, we need to evolve to a “full world” culture.
Strategies for sustainable living will require changes in individual behavior, in addition to public support for government policies with a long-term view [15]. For example, as Sterman ([16], p. 533) notes:
“.. [R]educing GHG emissions requires billions of individuals to cut their carbon footprints by, e.g., buying efficient vehicles, insulating their homes, using public transit, and, crucially, supporting legislation implementing emissions abatement policies. Changes in people’s views and votes create the political support elected leaders require to act on the science. Changes in buying behavior create incentives for businesses to transform their products and operations. The public cannot be ignored.”
It is also clear that because of the dynamic interconnections between ecological processes and human activities, and uncertainties about those connections, management must be adaptive. Environmental management over a long time frame has to adapt to changing environmental conditions, evolving stakeholder goals, and fluid scientific knowledge. Thus the emphasis in sustainable development is shifting from the drive to “know more” about the system to building adaptive capacities that integrate stakeholder knowledge across sectors, and are able to respond concretely to the changing dynamics of social-ecological systems [17,18]. Dietz et al. [19] argue that sustainable management requires adaptive governance, in which management rules evolve as our understanding changes and biophysical and social systems themselves change. Some argue that resource management depends as much on social capital—bonds of trust, reciprocity, and social connections and social learning among stakeholders—as scientific knowledge [10,17,20].
In sum, sustainable environmental management requires “… transparent decision-making that is flexible to changing circumstances, and embraces a diversity of knowledge and value ([10], p. 2418).” It requires an analytic-deliberation framework that can engage stakeholders in discussions about their goals, integrate diverse stakeholder knowledge, perspectives, and interests, make assumptions and uncertainties transparent, facilitate social learning and learning about the system, enable scientific analysis driven by stakeholders, and adapt to changing knowledge and objectives. Social capital, and participatory processes that promote it, are particularly important for questions of sustainability.

3. Participatory Modeling for Analytic-Deliberation

Participatory modeling is an approach for including a broad group of stakeholders in the process of formal decision analysis. It generally includes the development or use of a computer model, although some “soft” approaches, such as cognitive mapping, do not (e.g., [21]). Voinov and Gaddis ([7], p. 198) describe participatory modeling as “… the process of incorporating stakeholders, often including the public, and decision makers into an otherwise purely analytic modeling process to support decisions involving complex environmental questions.” It is used to engage non-scientists in the scientific process. Stakeholders can be involved to a greater or lesser degree in the process. A fully participatory process would be one in which participants help structure the problem, describe the system, create an operational computer model of the system, use the model to identify and test policy interventions, and choose one or more solutions based on the model analysis. A minimally participatory modeling process might be one in which a model is used to help stakeholders understand the basis for an already selected decision.
Including stakeholders in the development process helps them understand a system’s interactions and behavior and can help make environmental management relevant to local concerns. It provides a mechanism for integrating scientific knowledge with local knowledge and building a shared representation of the problem. Involving stakeholders in setting goals, making sure model assumptions are appropriate, and developing politically feasible solution scenarios builds trust among stakeholders, a key factor in social learning [22]. Simulation models also allow experimentation, which helps users learn about the system and its connections.

4. Potential for Participatory System Dynamics

Participatory system dynamics modeling is the use of a system dynamics perspective in which stakeholders or clients participate to some degree in different stages of the process, including problem definition, system description, identification of policy levers, model development and/or policy analysis. Participatory system dynamics modeling is more than simply eliciting knowledge from clients about the problem and the system. It involves building shared ownership of the analysis, problem, system description, and solutions or a shared understanding of the tradeoffs among different decisions.
The goal of a system dynamics approach is to understand how a dynamic pattern of behavior is generated by a system and to find leverage points within the system structure that have the potential to change the problematic trend to a more desirable one. The key steps in a system dynamics approach are identifying one or more trends that characterize the problem, describing the structure of the system generating the behavior, and finding and testing leverage points in the system to change the problematic behavior. System dynamics is an appropriate modeling approach for sustainability questions because of the long-term perspective and feedback dynamics inherent in such questions. One of the key benefits of participatory system dynamics modeling is participant learning about system connections and feedback, both about the system and about other participants.
System dynamics modeling has been used in a number of applications with stakeholder groups examining difficult environmental issues (e.g., [5,21,23,24,25]). Most recently, Beall and Ford [25] compared nine cases that used a variety of techniques for engaging stakeholders and problem-solving. These case analyses illustrate the range of issues and settings in which participatory system dynamics modeling can be used and represent the beginnings of efforts to develop best principles and practices.

5. Four Cases of Participatory System Dynamics Modeling

The following cases further illustrate the use of participatory system dynamics modeling for environmental management. All used system dynamics simulation models to facilitate stakeholder participation. The cases represent varying degrees of participant involvement in model development and decision-making, and different objectives for stakeholder participation. In the first two cases, participants were not at all involved in model development. They used the simulation model to learn about the system and to experiment with the effects of different policy decisions on the problem variables. Decision-makers were simply interested in helping participants understand the issues and decision choices. In the second case, the decision-makers were seeking participant input on potential policies. Participants in the third and fourth cases were significantly involved in model development, model analysis, and developing policy recommendations. Details about the cases, models and policy results are reported elsewhere [26,27,28,29]. The following section summarizes the aspects of each case relevant to discussing how the approach is suitable for sustainable management. Table 1 gives an overview of the cases across a set of context, process, and outcome variables. The user interfaces for the models used in each case are shown in Figure 1, Figure 2, Figure 3, and Figure 4. In all cases, the model interface was limited to one screen.
Table 1. Summary of Participation Cases.
Table 1. Summary of Participation Cases.
1. Water Supply and Demand2. Municipal Solid Waste Management3. Transportation and Air Quality4. Land Use, Transportation and Air Quality (LUTAQ)
PROBLEM CONTEXT
IssueKeep water supply above demand into the near futureReduce waste sent to landfill to zeroReduce traffic congestion and improve transportation-related air qualityMinimize growth-related traffic and air quality problems
Planning horizon50+ years30+ years25 years35 years
PARTICIPATORY PROCESS
Purpose of processCommunication from agency to public, individual learning about the systemCommunication to public, input from public, individual learning about system, feedback to agency on policiesStakeholder policy recommendations to agencyParticipant policy recommendations and integrated framework for growth management
Process descriptionOne-time, 2–3 hour workshopOne-time 3 hour workshopOne year process with monthly 2-hour meetingsTwo year process with monthly 2-hour meetings
Participant numbers and types5–30 people/workshop, in multiple, independent workshops, general public100 people, general public30 people, representatives of stakeholder groups chosen by lead agency, including business leaders, elected officials, environmental groups, transit riders20 people, agency staff including land use planners, air quality modelers, transportation planners
Participant involvement in model developmentNoneNoneMediumHigh
Focus of participationModel use for policy analysisModel use for policy analysis and recommendations to agencySub-group: model development Full group: problem structuring, model use for analysis and recommendationsProblem structuring, model development, model use for analysis and recommendations
Group size for model use2–3 people per computer, self-directed2–4 people per computer, self-directed8–10 people per computer, facilitator-led1–4 people per computer, self-directed
MODEL DETAILS
# decision variables in model691112
# decision criteria in model2644
OUTCOME
Effect of process with respect to intended outcomeIndividual insightHigh quality recommendations, but individual frustrationAgreement on a set of policy recommendations.Agreement on policy recommendations, social learning
Unexpected effects of processgroup communication, high level of engagement in discussionInterest in model structureIndividual insight, shared language, communicationIndividual insight, shared vision, mental model alignment
Figure 1. Water Model User Interface.
Figure 1. Water Model User Interface.
Sustainability 02 02762 g001
Figure 2. Waste Management Model Interface.
Figure 2. Waste Management Model Interface.
Sustainability 02 02762 g002
Figure 3. Transportation and Air Quality Model Interface.
Figure 3. Transportation and Air Quality Model Interface.
Sustainability 02 02762 g003
Figure 4. Land Use, Transportation and Air Quality (LUTAQ).
Figure 4. Land Use, Transportation and Air Quality (LUTAQ).
Sustainability 02 02762 g004

5.1. Water Management, Las Vegas, Nevada [26]

In 2000, Las Vegas, Nevada was one of the fastest growing metropolitan areas in the U.S., located in one of the most arid climates. The population had been increasing at almost 6% per year for over a decade with no sign that growth was slowing. Most (about 88% in 2000) of the water supply for Las Vegas comes from the Colorado River. The rest comes from groundwater. In 2000, per capita water demand was among the highest in the U.S. Many homes and businesses had grass lawns, and about 40% of the water used in the area was used for outdoor irrigation. At the time there was very little incentive for water conservation, because water rates were relatively low and the water authority was focusing its efforts on finding new sources rather than on conservation. In the last five years, the water authority has shifted its emphasis and now aggressively promotes conservation with rebates and advertising campaigns.
The U.S. Bureau of Reclamation, which is very involved in water resource issues in the region, funded the development of a system dynamics simulation model for examining tradeoffs between water management options for Las Vegas. The intended use for the model was to educate stakeholders about water management issues in the region, particularly about the value of water conservation. The model was developed in consultation with experts from the Bureau of Reclamation and the water authority. The problem definition was based on projections showing that if population growth continued at the same rate and no new sources were introduced, projected demand would begin to exceed supply around the year 2025. The underlying causal structure of the model was also developed in consultation with the system experts. The model was built by the author’s team and verified with the client. The model is relatively small and simple. The stocks and flows follow the water system closely. The model gives users two options for increasing supply and five options for reducing demand. The effects of policies on water supply and demand in the system are not straightforward because of the structure of the system. Although the feedback loops governing water supply and water demand are easy to understand by themselves, when combined, they generate the somewhat counterintuitive result that reducing residential outdoor water use has a much greater effect on water demand than reducing indoor water use by the same amount.
The model has been used many times in small workshops of 5–30 participants since it was first developed. The purpose of the workshops is to introduce participants, generally Las Vegas area residents and visitors, to regional water issues and stimulate discussion about potential water management strategies. A typical workshop runs for about two hours. Participants are given a brief introduction to the problem, divided into groups of 2–3, and asked to come up with suggestions about how to extend the time at which demand would outstrip supply—the “crossing point” of demand and supply. We let participants experiment with the model for about 45 minutes, then facilitate a whole-group discussion. This process represents the most basic use of participatory modeling, in which participants analyze policy options using the model, but were not part of the model development. While participants have no direct authority for water management policies, they may be asked to vote on bond issues or other referenda promoting specific policies in the future. In addition, some management strategies require changes that can be made on an individual level to affect indoor and outdoor water use, including changes in water use habits, technology, or landscape design.
In every session, regardless of participant familiarity with the topic, with other participants, or with computer simulation models, participants become engaged in the discussion very quickly. The noise level in the room rises almost immediately as they propose policies to each other and explain their reasoning, then again when they discuss the model results. The debriefing discussion is always lively, with many participants expressing surprise about model results and asking questions about why they got those results. Debriefing discussions always lead to an exploration of which policies would be feasible, and how they might be implemented.

5.2. Zero Waste Initiative, Los Angeles, California [27]

The City of Los Angeles, CA is one of many cities around the world that have undertaken a Zero Waste Initiative. The city’s goal is to reduce the amount of municipal solid waste sent to landfills to zero by 2030. In 2007, 62 percent of the waste generated by residents and businesses was diverted from landfills. To look for ways to increase diversions to 100 percent, the city conducted an intensive strategic planning and stakeholder involvement process. The first year of this Solid Waste Integrated Resource Planning (SWIRP) process was focused on stakeholder involvement. Between July 2007 and May 2008, the city held regional and citywide workshops to solicit stakeholder input on guiding principles for the strategic plan.
Representatives of the city and the group running the stakeholder involvement process contracted the development of a simulation model for engaging the public in the planning process at a citywide conference. They saw the model as different way to get participants excited about the goal of zero waste, help them understand the strategic options for achieving it, and elicit their input about the different options. The main purpose was to demonstrate the benefits and drawbacks of strategic choices over a 20-year period, help stakeholders understand uncertainty in the system, and illustrate the effect of taking no action. The model was to be a simplified, strategic-level model sufficient to allow non-expert stakeholders to compare the relative effects of major policy, program and facility decisions on critical performance measures. It was to include key policy levers that system managers might realistically use.
The model was developed in consultation with city representatives running the planning process. These clients defined the problem and the initial causal structure. The model structure is based on a “recycling loop” connecting two “upstream” sectors (product manufacturing and consumer consumption) and three “downstream” sectors (consumer disposal, collection of discarded material, processing, and disposal). Virgin material enters the stock-and-flow structure through the production, or manufacturing process, is transferred to consumers based on consumer demand, and, if it is not fully consumed, continues through the system when it is discarded by consumers. Users can choose from eight decision levers, representing the city’s main strategic options (increase product durability, decrease waste in products and packaging, increase recycled content of products, increase product recyclability, decrease consumption, increase consumer diversion rates, increase diversion processing capacity, increase alternative disposal capacity). Model output shows six measures (waste sent to landfill, material diverted, diversion rate, relative greenhouse gas emissions, relative cost, and relative effort).
Model analysis reveals several key points about the waste system. First, without any action, zero waste is not possible. In fact, taking no action will erode the current 62% diversion rate of which Los Angeles is justifiably proud. Second, zero waste is possible by several means, but there are significant costs and tradeoffs in environmental impact and political/social effort required. Third, reducing greenhouse gases in any significant way requires reducing the material that enters the waste system. Because the largest amount of greenhouse gas emissions in this system is generated by virgin materials in the production of goods, any policy lever that reduces the amount of virgin materials used reduces greenhouse gas emissions. These include increasing product durability, increasing the recycled content of products, and reducing consumption. Even small changes in these parameters can have marked effects on greenhouse gas emissions.
The model was used at one public conference by half of the conference attendees, approximately one hundred people. (The other half participated in facilitated small-group discussions without a simulation model. A comparison of the experiences of the two groups will be reported in a forthcoming paper by Stave and Turner.) All conference attendees were given an overview of the waste system, then presented with the eight main strategic options the city has for achieving zero waste. They were asked for their input on how much emphasis the city should give each option. After the introduction, participants were divided into two groups located in two large rooms. Each large group was further divided into small groups for discussion. The group using the simulation model was divided into 26 groups of 3–5 people. Each group was provided with a laptop computer to run the model. There was one facilitator for every two groups helping the groups use the model. The small groups had approximately one hour to use the model, after which they spent about 30 minutes reporting their recommendations to the large group. Although groups commented on their experiences in their brief reports to the large group, there was no time for debriefing the modeling exercise as a whole.
Like participants in the water management workshops, participants in this process were not involved in model development. They used the model to analyze policy options. However, some of the participants had been involved in earlier scoping meetings or a previous informational conference. They also had no direct decision-making authority, but in this case, their input was specifically requested by the decision-makers. They were not a formal advisory committee, but their input was seen as important for gaining future public support for policy decisions. Also, similar to the water model, some management strategies require changes that can be made on an individual level, in this case, recycling habits, consumption patterns, and purchasing behavior.
Overall, participants who used the simulation model expressed frustration that they did not have more time to work with the model, or more information about what was in the model. However, the recommendations they made to the city were qualitatively more effective for achieving the city’s zero waste objectives than the group that did not use the simulation model. The detailed results of the research [30] support the idea that use of the simulation model in a small group, even under difficult time constraints, helped people understand the complex connections in the system.

5.3. Transportation-Related Air Quality, Las Vegas, Nevada [28]

From 1990 to 2000, traffic congestion in the Las Vegas metropolitan area increased steadily with population growth, from a system-wide volume/capacity average of approximately 0.45 in 1990 to over 0.55 in 2000 (where 0.5 represents “free-flow” conditions). The regional transportation agency projected that if congestion continued to follow that trend, volume/capacity would exceed 0.8 by 2025. Congestion in some places was already reaching 1.0 periodically (a “full road” condition where traffic volume equals road capacity). Costs of traffic congestion, including the extra cost of fuel and the cost of increased travel time, were estimated to be $0.5 billion per year. At the same time, air quality in the Las Vegas Valley also worsened, largely due to automobile emissions. Degraded air quality contributes to respiratory health problems in the region. It also feeds back to pose a further problem for transportation because of the connection between compliance with federal air quality standards and federal funding for transportation projects. If air quality does not meet federal standards, the Las Vegas region could lose $80 million in transportation funding annually.
An ad hoc stakeholder advisory group was convened to develop policy recommendations to address these interconnected problems of traffic congestion and regional air quality. The group consisted of 30 community stakeholders including elected officials representing the county and cities within the county, and representatives of the business community and tourism industry, environmental groups, bus riders, other public agencies, and community residents. These stakeholders had no particular knowledge of the transportation system other than their observations as system users. The group was to meet once per month for a year and was asked to make recommendations to the Regional Transportation Commission at the end of the year about how it should address the transportation problem in the region.
Four months after the group started meeting, the author’s group began working with the group to develop a simulation model to support the advisory process. The two goals of model development were to help advisory group members develop a general understanding of the issues and identify and compare general options for solving the problem. We developed the initial problem statement and causal map with the full group of 30 participants in their fourth month of meeting. A smaller workgroup of five people worked with us separately for the next four months to refine the model structure and test the operational model. During those months, the main group received informational presentations about the transportation system. We used the final model with the full group in their ninth and tenth months of meeting. Model simulations were run in two workgroup meetings and two full group meetings.
The group identified three trends describing the problem: the rising system-wide traffic congestion noted and projected by the Regional Transportation Commission, decreasing traffic flow as measured by average system-wide traffic speed, and increasing frequency with which carbon monoxide emissions from cars exceed the region’s federally-determined carbon monoxide budget. They also included cost as a decision criterion. Similar to the water management model above, the transportation and air quality model gave users several options for increasing the supply of transportation capacity, and several options for reducing road transportation demand. Traffic reduction options included alternative means of transportation and carpooling. The model workgroup developed and tested several scenarios before presenting the model to the full group. When first presented to the full group, the scenarios generated a lively discussion that included questions of clarification about the model and system (costs of system improvements, maximum possible improvements, and connections between transportation and carbon monoxide production, for example). Full group members who were seeing the model for the first time wanted to know what the most effective driver was for improving air quality with the least cost. Instead of proposing different scenarios and running the model to test the effects, they wanted the model to give them the optimal answer.
The workgroup used the full group’s questions to test more policies. They tried “maxing out” different policy levers to compare relative consequences and experimented with different combinations. The workgroup took their findings and further simulation runs back to the full group. Full group members tried a few runs to reduce costs, but in the end used a package the workgroup had developed using the model as their policy recommendations.
This case represents a relatively high degree of participant involvement in model development. The full group was engaged in deliberations about problem definition and system structure. The smaller workgroup was fully engaged in the model specification, including identifying parameter data and developing or verifying model equations, testing the model, and conducting initial policy analyses. The full group further tested the model, and used it for policy analysis. In this case, participants had more decision-making authority than in the water and waste management cases. They were specifically asked to make policy recommendations to the transportation agency. While they were technically only an advisory group, the members of the group represented important constituencies whose support was being sought for a vote on how to pay for the chosen policies. On an individual level, this model also showed the value of even small changes in individual behavior. Changing the vehicle occupancy rate by even a small amount, that is, increasing the number of people who carpool, has a very large aggregate effect on the output variables.
The greatest learning benefit, and perhaps social capital development, took place in the small workgroup that helped develop the structure of the model and conducted the model analysis. This group used the completed model in two analysis sessions. The 8–12 participants in this group used one computer operated by the facilitator. After discussing the policy options, different members suggested different options. The suggestions were run on the computer and participants were asked to describe what had happened to the output criteria. By the second session, they were making comments such as: “this shows we can’t make things better; we can only keep them from getting worse”, “there is no silver bullet”, “carpooling is like water conservation—it’s a way of making better use of existing infrastructure.” Unlike the water model workshops, the model analysis workshops in this case were more sober. The participants took seriously the responsibility of making sound decisions to fix the problems, and found their initial ideas were not as effective or were more expensive than they thought. The model analysis seemed to make the discussion more thoughtful.
One of the indicators that the model use led to social capital development was that several participants who started out the process with a healthy skepticism of the value of the model exercise became ardent supporters by the end. The head of the agency that sponsored the process felt it was successful enough that he suggested it be used in further, more integrated planning in the region.

5.4. Land Use, Transportation and Air Quality (LUTAQ), Las Vegas, Nevada [29]

Two years after the previous project ended, the head of the transportation agency proposed that we extend the transportation and air quality model to examine issues of urban sprawl in the Las Vegas area. Population was continuing to increase, traffic congestion was still increasing, and residential construction was extending far out to the edges of the valley. As the economy boomed and housing prices started to rise steeply, planning officials were starting to talk about trying to change the pattern of land use by promoting “densification” of the urban core of the metropolitan area. Some people called the idea “Manhattanizing” Las Vegas. The regional planning coalition, a group made up of the elected officials from all the regional governmental entities, hired us to use a participatory modeling approach to integrate land use, transportation and air quality (LUTAQ) management efforts in the region.
Approximately 20 upper level staff from the governmental entities in the region and the air quality and transportation agencies comprised the LUTAQ working group. Using a group model building process, this group developed a computer simulation model over a two-year period. Group members clarified the model purpose and problem definition, developed the model structure, quantified relationships between variables, and provided data for model parameters. Quantification was done “behind the scenes” by the author’s team. The model was validated by the working group, technical experts in the planning, transportation, and air quality agencies, and external reviewers. When the model was complete, the entire group used the model to test a set of policy scenarios corresponding to real changes being proposed by or discussed among planners in the region. Over 24 months, the LUTAQ group met 36 times, with each meeting lasting approximately 2 hours.
LUTAQ policy analysis produced several key findings. First, it showed that maintaining the status quo would mean significant increases in traffic congestion and air pollution. Second, it showed that densification alone, at any level and in any area, makes things even worse than simply maintaining the status quo. Finally, it showed there are ways to achieve the policy goals that do not require extreme changes in land use and transportation design. As the work group noted throughout the model building and model analysis phases, land use, transportation and air quality are linked in critical ways. Changes in one part of the system cannot be made without consequences in other parts of the system. For instance, the model demonstrates that any increase in density has detrimental effects on traffic and air quality. Such increases may be necessary, however, to keep housing development economically viable as the price of land increases. Analysis showed that other factors in the system can balance the negative consequences of one factor, such as densification, to achieve an overall desirable outcome.
This case is similar to the previous case in that it represents a very high degree of participant involvement in model development. Here, the entire working group was engaged in all phases of model development except the detailed equation development and testing. This group participated fully in model testing. Model analysis was done both with the group as a whole, but members were also given copies of the model to use on their own computers outside the meetings, which about five participants did. These participants had more decision-making authority than those in the transportation case in that they were assigned to the working group by their planning directors or agency managers, and were responsible for reporting their work to those decision-makers.
This is the clearest case of social capital development of the four cases presented. One of the stated goals of the LUTAQ project was to better integrate land use, transportation, and air quality planning. The model was expected to be the means for the integration. But in this project, the process was at least as useful at improving integration as the model. The 15 or so regular participants in the group included land use planners, transportation modelers and planners, and air quality planners. In the beginning of the process, participants started out talking about the details of their parts of the system. Initial discussions of land use were focused on the subdivision level, spatial distributions of demographic and economic variables, and street networks. Transportation staff focused on the goal of reducing vehicle miles traveled and increasing mass transit ridership. At the same time, they complained that planning of large projects did not consider interrelated effects.
“At the regional level, when they are considering a “big” project (1,000 acres, for example), policy-makers never assess overall impact on traffic congestion and air quality—especially beyond the boundaries of the development—for example impact on under-designed roads that end-up being major ingress-egress to the development. The ‘elected’ people aren’t making the connection between land-use and transportation and air quality. We are trying to get our elected officials to buy into ideas like increasing density along existing major transportation corridors.”
By the end of the process, after working through causal diagrams of how their pieces fit together, the group was talking in a more unified way about the issues and their solutions. They shifted their perspective from their area of expertise to the system as a whole. They used concepts of feedback and interconnections to illustrate the points they made in discussions, and they were unified in their support for the model as a clear way to communicate the importance of the whole-system view to the elected officials and policy-makers. Group comments about the utility of the model include:
“This shows that you can’t simply do one thing, like increase residential density, and improve traffic congestion or air quality. You have to do a number of things all together.”
“Most important message is: It is only by a combination of land use and transportation that we can achieve the three goals. This is what the model proves. We must use both. If you just use density—you won’t get there.”
“Some of the elected officials might say: We always knew this. The value of the model is it (1) quantifies policy ideas and consequences, and (2) lets us try out different parameters for ‘investment’—and reveals the sensitivity of parameters. It tells you how to combine the variables to get what we want.

6. Discussion

Modelers often say that the person who develops and uses the model for analysis is the one who learns most about the system in question. In a standard modeling exercise, when a modeler is hired to examine a client’s problem and recommend a strategy or set of potential solutions, the recommended solution is the primary product. The cases discussed here, and many similar participatory modeling exercises, show that simulation modeling can have significant problem solving benefits beyond the model output and irrespective of whether or not a decision is implemented. Engaging stakeholders in model development and analysis broadens the group of people who can learn from the modeling exercise. In these four cases, the process of using or developing and using the model improved stakeholder understanding of the system structure and behavior even though only the transportation-related air quality case led to implementation of the group’s model-based recommendations. When participants were involved in model development as well as use, the process led to significant social learning and social capital development. This kind of broad-based stakeholder learning and social capital builds a foundation for long-term stakeholder engagement in sustainable environmental management.

6.1. System Learning and Paradigm Change

Even the simplest form of participatory system dynamics modeling, in which participants use a completed model for analysis and discussion, helps participants think about environmental management problems in a long-term, dynamic, interconnected context. First, representing the problem as a trend over time puts the problem in a long-term context. In all the cases discussed here, projections of the consequences of “business-as-usual” actions into the future led to discussions about the urgency of the problem and what a more desirable future condition might be. Second, the model shows a set of potential strategies for changing the problem. In addition to allowing participants to experiment with the options presented, the set of initial options usually generates discussion about how they might be implemented and whether other strategies are possible. Third, experimentation almost always leads to surprising results that make participants curious about the system generating the results. As Barry Richmond [30] points out, this learner-directed approach helps model users actively construct knowledge instead of simply assimilating it. Sometimes participants feel they do not know as much as others do about the system and may not participate in discussions. The model lets users test their assumptions about the effects of policy change privately, without fear of exposing any perceived ignorance, and encourages them to raise questions about the system. Model use leads to better understanding of why one option yields a better outcome than another. In the cases described above, model results that were not what users expected led to rich discussion and further questions about why the model, and the real system, might behave that way. Even in the waste management case, when participants were frustrated with the process because the time for experimentation and debriefing was short, their frustration led to questions about the system structure and other potential policy options.
The model can also provide a neutral platform for the evaluation of contentious policies. When stakeholders differ on what aspects of the issue are most problematic (such as cost, environmental pollution, convenience, or difficulty of implementation), or which solution is the most desirable, discussions can become mired in interpersonal conflict. Particular suggestions or positions can be resisted or rejected because of who suggests them. Parkins and Mitchell [31] note that successful deliberation requires internal inclusion (making sure the people at the table all have equal weight in discussions) as well as external inclusion (getting the right people to the table). Having the model “tell” the group the likely consequences of a particular decision can be more powerful than having a person tell the group. In the transportation-related air quality case, for example, the model showed an adverse air quality effect of a particular decision, leading the group to revise its recommendation. After the discussion, an environmental group representative said he felt the others in the group had been more receptive to the result because the model showed it than they would have been if he had tried to convince them of the same thing. Learning about a system, particularly through surprise, is the first step in changing one’s mental model or paradigm of how something works. Surprise is a reaction to a result that is not what would be predicted by one’s mental model. Surprise could be considered, in Thomas Kuhn’s [32] terms, an unexplained anomaly that challenges one’s existing mental model or paradigm.

6.2. Specific Surprises: Small Changes Made by Many People Can Have Big Payoffs

In all four cases, a key component of the most effective resource management strategies was to make the most of existing resources and infrastructure. The models all show that conserving existing resources before seeking new ones yields the best return on investment. In the water model, reducing water use by just a small amount per person in the right place takes minimal effort for a large effect. In the Zero Waste model, small changes in product durability, recycled content and consumption have greater effects on waste reduction and greenhouse gas emissions than incentives to increase recycling. In the transportation and air quality system, increasing the carpool rate by just a small amount reduces air pollution and traffic congestion with very little cost. The LUTAQ model analysis shows that small changes in distance traveled per trip and number of trips per day (factors that can be influenced by residential design) reduce traffic congestion and air pollution significantly.
These are commonsense results, but are not necessarily obvious. For all these cases, the first solutions participants suggested were top-down, large-scale, or supply-oriented approaches such as finding new sources of water or building more roads. The small-scale, widely distributed, demand management solutions that had the same or greater effect for less overall cost were initially dismissed. Using the models allowed the users in all these cases to discover the relative value of these kinds of efforts themselves. The model allowed them to identify solutions that might have been different from what they originally thought would have been required [7]. In this way, such model-supported participatory exercises have the potential to lead to stakeholder buy-in to individual lifestyle or paradigm changes that may be needed for sustainable environmental management.

6.3. Social Learning and Social Capital Development

Social learning is learning about different points of view and types of knowledge of others in the system. It is promoted by opportunities to examine assumptions and cultural frameworks that are taken for granted, recognition of interdependencies, and examination of cause and effect relationships [22]. People do not simply process information. Rather, they filter information, making selections and adding interpretation [11]. Involving stakeholders more deeply in model development provides a framework for describing their mental models and allowing them to see how others understand the system. Social learning takes place through deliberation about problem structuring and system conceptualization.
Social capital describes the social bonds and norms that enable and regulate the interactions of people in communities. Two key features of social capital that promote cooperation in resource management are relations of trust and social connections [20]. Pretty [20] describes three key types of connections as bonding capital (linking people within communities with similar objectives), bridging capital (building the capacity of people to link with others that may have different views), and linking capital (building the capacity to engage with others beyond communities). Social capital development is based on social learning. Participatory system dynamics modeling builds bridging and bonding capital as stakeholders learn how others see the problem and the system in the problem structuring and causal mapping phases, and how different policy option affect the interests of other participants in the model analysis phase.

7. Observations toward the Development of Best Practices

Experiences with these four cases highlight several general principles. To best promote system learning, it is important to cultivate surprise, facilitate discovery, and allow enough time for debriefing. To support social learning and social capital development, it is important to allow enough time in the process for participants to get to know each other. While straightforward, these commonsense principles are sometimes difficult to put into practice. When time is limited in a model use workshop, it is tempting to allow more time for an introduction or for users to experiment with the model. Since the debriefing session comes last, running over time in the early phases can mean the debriefing is cut short. In our experience, however, some of the best learning takes place during the group discussions after participants have spent time with the model individually or in small groups. Over repeated use of the water model, we have gradually decreased the introduction and model use and expanded the debriefing session. We encourage participants to describe their reactions and discoveries and guide them to probe surprising results by asking why they think they got those results. Doing this in a group debriefing session helps participants make sense of individual surprises and learn from the experiences of others. Debriefing discussions are most lively when we encourage the group to respond to each other’s questions. In the zero waste workshop, the recommendations from the group that used the model were systemically better, indicating that they understood from the model which options generated output that best met the decision criteria. However, the lack of debriefing limited the connection they made between why one approach was better than another. The model raised questions that would have led to lively discussion if we had had more time, but participant survey responses after the workshop showed they were frustrated with the experience.
Good facilitation is important for achieving the best outcome in any participatory process [10]. Although self-directed model use can lead to user learning, facilitated debriefing can help participants get more from the exercise than they would have alone. Vennix [11] suggests the facilitator has an important role in fostering reflection and learning among group members. The facilitator’s role is to ask questions, rather than provide answers, to be curious about how people perceive and interpret situations.
In all the cases described here, model use led to surprise. Using models helped people see the aggregate effects of small, individual changes in behavior such as water use, carpooling, length of car trips, number of trips, and purchasing decisions about durability of goods consumed. The models showed long-term and indirect effects of policy options, and demonstrated the economic value of avoided costs. Model output challenged the mental models participants had about what the “best” solutions were, and disconnected the message from the messenger, making participants more receptive to ideas that they might otherwise have rejected. Participant learning about the system seemed to be greatest when people discovered things through their own use of the model, in small groups of 2–3 people, rather than alone or in a very large group. When they used the model by themselves, they reported that the model confirmed what they already knew. When the model was used in very large groups in which participants did not know each other well, few of the group members were engaged in model experimentation and discussion. Members of the two-to-three-person small groups using the water model tend to discuss the strategies they want to try in depth while running the model. Members of the slightly larger LUTAQ and transportation groups knew each other well by the time they ran the model and contributed equally to the model runs and discussions.
To create the best opportunities for surprise and discovery, and for learning from surprise, we have found it important to guide model users to consider what they expect to happen before they run the model. Model users are often inclined to either want to play with the model by entering relatively random decision values (if they are experienced computer users or gamers), or being very tentative about trying anything at all (if they are not comfortable with the technology). We have found it most useful to begin a model use session in a large group. We restate the problematic behavior and our goal (to extend the point at which water demand exceeds supply further into the future; to reduce waste send to the landfill to zero by 2030; to reduce traffic congestion and carbon monoxide while keeping costs low, for example). We ask for suggestions for how to achieve the goal. Just before running the model with the decision variables suggested, we ask what they think will happen. After running the model, we compare the model results with the stated expectation. If it is different, we ask why they think it was different. After a few runs in the large group, when participants seem to be getting comfortable with the way the model works, we divide them into small groups of 2–5 people. In the water and waste cases, we provided worksheets that prompted the group for a set of decision values, and the result they expected. We asked them to draw the expected graph, or describe the expected results in words. After they ran the model, we asked them to note the result and discuss or describe in writing whether it was different than their expectations and why. We have found that people often re-write their expectations when results are different from their expectations. We try to find ways to encourage them to reflect on how their thinking is being affected by the model runs that do not embarrass them. This seems to work best when the “Aha!” moment is either private or collective, when it comes from their personal or small-group reflection, or when the entire group clearly shares the surprise.
Finally, social learning and social capital development take time. It takes time for participants to learn about each other’s perspectives and build trust. In the two cases where participants were involved in model development, the problem definition phase took more than one-third of the total time spent on the process. To be most successful at promoting social learning and building social capital, participants need enough time to frame the problem, explore differences in stakeholder perspectives, identify and discuss assumptions, develop causal maps, and identify and examine relevant data. It can sometimes be difficult for the lead decision-making agency to allocate enough time to the problem structuring phase of participatory modeling, because from the agency’s perspective, the problem is generally well-defined.

8. Conclusions

The four cases described here span the participation continuum from communication, to consultation, to full stakeholder participation in decision-making. All of these cases engaged the participants in analytic-deliberation. Two of the cases, the water model and the waste management model, were used for communication. The other two were used for consultation and decision-making. In all cases, engaging stakeholders in model analysis provided the basis for group deliberation about different options. It focused participant questions on the problem at hand, the information provided, and policy options, leading to rich discussions during and after the model was used. The cases where stakeholders were more deeply involved in model development show the potential for social learning and social capital development.

References

  1. Beierle, T.C.; Cayford, J. Democracy in Practice: Public Participation in Environmental Decisions; Resources for the Future: Washington, DC, USA, 2002. [Google Scholar]
  2. Stern, P.C. Deliberative Methods for Understanding Environmental Systems. BioScience 2005, 55, 966–982. [Google Scholar]
  3. Vennix, J.A.M. Group Model Building: Facilitating Team Learning Using System Dynamics; John Wiley & Sons: New York, NY, USA, 1996. [Google Scholar]
  4. Anderson, D.F.; Richardson, G.P. Scripts for Group Model Building. Sys. Dyn. Rev. 1997, 13, 107–129. [Google Scholar]
  5. van den Belt, M. Mediated Modeling: A System Dynamics Approach to Environmental Consensus Building; Island Press: Washington, DC, USA, 2004. [Google Scholar]
  6. Chilvers, J. Deliberating Competence: Theoretical and Practitioner Perspectives on Effective Participatory Appraisal Practice. Sci. Technol. Hum. Values 2008, 33, 155–185. [Google Scholar]
  7. Voinov, A.; Gaddis, E.J.B. Lessons for successful participatory watershed modeling: A perspective from modeling practitioners. Ecol. Model. 2008, 216, 197–207. [Google Scholar]
  8. Fiorino, D.J. Citizen Participation and Environmental Risk: A Survey of Institutional Mechanisms. Sci. Technol. Hum. Values 1990, 15, 226–243. [Google Scholar]
  9. Dietz, T.; Stern, P.C. Public Participation in Environmental Assessment and Decision Making; National Research Council: Washington, DC, USA, 2008. [Google Scholar]
  10. Reed, M.S. Stakeholder participation for environmental management: A literature review. Biol. Conserv. 2008, 141, 2417–2431. [Google Scholar]
  11. Vennix, J.A.M. Group model-building: Tackling messy problems. Sys. Dyn. Rev. 1999, 15, 379–401. [Google Scholar]
  12. Costanza, R.; Patten, B.C. Defining and predicting sustainability. Ecol. Econ. 2008, 15, 193–196. [Google Scholar]
  13. Farley, J.; Costanza, R. Envisioning shared goals for humanity: A detailed, shared vision of a sustainable and desirable USA in 2100. Ecol. Econ. 2002, 43, 245–259. [Google Scholar]
  14. Beddoe, R.; Costanza, R.; Farley, J.; Garza, E.; Kent, J.; Kubiszewski, I.; Martinez, L.; McCowen, T.; Murphy, K.; Myers, N.; Ogden, Z.; Stapleton, K.; Woodward, J. Overcoming systemic roadblocks to sustainability: The evolutionary redesign of worldviews, institutions, and technologies. PNAS 2009, 106, 2483–2489. [Google Scholar]
  15. O’Riordan, T. Environmental science, sustainability and politics. Trans. Inst. Br. Geogr. 2004, 29, 234–247. [Google Scholar]
  16. Sterman, J. Risk Communication on Climate: Mental Models and Mass Balance. Science 2008, 322, 532–533. [Google Scholar]
  17. Pahl-Wostl, C.; Mostert, E.; Tàbara, D. The Growing Importance of Social Learning in Water Resources Management and Sustainability Science. Ecol. Soc. 2008, 13, 24. Available online: http://www.ecologyandsociety.org/vol13/iss1/art24/ (accessed on 17 April 2010). [Google Scholar]
  18. Thabrew, L.; Wiek, A.; Reis, R. Environmental decision making in multi-stakeholder contexts: Applicability of life cycle thinking in development planning and implementation. J. Cleaner Prod. 2009, 17, 67–76. [Google Scholar]
  19. Dietz, T.; Ostrom, E.; Stern, P.C. The Struggle to Govern the Commons. Science 2003, 302, 1907–1912. [Google Scholar]
  20. Pretty, J. Social Capital and the Collective Management of Resources. Science 2003, 302, 1912–1914. [Google Scholar]
  21. Mendoza, G.A.; Prabhu, R. Participatory modeling and analysis for sustainable forest management: Overview of soft system dynamics models and applications. Forest Pol. Econ. 2006, 9, 179–196. [Google Scholar]
  22. Tàbara, D.; Pahl-Wostl, C. Sustainability Learning in Natural Resource Use and Management. Ecol Soc 2007, 12, 3. Available online: http://www.ecologyandsociety.org/vol12/iss2/art12/ (accessed on 17 April 2010). [Google Scholar]
  23. Langsdale, S.M.; Beall, A.; Carmichael, J.; Cohen, S.J.; Forster, C.B.; Neale, T. Exploring the Implications of Climate Change on Water Resources through Participatory Modeling: Case Study of the Okangan Basin, British Columbia. J. Water Res. Pl-ASCE 2009, 135, 373–381. [Google Scholar]
  24. Beall, A.; Zeoli, L. Participatory modeling of endangered wildlife systems: Simulating the sage-grouse and land use in Central Washington. Ecol. Econ. 2008, 68, 24–33. [Google Scholar]
  25. Beall, A.; Ford, A. Reports from the Field: Assessing the Art and Science of Participatory Environmental Modeling. Int. J. Sys. Soc. Change 2010, 1, 72–89. [Google Scholar]
  26. Stave, K.A. A System Dynamics Model to Facilitate Public Understanding of Water Management Options in Las Vegas, Nevada. J. Environ. Manage 2003, 67, 303–313. [Google Scholar]
  27. Stave, K.A. Zero Waste by 2030: A system dynamics simulation tool for stakeholder involvement in Los Angeles’ solid waste planning initiative. In Proceedings of the 26th International Conference of the System Dynamics Society, Athens, Greece, 20–24 July 2008; Available online: http://www.systemdynamics.org/conferences/2008/proceed/papers/STAVE416.pdf (accessed on 20 July 2010).
  28. Stave, K.A. Using system dynamics to improve public participation in environmental decisions. Sys. Dyn. Rev. 2002, 18, 139–167. [Google Scholar]
  29. Stave, K.A.; Dwyer, M. Lessons from LUTAQ: Building systems thinking capacity into land use, transportation, and air quality planning in Las Vegas, Nevada. In Proceedings of the 24th International Conference of the System Dynamics Society, Nijmegen, The Netherlands, 23–27 July 2006; Available online: http://www.systemdynamics.org/conferences/2006/proceed/papers/DWYER367.pdf (accessed on 20 July 2010).
  30. Richmond, B. Systems Thinking: Critical Thinking Skills for the 1990s and Beyond. Sys. Dyn. Rev. 1993, 9, 113–133. [Google Scholar]
  31. Parkins, J.R.; Mitchell, R.E. Public Participation as Public Debate: A Deliberative Turn in Natural Resource Management. Soc. Nat. Resour. 2005, 18, 529–540. [Google Scholar]
  32. Kuhn, T. The Structure of Scientific Revolutions; University of Chicago Press: Chicago, IL, USA, 1962. [Google Scholar]
Back to TopTop