Delphi Research Methodology Applied to Place-Based Watershed Education

Abstract

This research focuses on the results of the Flathead Watershed Delphi survey, a consensus-building methodology used to establish foundational knowledge, skills and dispositions for the Flathead Watershed Educators Guide, a place-based watershed curriculum for middle school grades based on the Flathead Watershed Sourcebook. Survey participants (n = 33) were chosen based on their expertise as educators, resource managers and scientists living and practicing in the Flathead Watershed in northwestern Montana, USA. Participants’ responses were gathered through a three-round survey conducted by the Montana State University (MSU) research team using MSU’s online course management system, Desire 2 Learn (D2L), an anonymous, asynchronous platform with distance accessibility. Round One responses gathered through the D2L discussion tool allowed participants to read responses and reply if desired. Round One discussion responses were reformatted into statements, which were then rated through two successive rounds using a 1–5 Likert scale. Of the initial 142 statements, 91 statements were retained in the final round. Final statements were cross-referenced with the Flathead Watershed Sourcebook to identify learning objectives for the Flathead Watershed Educators Guide. Final statements identified the knowledge, skills, and dispositions deemed most important for students in the Flathead Watershed to learn. Statements supported the need for place-based watershed education in fostering positive attitudes toward conservation and protection of the natural environment.

1. Introduction

Delphi surveys have been used in a number of fields to determine, through consensus opinions of acknowledged experts, foundational practices within those fields. The Flathead Watershed Delphi survey determined, through consensus of knowledgeable community members, the knowledge, skills and dispositions considered most important for students to learn and know about the Flathead Watershed.

To determine the curriculum foundations, conservation and education-minded community members living in the Flathead Watershed of northwestern Montana, USA, collaborated with education professionals in the Department of Education at Montana State University to design and implement the Flathead Delphi survey. The results show that survey responses in the foundational curriculum areas indicated strong consensus around place-based education principles valuing community awareness and service, ecosystem conservation and environmental stewardship.

1.1. The Flathead Watershed

A watershed is a complex system in which water withdrawals, land use, developments, and industries cumulatively affect all living organisms that rely on the river and watershed system. These vital but conflicting ecosystem services result in tradeoffs between water quality, water quantity, ecosystem conservation and human benefits [1].

The Flathead Watershed has its headwaters in the pristine regions of Glacier National Park, the Great Bear Wilderness Area and the Bob Marshall Wilderness Area. The watershed of the Flathead River extends throughout a significant portion of northwestern Montana with one of its tributaries, the North Fork of the Flathead River, originating in the Canadian province of British Columbia. The North, Middle and South Forks of the Flathead River join with the Stillwater, Swan and Whitefish Rivers to feed Flathead Lake, at 191.5 sq. miles, one of the largest freshwater lakes in the Western United States. Flowing out of Flathead Lake, the lower Flathead River joins the Clark Fork River, after which its waters flow west into the Columbia River and eventually into the Pacific Ocean [2]. The Flathead Watershed is one of the largest, most biologically intact ecosystems in North America with over 400 terrestrial wildlife species, including 11 amphibians, 11 reptiles, 319 birds, and 71 mammals [2].

Population in the Flathead Watershed is approximately 125,000 people. The Flathead Reservation of the Confederated Salish and Kootenai Tribes (CSKT) consists of over 1.2 million acres and comprises the southwestern quarter of the watershed. There are approximately 7753 enrolled tribal members, with about 5000 members living on or near the reservation (http://tribalnations.mt.gov/cskt). Population studies in the Flathead Watershed show that the land and the people living on it are inextricably bound together through economics and recreation. Rich natural resources, along with the dramatic natural beauty of the Flathead Watershed, create strong draws for people to move to and stay in the Flathead area. The quality of the natural environment is highly valued throughout the watershed and is considered a chief asset [2]. People choose to live in the Flathead Watershed for what the land offers and they remain in the Flathead Watershed for cultural traditions of family and tribe. Figure 1 shows the geography and scope of the Flathead Watershed and its relative location in the western United States.

1.1.1. Place-Based Flathead Watershed Education

Watersheds are a successful organizing principle for curriculum. Everyone on earth lives within a watershed; the quality of life is greatly affected by the condition of the local watershed; and watersheds can serve as an instructional focus for active learning in science, mathematics, social studies, environmental education, and other subject areas [3].

Place-based education taught through the lens of a watershed recognizes that students are part of and can learn about their place within the natural and cultural boundaries of a local watershed.

In a defining quote about the meaning and value of ‘place’ from the book: Places: Linking Nature, Culture and Planning, Nelson and Lawrence write:

The way people understand and value places strongly influences decisions they make about their conservation and development. Places are more than mere locations. They consist of a broad set of natural and human processes and features, whose interactions and characteristics change through time. Bedrock, weather and climate, plants, animals, soils, human land uses, technology, institutions and social learning, or culture, vary among and define places as we know them.

[4] (p. 1)

The attributes of the Flathead Watershed draw people who choose to live, work and recreate in and near its native ecosystems. Flathead Watershed residents active in organizations dedicated to conservation and education in the Flathead Watershed work to protect the relatively pristine quality of their surroundings. Place-based curriculum about the Flathead Watershed constructed with input from watershed community members creates a circle of connection to the watershed and to the students living there.

1.1.2. Conceptual Framework for Place-Based Watershed Curriculum

The theoretical framework of place-based education is exemplified in community-generated curriculum serving local students. The conceptual and epistemological framework that informs this research is the idea that place-based education develops closer regard and appreciation for the local environment and community.

Placed-based education is the process of using the local community and environment as a starting point to teach concepts in language arts, mathematics, social studies, science and other subjects across the curriculum. This approach to education increases academic achievement, helps students develop stronger ties to their community, enhances students’ appreciation for the natural world, and creates a heightened commitment to serving as active, contributing citizens.

[5] (p. 11)

One of the basic premises of place-based education is that education should prepare people to live and work to sustain the cultural and ecological integrity of the places they inhabit [5]. In order to do that, Orr proposes that place-based education ensures that people become aware of ecological patterns, systems of causation, and the long-term effects of human actions on those patterns and systems [6]. Proponents of place-based education often envision a role for it in achieving local ecological and cultural sustainability [7].

The assumptions of place-based education find its roots in constructivist theory; the building of knowledge on known concepts and information. John Dewey, in his ‘Pedagogic Creed’ stated “all school life should grow out of home life… giving a background of past experience to the new ideas given in school” [8]. Constructivist educators contend that the background knowledge, previous experiences and fundamental worldview of students profoundly affect their interpretation of subject matter [9]. In support of that view, Gruenewald states:

A multidisciplinary analysis of place reveals the many ways that places are profoundly pedagogical. That is, as centers of experience, places teach us about how the world works and how our lives fit into the spaces we occupy. Further, places make us: as occupants of particular places with particular attributes, our identity and our possibilities are shaped.

[10] (p. 631)

Gruenewald goes on to say that place-based education “aims to enlist teachers and students in the firsthand experience of local life and in the political process of understanding and shaping what happens there” [10] (p. 620). In an illustration of Gruenewald’s understanding of the central role of place, Sobel writes of the Lubec High School in Lubec, Maine, where students in the school aquaculture study facility research the issues surrounding the historical fishing industry; actions that have led to revitalization of the threatened economic base of their town [5] (pp. 25–26).

1.1.3. Community in Curriculum Development

Smith discusses five thematic patterns in placed-based learning efforts: cultural studies, nature studies, real-world problem solving, internships and entrepreneurial opportunities, and induction into community process [11]. In his opinion, induction into the community process brings place-based education full circle in creating citizens who are educated within their community, by community members, and who then become acting community members themselves. Smith states that “The primary value of place-based education lies in the way that it serves to strengthen children’s connections to others and to the regions in which they live” [11] (p. 594). Sobel concurs in his statement about community in his definition of place-based education: “Community vitality and environmental quality are improved through the active engagement of local citizens, community organizations, and environmental resources in the life of the school” [5] (p. 11).

Place-based education has advantages over standardized curriculum through increased student engagement leading to greater student achievement [12,13]. A significant barrier to place-based education stems from standardized curriculum in school districts, measured through standardized testing imposed through federal law by the No Child Left Behind (NCLB) Act of 2001 [14].

According to Smith, “Because place-based education is by its nature specific to particular locales, generic curricular models are inappropriate in a place-based educational model” [11] (p. 587); an insight that points to the appropriateness of the local watershed concept to develop a place-based curriculum. This thought reinforces the validity of asking knowledgeable community members about what values they consider to be most important on which to build local watershed curriculum.

1.2. Delphi Survey Research Method History and Use in Curriculum Design

1.2.1. History and Nature of the Delphi Survey

The Delphi survey is named for the Delphic Oracle who was active in the 5th century B.C. in the Greek city of Delphi. In Greece at that time, the Delphic Oracle served as an institutional process to help make policy decisions [15]. The oracle gathered information from a number of informants to deliver a truthful statement, compiled as a result of data from many sources [16]. Within the past century, the Rand Corporation first used the Delphi survey in the 1950s as a prediction method for the implications of the use of warfare technology on armament targets. Experts on the topic of nuclear weapons were consulted and their opinions pooled to formulate a prediction or consensus opinion [17].

Delphi surveys as census-building instruments can be characterized as “a method for structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem” [18] (p. 3). Since Linstone and Turoff’s study of the Delphi survey process, Delphi surveys have become widely used in more academic domains and for more purposes [19]. Delphi surveys, in seeking to uncover a reliable consensus, solicit the opinions of acknowledged experts in the field [20]. Knowledgeable participants can be experts in the field of study, as well as, being community members with personal stakes in the outcomes of the application of the survey results [21].

A Delphi survey is an anonymous, iterative process, where initial responses are discussed and then ranked by individuals in the group as to their importance. This ranking process was initially described in the book, Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes, Delbecq, Van de Ven and Gustafson [22]. Rated responses are winnowed down by group consensus determined by ranked scores to a core set of statements that participants feel are most valuable [18].

A Delphi survey is comprised of a minimum of two groups of individuals, the survey manager or managing team and the survey participants. A third interested party may include the entity for whom the data is being gathered [22]. The participants in the Delphi survey are chosen for their expertise in the area being studied or the complex problem being analyzed. Baker et al., in researching Delphi participants in nursing studies, discusses the definition of expert in terms of recognized knowledge, experience, and/or the ability to influence policy in the field [21]. An expert has been defined in various related ways, such as a group of informed individuals, specialists in their field, or someone who has knowledge about a specific subject [23]. In the matter of participants who are both willing to engage in the process of a Delphi survey and also considered experts in the subject area, Keeney et al. go on to say that:

Simply because individuals have knowledge of a particular topic does not necessarily mean that they are experts. In fact, those who are willing to engage in discussion are more likely to be affected directly by the outcome of the process and are also more likely to become and stay involved in the Delphi. Hence, the commitment of participants is related to their interest and involvement with the question or issue being addressed.

[23] (p. 196)

The same consideration is expressed by Moore, “A nuclear physicist is an appropriate expert if the Delphi concerns atomic energy and a resident of a neighborhood is an expert on what should be a community’s goals” [24] (p. 51).

Four key features define a survey as a ‘Delphi’: anonymity, iteration, controlled feedback, and the statistical aggregation of group response [19]. These features can offer specific advantages in a group decision-making process. Anonymity prevents domination by individuals, an advantage when participants may be hostile to each other, and it alleviates situations where personality styles would be distracting in a face-to-face setting [22]. Anonymity also contributes to the democratic nature of a Delphi [25]. Since a Delphi survey can be conducted online or at a distance, it is appropriate to use when personal contact is not possible due to distance, time and cost constraints. Online surveys can offer the advantage of asynchronous participation.

The Delphi survey structure of iterative feedback, the repeated exposure of participants to statements generated by the group, develops insights that in the end are more than the sum of the parts [25]. Theoretically, the Delphi research process can be continually iterated until consensus is achieved. However, typically three iterations are adequate to collect sufficient data and reach consensus [26]. In a Delphi survey, feedback is controlled through the survey structure put in place by the survey manager. Clayton states that:

The Delphi process aims to arrive at a level of consensus among the panel members. Consensus is assisted by the researcher providing feedback to each panel member of their previous rating together with a group measure of central tendency.

[27] (p. 382)

Statistical aggregation of responses occurs through the analysis of ratings, often generated by Likert scale, to determine mean scores for items in the survey.

The rigor of Delphi surveys, as determined by measures of reliability, validity, and trustworthiness, was researched by Hasson and Keeney in a study of nursing research Delphi surveys [28]. According to Hasson and Keeney, two characteristics of the Delphi survey create issues with establishing rigor. The first characteristic that challenges measures of rigor is the dual nature of classical Delphi surveys, depending on whether the survey is used for prediction or to explore and understand situations. Hasson and Keeney express the duality in philosophy as “juxtaposition of a classical Delphi technique between positivist and naturalistic paradigms that raises a problem about which standards to adopt” [28] (p. 1696). The second characteristic of Delphi surveys leading to dilemmas around rigorous testing is the continual modifications of the method, with at least ten different recognized variants and applications of Delphi designs. While methods to establish rigor through quantitative means remains problematic, the qualitative measures of trustworthiness, obtained through the characteristics of the classical Delphi survey such as iterative rounds, the range and representativeness of participants, and detailed description of the collection and analysis process, are considered a more credible means of establishing rigor [28].

1.2.2. The Use of Delphi Surveys to Develop and Inform Curriculum

Curriculum development results based on a Delphi survey depend on the participants chosen to respond, as well as, the initial prompts that begin the discussion. Depending on the goals of the survey, participants may be stakeholders in the survey outcomes. In curriculum development, stakeholders participating in a Delphi survey may be people invested in the community or profession who will be sharing responsibilities with future colleagues or community members trained in the proposed curriculum [29]. Delphi surveys have been found useful in curriculum development in business, nursing, medicine, agriculture, and technology along with environmental education.

The Delphi survey has been used in curriculum development in several areas in environmental education. Project WET (Water Education for Teachers) is an example of curriculum development based on this approach. The necessary skills, knowledge and dispositions related to water and water resource education were determined by survey participants. The study sample of 268 included a range of people in natural resource management, management education, elementary, secondary and nonformal education, university researchers and natural resource agency managers representing every state in the United States [30]. Of the 5446 initial responses of the participants, 80 had a high level of agreement for the determining criterion; those responses became the framework for the Project WET curriculum [28]. Project WET is a widely respected curriculum used by school and natural resource educators in the United States and in many other countries (http://www.projectwet.org/).

Archaeological Science for All (ASFA) developed a Delphi survey to establish a national framework that delineated the essential science concepts, skills, and dispositions that can be addressed in informal archaeology education activities [31]. The ASFA project consulted experts in the fields of informal science, archaeological science and archaeological education, choosing participants who were geographically situated across the United States. The ASFA Delphi was successful in reaching a consensus regarding archaeological content, skills and dispositions related to teaching science inquiry to under-represented populations. Results of the ASFA survey inform the development of a national plan to assist informal science educators throughout the nation [31].

2. Materials and Methods

2.1. The Flathead Watershed Sourcebook and Educators Guide Development Process

Due to the changing use of land in the Flathead Watershed, and the very high value both long-time residents and new arrivals place on the natural characteristics of the environment, there is a strong concern in watershed communities that growth be balanced with environmental awareness and resource conservation. Concern for sustaining the unparalleled qualities of the Flathead Watershed in the face of increasing pressures for development led to undertaking the Flathead Watershed Delphi survey for the development of the Educators Guide.

Developing awareness of the Flathead Watershed’s unique natural qualities through education was the impetus for the creation of the Flathead Watershed Sourcebook and the proposed environmental education curriculum, the Flathead Watershed Educators Guide (Educators Guide) [2]. The Sourcebook was researched and written by Lori S. Curtis, resident of the town of Whitefish in the Upper Flathead Valley. Curtis was guided and advised by members of the organization known as the Flathead Community of Resource Educators (CORE). The CORE members who worked with Curtis are active in or employed locally with agencies and organizations such as the Forest Service, the Flathead Conservation District, the Flathead Lakers, the Whitefish Institute, Flathead Audubon Society, and the Flathead Basin Commission. Together with Curtis, the advisory group was known as the Steering Committee.

With the completion of the Sourcebook, the Steering Committee planned the development of the Educators Guide as a set of curriculum material for middle school grades, with the Sourcebook being the source of content information. Following the advice of Dr. Michael Brody, Montana State University professor in science and environmental education, the Steering Committee chose the Delphi survey methodology to establish the foundation of the Educators Guide curriculum. The Steering Committee identified and contacted people who would be appropriate Delphi survey participants. Delphi survey participant selection was based on community members’ expertise in areas pertaining to education and to the Flathead Watershed, such as local middle school teachers, natural resource scientists and managers and historians. Dr. Brody, Dr. Yates and Rose Vallor became the MSU research team for the Delphi survey and creation of the Educators Guide.

2.2. Research Method and Design Appropriateness

The Delphi survey method was chosen specifically for its consensus building capability. The Delphi survey research method was appropriate for this use because it created a circumstance in which knowledgeable stakeholders within the Flathead Watershed were able to provide input in an endeavor that may impact the education of their children about the watershed they live in and care about. As an anonymous and iterative consensus building tool, participants were able to learn about the views of other participants and respond with their own thoughts without pressure to conform.

2.3. Validity and Reliability

Descriptive validity is described and achieved by maintaining factual accuracy of the collected data [30]. Descriptive validity was ensured when researchers kept the original wording and intent of participants’ responses to the initial Round One prompt in creating statements for Round Two of the survey.

Table 1. Example of creation of Round Two Statement from Round One responses.

Round One response	Round One response
“I believe in the four C’s- clear, connected, complex and cold. Those four things are essential for the bull trout, an essential animal of the Flathead Watershed”, written by Participant 003.	“Students seem to grasp the 4 C’s quickly and are able to relate the concepts to the health of Big Creek. Students come back to the class room and create a presentation slide show illustrating the 4 C’s and why they are essential components for the native bull trout”, written by Participant 024.
Combined statement for Round Two
The four C’s-clear, connected, complex and cold, are an essential concept for understanding of the Flathead Watershed, particularly for animals such as the bull trout, an essential animal of the Flathead Watershed, compiled by the MSU research team.

is an example showing participants’ responses from the knowledge section of the survey made into declarative, positive statements while preserving the intended concepts (participants’ actual responses are italicized).

Construct validity was established through internal measures, where it became evident that similar or identical themes were repeated throughout participants’ comments. Participants then rated those same themes highly when evaluating them on Likert scales, resulting in those themes remaining in the final data collected in the Delphi survey.

As a further measure of validity, MSU researchers kept an audit trail of the research, starting with selection of participants [32], through initial prompts and responses and ending with the final rating and ranking of the Delphi survey statements. Information was captured electronically through the online course management system D2L and research team meeting notes were recorded and kept in hard drive storage. Conference call notes and email transcripts with Steering Committee members were also collected during the Delphi survey process.

Reliability in this study is defined as the degree in which the research instrument consistently measures what it is designed to measure [33]. In Round Three of the Flathead Delphi survey, reliability was shown by the stability of scores achieved through asking participants to rerate the high ranking statements from Round Two for Round Three of the survey, creating a test-retest scenario for the high ranking statements in the survey.

Table 2. Comparison of Number of Statements from Round Two to Round Three.

Category of Response	Number of Round 2 Statements after Initial Comments	Number of Statements Remaining after Final Ranking	Cut-off Point in Likert Scale Ranking (1–5 Scale)	% of Statements with ‘Important’ or above Rating
Knowledge	57	36	4.06	65%
Skills	40	19	4.06	48%
Dispositions	34	27	3.94	79%
Teaching	12	12	3.94	100%

shows the percentage of statements in each category that received a repeated measure of rating of important or very important (at or near 4.0) on the 1–5 Likert scale, with 5 being considered very important for students to learn. Table 2 also shows the inclusion of a fourth category of statements related to teaching that emerged from the analysis of participant responses from Round One.

2.4. Participant Selection

Enactment of a Delphi survey for the purposes of curriculum development required the participation of individuals with specialized knowledge and skills in areas for which information and opinions were collected. In the Flathead Watershed, the individuals identified were people who were knowledgeable about the Flathead Watershed in a wide range of fields such as natural science, economics, culture and history. Local educators were also considered important contributors to the process. Members of the Sourcebook Steering Committee and the CORE group formed the Curriculum Steering Committee (Steering Committee), who identified the individuals they felt would bring the needed experience to the survey. The Steering Committee decided on three categories of participants: educators, resource managers, and scientists. Members of the Steering Committee each contributed names of individuals that they knew from personal or professional contact within the watershed. The process is described here by one of the participants:

All of us on the committee submitted suggestions, within the groups we identified…teachers, mgrs. and scientists…using our own knowledge of the community and asking people that participated in the original Sourcebook development. We wanted to get representation across the watershed.

[32]

Decisions of who to invite to participate in the Delphi survey were made through consensus of the Steering Committee, who then issued invitations to the proposed participants. The initial number of respondents in the Delphi survey was 29 community members and 4 MSU researchers. Several members of the Steering Committee became survey participants. Participants in the survey came from a wide variety of backgrounds, but had common elements within their experiences. Those elements were engagement with the natural world, teaching experiences, typically in the sciences, a history of community service, and/or employment in natural resources infrastructures such as dams or fish hatcheries.

Table 3. Profession and Affiliation of Initial Delphi Survey Invitees.

Position	Affiliation
Educator	Confederated Salish and Kootenai Tribes (CSKT)
Educator	Salish Kootenai College
Educator	Polson Middle School
Educator	Olney-Bissell School
Educator	Helena Flats School
Educator	Somers Middle School
Educator	Kalispell Middle School
Educator	Whitefish Middle School
Educator	West Valley School
Educator—informal science	Flathead National Forest
Educator—informal science	Flathead Lakers
Educator—administrator	Retired Teacher
Resource Manager/Educator	US Geological Survey/Flathead Lake Biological Station
Resource Manager	CSKT Wildlife Manager
Resource Manager	Kalispell Wastewater
Resource Manager	Rancher
Resource Manager	Creston National Fish Hatchery
Resource Manager	US Forest Service
Resource Manager	F H Stolze Land and Lumber
Resource Manager	Hungry Horse Dam
Resource Manager	Whitefish Lake Institute
Resource Manager	Flathead Conservation District
Scientist/Educator	Whitefish Lake Institute
Scientist	City of Kalispell
Scientist	Flathead National Forest
Scientist	Museum at Central School
Scientist	Flathead Lake Biological Station
Scientist	Flathead Valley Community College
Scientist	CSKT
Investigator	Montana State University
Investigator	Montana State University
Investigator	Montana State University
Investigator	Montana State University

lists the affiliation and profession of the initial Delphi invitees.

Final participation in Round One consisted of 25 people, down from the 33 initially invited. Final numbers of respondents in the first round in each category were nine educators, eight resource managers, and four scientists. In the subsequent rounds, five of the original participants either dropped out (2) or attempted to log on to the server carrying the survey and were not able to (3). Sixteen participants completed the third and final round.

Table 4. Number and Category of Participants in the three Delphi rounds.

Professional Association	Total Number Invited Initially	Participants in Round One	Participants in Round Two	Participants in Round Three
Flathead Watershed (FW) Community Members
Educators	12	9 (75%)	8 (66%)	5 (42%)
Resource Managers	10	8 (80%)	9 (90%)	7 (70%)
Scientists	7	4 (57%)	4 (57%)	4 (57%)
Total FW Participants	29	21 (72%)	21 (72%)	16 (55%)
MSU Research Team	4	4 (100%)	0	0

shows the number of participants in each round of the Delphi Survey.

2.5. Flathead Delphi Survey Process

The three rounds of the Flathead Watershed Delphi survey were delivered through the online course management system known as Desire to Learn (D2L), now called BrightSpace. D2L was selected as the survey implementation tool due to its flexibility in supporting long distance, asynchronistic, and anonymous dialogue. Participants were assigned login ID numbers for D2L for the purpose of anonymity and confidentiality in the Delphi survey.

The first round of the survey consisted of responses written by participants to the survey prompts and participant comments made to other participants’ responses. In the following two rounds participants rated statements created from Round One responses on a 1 to 5 Likert scale based on their opinions of the importance of the statement. In the second round participants were able to comment about individual statements, but only the research team was able to view the comments. In the third round participants had only the option of rating the statements retained from the second round.

2.5.1. Round One

The first round asked that survey participants respond to three prompts, one for each category: knowledge, skills and dispositions. The prompts were:

• What are the essential concepts/issues that students should learn related to the Flathead Watershed? (For example; water quality or fish species);
• What skills should students develop that would help them learn about and live in the Flathead Watershed? (For example; reading maps or writing a letter to city council);
• What attitudes and values should students develop regarding to the Flathead Watershed? (For example; appreciating indigenous perspectives or valuing diverse opinions about resource management).

During Round One participants were able to read and respond to each other’s comments. This allowed ideas to be expressed and developed through dialogue and discussion. Participants were able to write at length about their ideas pertaining to the prompts and to respond to posts made by other participants. No restrictions were given to the number or the length of replies or responses. See

Table 5. Example of Participant Responses for Round One, Disposition Prompt.

Title of Response: Appreciating Where They Live
Response Created by Participant 021 on 28 February 2014 12:42
Students should first be aware of the Watershed and the incredible gift they have been given in living within it. Many don’t recognize that where they live is one of the most incredible places in the world. People migrate back to the Flathead more often than not after experiencing differences in other parts of the country or they experience a sense of awe when flying back into the valley. Getting them to appreciate where they live is paramount in order for them to buy into taking care of their environment. A big part of appreciating the Flathead Watershed will come through honing their observation skills.
Response by Participant 001 2 March at 05:42 I like the idea of explicitly creating learning opportunities that develop real awareness of this remarkable place. Sometimes when you see it and benefit from it every day, you take it for granted. Good point made.
Response by Participant 025 3 March at 09:26 I concur as well. We have an amazing place here. The watershed is as good as it gets ecologically in the Lower 48. So I always stress that first and foremost, before I get into the threats and concerns that we have about our aquatic ecosystem.
Response by Participant 018 4 March at 06:29 I agree. As a person who grew up here I can vouch for the fact that it’s easy to not understand at a young age how fortunate we are to live in this place. When this is all you’ve ever known, you don’t really have good perspective.
Response by Participant 004 6 March at 18:56 Appreciation is a hard thing to develop in a young person but it is worth trying.

for an example of Round One comments and responses.

During this initial round the research team was actively but anonymously participating to help encourage discussions. Researchers’ responses typically were clarifying questions based on participants’ contributions, or statements that explored or agreed with ideas that participants had already expressed. Round One was open to participants for 14 days, from 17 February to 2 March 2014. At the conclusion of Round One, participants’ comments in the three categories, knowledge, skills and dispositions, were transformed into positive, declarative statements. Statements were constructed to preserve the original intent and wording whenever possible. Efforts were also made to eliminate redundancy. A total of 142 statements were formed from participant responses. During Round One comment analysis, the research team recognized the need to create a fourth category due to the number of statements related to teaching methods. Teaching was added as a fourth category in the survey for Rounds Two and Three.

2.5.2. Round Two

Round Two of the survey was designed to narrow the focus of topics and begin to come to a consensus. Using a quiz tool in D2L, participants were asked to rank the revised Round One statements using a 1–5 Likert scale; 5—very important, 4—important, 3—neutral, 2—unimportant, and 1—very unimportant. Participants were also encouraged to comment on and explain their answers. The research team did not participate in rounds two and three.

Table 6. Example of a Round Two Statement with Comments.

Statement #19. Indian Education for All is knowledge that students should have.
Comment # 19: Native American values on water and its importance to life is powerful.
Important to demonstrate how curriculum would meet Indian Education for All standards;
There should be more resources/training available for teachers. I believe that some teachers are reluctant to teach Indian Ed. because they don’t feel comfortable with the amount of knowledge they have on the subject.
Students will connect better with the landscape if they understand that it has been important to people for thousands of years. The more they are taught that they are a part of the landscape just as Native Americans are and were, the more they (hopefully) will realize that this isn’t a place for them to take for granted. Students need to understand that it is the SAME watershed. I think that somehow (perhaps because of elapsed time when Native Americans were the only ones here), students get the impression that Native Americans were part of a different world.
I prefer having the Indian Ed for All mixed in with the rest, not taught separately.

provides an example of comments from Round Two in the disposition category that a participant included with their rating of statement #19.

Round Two was open for participation for 14 days, from 17 April through 20 April 2014. During this round, participants were able to provide comments supporting their ratings. The values from the Likert scale ratings were not made visible to other participants. At the close of Round Two mean scores were calculated and the statements were ranked from most to least important.

Rankings were examined for a natural break or gap in mean scores to determine which items would remain in the final round of the survey. Each category revealed natural breaks at different numeric values, with all of those values at or above the 3.94 range, indicating statements were considered important or very important. Participant comments were taken into consideration to determine if any of the lower ranking statements had strong appeal for participants. There were no comments that suggested reasons to keep lower ranking statements. After eliminating the lowest ranking statements, the remaining statements became the final Round Three materials.

Supplementary Materials Table S7 in Supplementary Materials contains Round Two Delphi Statements in Ranked Order

2.5.3. Round Three

Round Three participants responded to the remaining statements in the four categories by again rating the statements on a 1–5 Likert scale. In this final round participants were asked only to rate the statements, without commenting. The timeframe for Round Three was shortened from 14 days to 5 days due to participant availability. At the conclusion of Round Three, mean scores were calculated from the Likert scale ratings and statements were placed in rank order. The results were again examined for natural breaks. The gap, or break, in mean scores appeared at approximately 4.0, slightly higher than the previous round. This also indicated that the participants had come to a consensus on what was deemed most important for the Educators Guide.

The research team consulted with the Curriculum Steering Committee by teleconference to discuss the results, and it was decided by consensus to include a few selected statements from just below the cut-off point of 4.0, since those statements contained concepts such as education about Native Americans, and use of nature journaling, considered important to include in the Educators Guide and that were not represented in other statements. Due to the limited input from CSKT members, it was the opinion of the research team and the Steering Committee that education about Native American tribal knowledge and history be included in the final statements. The use of nature journaling as a teaching tool was considered by the educators on the team as an important pedagogical practice in a watershed curriculum. A total of 95 statements were retained after eliminations in Round Three.

Supplementary Materials Table S8 in Supplementary Materials contains Round Three scores in ranked order with Means and Standard Deviations

3. Results

The Flathead Watershed Delphi survey sought to answer the following research questions:

• What is the essential knowledge, related to the Flathead Watershed, that students should learn?
• What are the essential skills, related to the Flathead Watershed, that students should be able to do?
• What are the essential dispositions related to the Flathead Watershed that students should have?
• What strategies could be used to guide Flathead Watershed curriculum?
• How do the Delphi survey results relate to the Flathead Watershed Sourcebook?

Note: in the following paragraphs, statements taken directly from the Delphi survey are shown in italics. The statement-identifying letter and number are indicated in parentheses following the italicized statement. The complete list of Round Three statements in ranked order in each category can be accessed in Supplementary Materials Table S8.

3.1. Research Question One

What is the Essential Knowledge, Related to the Flathead Watershed, That Students Should Learn?

This research question was addressed in the initial prompt from Round One of the Delphi survey in the category of knowledge: What are the essential concepts/issues that students should learn related to the Flathead Watershed? (for example; water quality or fish species). This prompt provided the greatest number of responses. Figure 2 shows the final knowledge statements in rank order with standard deviation (SD) bars indicating plus or minus one SD from the mean score. The shorter lengths of the bars on the highest-ranking statements indicate less SD from the mean and a higher degree of agreement of the importance of that statement. The increasing lengths of the bars indicate a wider range of opinion about the importance of that statement. In general, standard deviations are within one score range, indicating that participants were in fairly close agreement about the relative importance of each statement.

Several knowledge statements stand out as having larger SDs than typical for their rank. The first statement is K11: Students need to know the basic premise of a watershed, that it is where they hunt and fish and recreate, and work, and live, and eat and grow food and consume food, and that as a result; they are also influencing the watershed with all their everyday actions, M = 4.50. This statement received two scores of 3, neutral on the Likert scale, 4 scores of 4, important on the scale, and 10 scores of 5, very important on the scale. The next statement with a larger SD is K4: Students should be aware that a major threat to the Flathead watershed is water quality degradation, predominantly nutrient and sediment additions caused by human activities. The Likert scores for this statement were as follows, 1—unimportant, 1—neutral, 4—important and 10 very important. The third statement with a wide range of SD is K21: Students should be aware of the storm water conveyance system and that storm drains flow directly into our water bodies without treatment, M = 4.19. Scores for this statement were 1—very unimportant, 1—neutral, 7—important and 7—very important. K36 is a repeat of the statement K21. It received scores of 1—very unimportant, 1—unimportant, 0—neutral, 9—important and 5—very important for its second appearance on the list, M = 4.00. Having similar scores each time this statement was rated indicates consistency in participants’ decisions about the statements.

In the knowledge category the top ranking statement, with a mean score of 4.69, was K2: Students should be knowledgeable about the plants and animals of the watershed, including invasive species and their impact on the watershed. The next three statements have mean scores of 4.63. Those three statements are: K1: Students should be aware why the watershed is important to the species of fish and wildlife that utilize the watershed; K30: It is important for students to know how humans impact the watershed; K32: Students should understand the characteristics of a healthy watershed. The content of the top ranked statements indicated survey participants expressed the ideas that learning about animals, plants and watershed characteristics of the Flathead Watershed were important for students. Table S8 contains all the Round Three knowledge statements in ranked order.

3.2. Research Question Two

What are the Essential Skills, Related to the Flathead Watershed, That Students Should be Able to Do?

The Delphi survey prompt addressing essential skills was: What skills should students develop that would help them learn about and live in the Flathead Watershed? (For example; reading maps or writing a letter to city council). Figure 3 illustrates the ranking order for skill statements for Round 3 of the Delphi survey with error bars showing +1/−1 standard deviation. As noted with the knowledge statements, statements with the highest Likert scale ranking also show the smallest standard deviations, meaning survey participants were in closest agreement on the importance of those skills for inclusion in the Flathead Watershed Educators Guide curriculum.

All the statements in the skills category received a mean score rating of 3.94 or above with the exception of S5: “Nature journaling” or “field journaling” gives young students the tools they need to quietly observe what is going on around them. With a mean score of 3.69, field journaling ranked below 3.94, but in consultation with the Steering Committee, it was considered an important skill to keep and incorporate into the Educators Guide curriculum.

3.3. Research Question Three

What are the Essential Dispositions Related to the Flathead Watershed That Students Should Have?

The Delphi survey prompt for the disposition statements was: What attitudes and values should students develop related to the Flathead Watershed? (For example; appreciating indigenous perspectives or valuing diverse opinions about resource management) In the category of dispositions the mean scores vary from M = 4.56 to M = 3.81, less than a full point difference from the highest ranked score to the lowest ranked statement. In Figure 4 the disposition statements are shown in rank order by mean score, with +1/−1 SD bars. The relatively flat mean score line and short range of distribution of SD bars in Figure 5 show consistent ratings for disposition statements in Round 3. SD bars become longer for most of the statements as the rank order of the statement becomes lower. Table S8 contains all the Round Three skill statements in ranked order.

An exception to that trend is Statement D3: It is important that students engage in life styles (choices and decisions) that protect and sustain a healthy Flathead Watershed. M = 4.06, that had the highest standard deviation, SD = 1.12. This statement was considered very unimportant by one Delphi survey participant; the complete range of scores are: 1—very unimportant, 3—neutral, 5—important, and 7—very important.

The last three statements in the dispositions ranking had mean scores of 3.88, 3.88 and 3.81, respectively. These statements were included in the final statements in consultation with the Steering Committee because it was felt the ideas expressed in them were important relative to the Flathead Watershed. The statements are: D7: Salish, Kootenai and Pend d’Oreille history and cultural resources, such as films, books, have a tribal perspective and are important sources of information on science, history and contemporary information, including topics such as Indian Water Rights and native place names, which should be used in classrooms in the Flathead Watershed; D22: The major issues affecting watersheds today, such as water rights, toxic spills from rail or truck, toxic cleanup, industrial pollution, are important issues for students to learn about; and D18: It is valuable to know the Native American perspectives and to understand how the ecosystem has changed: such knowing could lead to a greater appreciation for the Flathead Watershed. The mean scores of the last three statements fall below the lower end of the standard deviation error bar of the first statement, indicating those statements have a lower rating of importance. Table S8 contains all the Round Three disposition statements in ranked order.

3.4. Research Question Four

What Strategies Could be Used to Guide Flathead Watershed Curriculum?

The teaching statement was an emergent category created after Round One of the Delphi survey when it was determined that a number of comments contained specific suggestions on ways to teach knowledge, skills or dispositions in a watershed curriculum. Figure 5 shows the ranked teaching statements.

The line for the ranked mean scores for the teaching statements trends slightly downward, while the SD bars do not show a strong trend to lengthening or shortening with descending rank. Several statements, such as T1, T4, and T9, have slightly longer SD bars. Statement T1: A great Socratic Seminar topic for students could be the statement: “Whatever happens to the water happens to the people”, M = 3.94 received one score of 1—very unimportant, 5—neutral, 3—important, and 7—very important. Statement T4: Teaching about diversity can be accomplished through expression in subject matter teaching. An example is the multiple ways water quality can be introduced: explaining what scientists look for in their work, explaining the cultural importance of native fish and their waterways to indigenous people, or discussing various approaches to resource management of the same water body and why each manager might choose this approach. M = 3.94, received scores of 2—very unimportant, 13—unimportant, 4—neutral, 1—important, and 0—very important. Statement T9: Reading about non-fiction topics is a good way to learn about animals and the watershed and satisfy common core standards requirements for young students M = 3.69, received scores of 6—very unimportant, 8—unimportant, 5—neutral, 1—important, and 0—very important. Supplementary Materials Table S8 contains all the Round Three teaching statements in ranked order.

3.5. Research Question Five

How Do the Delphi Survey Results Relate to the Flathead Watershed Sourcebook?

The creation of the Educators Guide was the goal of the Steering Committee and the motivation for conducting the Flathead Delphi survey. To determine how the Delphi survey results related to the Sourcebook, the Sourcebook contents were analyzed and sorted by major content areas. An outline of the Sourcebook was developed with the Sourcebook contents delineated into chapters, sub-chapters, sections and topics in each sub-section. The final Delphi statements were then matched to the Sourcebook topics in each sub-section in a matrix format. The Sourcebook contents were continually checked throughout the process for accuracy in capturing Sourcebook topic content and to ensure that Delphi statement interpretations were suitably matched to relevant Sourcebook content.

Twenty-three learning objectives were developed for the areas of greatest match between Delphi statements and Sourcebook content. The learning objectives sought to be inclusive of the maximum amount of both Sourcebook contents and Delphi statements. The Sourcebook Delphi Statement matrix was reviewed by the Steering Committee, who gave their approval of the scope and content of the learning objectives. The twenty-three learning objectives became the basis of the Flathead Watershed Educators Guide.

4. Discussion

The success of the Flathead Watershed Delphi survey in developing a foundation for the Educators Guide is demonstrated by the richness and depth of the statements that constitute the final round results. The initial responses to the Round One prompts were thoughtful, heartfelt opinions that indicated the value participants placed on stewardship of the environment and community in the Flathead Watershed. The Delphi survey methodology created the opportunity for community members to express their own concern for and commitment to the physical and cultural Flathead Watershed attributes.

The effectiveness of the Delphi research methodology in surveying experts to determine a foundation for curriculum is corroborated by evidence from several studies on curriculum development from Delphi consensus statements [30,31,34,35]. Examples of Delphi studies come from Brody, who discusses developing a water education curriculum based on the expert opinion of educators and scientists from across the United States, and Miller, et al. whose study determined a training curriculum within a specified field of enterprise, based on the consensus of experts in the field [30,34]. Common features of the Delphi studies that contributed to their success were the identification of experts in the field, the advantages in conducting the survey online, and the consensus building aspects created from the initial open discussion and multiple ratings of survey statements.

A unique and innovative element in the Flathead Watershed Delphi survey is the contributions of local community experts informing the Educators’ Guide curriculum for the essential knowledge, skills and dispositions that the youth of their own community should acquire. Building a census of local experts’ opinions about what they consider important for students to know about their communities and environment is an unexplored area in curriculum development.

In the case of the Flathead Watershed Delphi survey, the experts, who are also community members, played a key role in the quality of the statements generated from their responses to the survey prompts and to each other. Local experts who are community members themselves have a stake in the stewardship of their environment and the successful continuation of their communities. In a sense, asking local experts’ thoughts and opinions for curriculum for their youth is giving them the opportunity of passing on their knowledge and wisdom to educate and nurture the young people who will succeed them in management and conservation of the watershed.

An outcome of organizing a Delphi survey among people with common concerns and interests was the creation of a social construct that may not have existed prior to the survey. In the case of the Flathead Watershed Delphi survey, participants became members of a local network sharing the same concerns for the environment, of their watershed, and the education of the youth within the communities of the watershed.

A further innovative aspect of the Flathead Watershed Delphi survey is using the concept of a local watershed as an organizing principle for the survey. While there are curricula focused on watersheds in general, and there are examples of curricula for specific watersheds in the United States such as the Colorado and Missouri available through Project WET, the Flathead Watershed Delphi survey’s basis was the participation of local citizens responding from their knowledge of their own watershed.

Although the survey included individuals from a wide range of backgrounds, it could have been enhanced by the participation of tribal members of the Confederated Salish and Kootenai Tribes (CSKT). Two tribal members responded to the initial prompts in Round One of the survey and their ideas and opinions were included in the declarative statements rated in the second and third rounds, though they did not participate in those rounds. The Salish and Kootenai tribal members comprise approximately 6% of the population of the Flathead Watershed and own nearly one quarter of the land within the watershed boundaries. The Flathead Watershed is part of the traditional native lands of the two tribes. The CSKT are actively involved in conservation efforts to preserve native species and to maintain their ways of living based on Flathead Watershed ecosystems.

The contributions of more elementary and middle school teachers would have been welcomed as well. A total of 9 participating teachers drawn from the 68 schools within the watershed comprise a small representative sample. With the target audience of the Educators’ Guide being middle grades, the additional input from teachers would have been valuable.

While there is no further work being planned involving data from this survey at this point, any additions to the Flathead Watershed Educators Guide would likely benefit from further input from CSKT members and educators in the Flathead Watershed.

There are several areas for further research related to the Flathead Watershed Delphi survey. One area to explore is further analysis of the data collected in the Flathead Watershed Delphi survey. A comparison of the responses and rating data for each sub-group of participants, educators, resource professionals and scientists, may provide additional insight. Trends in the responses for each sub-group could point to consistent behavior and choices for each group, though the small sample sizes of the subgroups limit the applicability of the analysis. Another area of potential future analysis of Flathead Watershed data would be to look at statements receiving higher rating in comparison to statements rated with lower scores. Patterns in comparisons may point to methodology issues or trends in participant choices.

Two other potential research areas to consider are assessing the effectiveness of place-based watershed curriculum and examining the validity of the Delphi survey approach in gathering input from local communities to determine curriculum foundational concepts. One possibility for assessing curriculum effectiveness would be to pilot the Educators Guide in Flathead Watershed classrooms. Measures of effectiveness would need to be determined in such a pilot study. The validity of a Delphi survey as a viable option to determine a foundation for watershed place-based curriculums could be explored by conducting a community-based Delphi survey in a similar watershed and comparing results of the two surveys.

5. Conclusions

The Flathead Delphi survey to determine a basis for the Flathead Watershed Educators Guide was created to develop a consensus of opinions about the knowledge, skills and dispositions about the Flathead Watershed considered important by conservation-minded and environmentally-minded local resident watershed experts. The high ratings for the concepts expressed in the survey responses validated the hypothesis that there are foundational knowledge, skills, and dispositions that can be identified to form the basis for the Educators’ Guide.

This research study shows that surveying community members who are experts in aspects of their watershed results in a substantial pool of knowledge, skill and disposition statements, agreed upon by consensus, to form a basis for a place-based watershed curriculum. In essence, the Flathead Delphi survey is a research tool to learn about and examine what motivates resident stakeholders to conserve and protect environmental attributes and causes them to expend time and energy on education through curriculum that enhances appreciation and protection of local watershed attributes.

A local watershed-based curriculum embodies the concept of place-based education. A place-based curriculum based on the watershed concept, looked at in both biological and cultural terms, recognizes the interdependence of the people and the ecosystems within the physical parameters defined by the flow of water through the landscape. The goal of a place-based watershed curriculum is education to create awareness that leads to stewardship.

The use of a Delphi survey for curriculum content development based on the attributes of a watershed is a new concept in education. Determining the values community members hold in relationship to the watershed they live in is an unexplored area of study in environmental education. Further research into the implications of the place-based education and community watershed concept is needed to understand and validate the approach.