Sustainable Development (SD) and sustainability are omnipresent concepts in today’s world. In nearly all programmatic documents on sustainable development, education is an important component [1
]. It is internationally recognized that Education for Sustainable Development (ESD) is a substantial “element of quality education and a key enabler for sustainable development” [2
]. ESD should serve to enhance the ability of children and adolescents to participate in establishing SD [1
]. “ESD is expected both to make people more aware and better qualified to take part in shaping future developments responsibly and to raise their awareness of the problems related to sustainable development and bring forth innovative contributions to all economic, social, environmental and cultural issues” [3
]. In the years 2005 to 2014, the United Nations (UN) Decade of ESD fostered the role of education regarding SD [2
]. Also, one of the 17 Sustainable Development Goals (SDGs) addresses education. Goal Four reads, “Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all” [4
]. Particularly, the associated “Target 4.7” deals with SD. In the Global Action Programme (GAP) on ESD of UNESCO (2015–2019), the “building capacities of educators and trainers” is one of five priority action areas identified [2
]. Therefore, the education of future multipliers is an objective to support SD in a wider range. Because of its potential multiplier effect [5
], teacher education has a key function in fostering ESD.
Target 4.7: “By 2030, ensure that all learners acquire the knowledge and skills needed to promote sustainable development, including, among others, through education for sustainable development and sustainable lifestyles, human rights, gender equality, promotion of a culture of peace and non-violence, global citizenship and appreciation of cultural diversity and of culture’s contribution to sustainable development”.
Despite the UN Decade, teacher training has been insufficient so far. “To date there are hardly any structural changes in teacher education and training, and there is a need for development in the school structures and curricular requirements in order to promote the ESD concept” [6
]. There are several initiatives to promote ESD in teacher education. There are joint networks for teacher education promoting ESD around the world (e.g., the United States Teacher Education for Sustainable Development Network [7
], the Teacher Education for Equity and Sustainability Network (TEESnet) in the UK, and the Germanspeaking network Teacher Education for a Sustainable Development (LeNa) [8
1.1. Teacher Education and Knowledge Relevant for ESD
Regarding teacher education, professional action competence is crucial for successful teaching [9
] and can decisively impact ESD [11
]. Professional action competence is composed of professional knowledge along with motivational, volitional, and social willingness and skills [12
]. Professional knowledge can be divided into content knowledge (CK), pedagogical content knowledge (PCK), and pedagogical knowledge (PK) (cf. [9
]). Based on Shulman [13
], Baumert and Kunter [9
] developed a competence model for mathematics, which was adapted for ESD by Hellberg-Rode, Schrüfer and Hemmer (cf. [14
]). Cognitive competencies, which teachers are supposed to be equipped with, regarding ESD could be extracted in an expert study. Thirty-eight percentage points of these competencies were assigned to CK, 23% to PCK, and 19% to PK. Therefore, CK plays a critical role regarding ESD and should be focused on in this contribution.
To optimize ESD in teacher education, knowledge about the prerequisites for teachers facing SD challenges is essential [16
]. Kaiser and Fuhrer [17
] complain about the often undifferentiated measurement of knowledge which may result in underestimating the role of knowledge (cf. [17
]). In the literature, there exists a variety of knowledge classifications (e.g., [19
]). A common distinction is made between know-that/what (declarative, conceptual knowledge) and know-how (procedural, strategic knowledge) [21
]. Challenges of SD are often situations of high complexity with conflicts between different legitimate objectives, e.g., ecological, economic, social, and/or institutional perspectives. Therefore, solving problems is crucial for coping with SD challenges. The knowledge model of de Jong and Ferguson-Hessler [23
] explicitly focuses on problem-solving and thus on knowledge-in-use. The model defines four types of knowledge: situational, conceptual, procedural, and strategic knowledge [23
]. Situational knowledge comprises the knowledge about domain-specific situations. For example, this knowledge allows extracting relevant information from a given problem description and adding further information [23
]. Conceptual knowledge contains “concepts, facts and principles that apply within a certain domain” [23
]. It allows the problem solver to change the state of the problem. Procedural knowledge comprises actions that are suitable to certain types of problems in the specific domain [23
]. Strategic knowledge comprises the sequential action steps which are necessary to solve a problem and is “applicable to a wider variety of types of problems within a domain” [23
]. Other authors often do not distinguish between procedural and strategic knowledge, but rather include the step-by-step actions for problem-solving in the term procedural knowledge [19
The model of de Jong and Ferguson-Hessler [23
] turned out to be well applicable for measuring knowledge in the domain of SD. For example, the model was picked up to determine the knowledge of Indonesian university students concerning resource use problems [23
]. With the developed measurement instrument, situational, conceptual, and procedural knowledge (according to de Jong and Fergusson-Hessler [23
]) were recorded (n
= 882) [27
]. Situational and conceptual knowledge were measured by multiple-choice items. The single items focused mainly on ecological, socio-economic or institutional questions [27
]. For answering questions on situational knowledge, information from problem descriptions, presented as scenarios, had to be extracted and complemented. For answering questions on conceptual knowledge, knowledge about concepts and facts had to be applied.
In contrast to situational and conceptual knowledge, the different nature of procedural knowledge in the ESD domain requires one to consider core features of the concept of SD. Instead of clear (more discipline-focused) solutions like for situational and conceptual knowledge, different perspectives such as economic, ecological, institutional, and social factors have to be taken into account simultaneously [28
] when evaluating a solution strategy. Procedural knowledge is described as “the cognitive skill of identifying and judging potential solutions (‘strategies‘)” [27
] of environmental problems.
For being able to evaluate procedural knowledge of students in higher education, Koch et al. [27
] conducted an expert study (n
= 9) to establish a benchmark. The expert benchmark allowed them to judge the student answers concerning procedural knowledge items. The benchmark resulted from a single expert survey. The procedural knowledge items of the benchmark were developed by the authors without integrating further considerations and suggestions on problem-solving strategies of the experts. Thus, the procedure for benchmark development can be improved with respect to more intensively incorporating the expertise of the participants. Koch et al. [27
] showed how to measure knowledge for typical Indonesian resource use dilemmas, such as rattan extraction from the rainforest or dynamite fishing, in higher education in Indonesia. The study demonstrated substantial gaps in the students’ ability to solve complex environmental problems [27
For European teacher education and even teacher education in other continents, the Indonesia-related resource use dilemmas are not of similar relevance as they are for education in Indonesia. Therefore, this contribution focuses on challenges of SD relevant on a global scale: biodiversity loss and climate change [29
]. Both are implied in the ESD goals [29
]. Several studies reported that science teachers show large deficits in their biodiversity-related knowledge [16
]. For example, biology student teachers are unaware of the core facets attached to biodiversity, such as its sustainable use [33
]. Furthermore, student teachers’ understanding of “the terminology, distribution, and loss of biodiversity” [16
] does not equate with scientific understanding. Studies with teachers, pre-service teachers, and student teachers also revealed gaps in knowledge and understanding, e.g., misconceptions of the greenhouse effect and (the causes of) global warming [32
]. However, knowing the concepts of biodiversity and climate change, with the complex relations between ecological, social, and economic factors, is crucial for ESD. Thus, the ability to change perspectives is an indispensable ESD-specific competence [15
] and highly important for procedural knowledge in the domain of SD [27
]. However, interdisciplinary approaches to face controversial SD challenges are hardly implemented in science courses (cf. [40
1.2. SD Challenges Biodiversity Loss and Climate Change and the Corresponding Fields of Action
For the SD challenges biodiversity loss and climate change, there are many possible contexts. In the following section, an up-to-date, highly socially relevant and exemplary context concerning local and global biodiversity is described: insects and pollination. Second, a context with noteworthy impact on climate change is exemplified: use of peatlands.
Numerous crops and wild plants depend on insects as pollinators, particularly bees. Besides domesticated pollinators (honeybees), wild pollinator populations play an important role [41
]. Seventy-five percent of leading global food crops rely on animal pollination. In total, 35% of global production depends on pollinators [42
]. The global economic value of insect pollination amounts to about €153 billion per year [43
]. In the last few decades, managed honeybee populations, as well as wild pollinators and plants that depend on their pollination services, declined [41
]. A combination of different factors may be the cause of these declines: land use intensification with loss and fragmentation of habitats in agricultural landscapes [45
], improper use of plant protection products, particularly neonicotinoids [44
], and diseases or parasites like the Varroa mite [50
In Europe, over the last century, 52% of total peatland area has been converted for agriculture, forestry, and peat extraction [51
]. Peatlands influence the global climate system, particularly through carbon dioxide (CO2
) and methane (CH4
), and less through nitrous oxide (N2
]. Because they accumulate more carbon through photosynthesis than they release through respiration, most uncultivated peatlands are sinks for atmospheric CO2
]. Simultaneously, most peatlands emit the powerful greenhouse gas methane (CH4
), caused by the wet conditions [51
]. The most accumulation of peat occurred within the past 10,000–20,000 years [52
]. Land use activities like agriculture and peat extraction for potting soil require drainage of the peatland. Consequences of drainage include enhanced decomposition of peat, the generation of CO2
O emissions and reduced CH4
Thus, insects and pollination and peatland use are major contexts within the SD challenges biodiversity loss and climate change. It is common practice in current research, e.g., on knowledge [27
] or on socio-scientific reasoning [53
], to incorporate two socio-scientific issues (rattan extraction and dynamite fishing in [27
], land use management issue and fracking in [53
]) to assess the respective competencies. To measure the constructs to be investigated, independent from prior knowledge, problem descriptions are provided in the form of scenarios. This is conventional in competence research, e.g., in Programme for International Student Assessment (PISA) [54
Recently, measures for situational and conceptual SD-related knowledge have been developed for the following contexts: insects and pollination and peatland use [55
]. To complete the SD knowledge measurement instrument in the domain of SD, a measure for crucial procedural knowledge is still lacking. The term “measure” is used for clearly distinguishing from the term “measurement instrument”, which additionally includes situational and conceptual knowledge. Therefore, the core research question is: how can interdisciplinary SD-related procedural knowledge of student teachers be measured? The overall study aims to:
refine a procedure to measure procedural knowledge of student teachers for coping with SD challenges and thereby,
define a measure for such knowledge and a benchmark for its evaluation.
In the following section, the methodological approaches for developing a procedure to measure SD-relevant procedural knowledge of student teachers will be presented. In the current project, procedural knowledge refers to the domain of SD. Procedural knowledge equates with the cognitive skill to solve certain SD-related problems (cf. [23
]). Therefore, considering and weighing different perspectives is necessary.
The development of a measurement instrument for SD-related knowledge of student teachers considers the contexts of insects and pollination and peatland use. Both are complex environmental problems requiring an interdisciplinary and/or multidisciplinary perspective. According to common practices [27
], two scenarios of the real-world problem (one for each context) are given. They form the basis for evaluating situational, conceptual and, presented in this study, procedural knowledge. One scenario deals with a large bee colony loss in Germany in 2008 [49
]. The second context deals with the agricultural and industrial use of peatlands. The scenarios (abbreviated and slightly modified) are printed in Appendix A
and Appendix B
With respect to developing the measure for procedural knowledge, the following steps have been conducted: first, a literature review was performed to gather proposed solution strategies for SD challenges. Second, a two-round Delphi study was realized (cf. [58
]). The Delphi procedure was enriched by an intermediate think-aloud study with student teachers (Figure 1
). Before answering the questionnaire regarding solution strategies for both contexts, the students received the above-mentioned corresponding scenarios. The latter was not given to the experts. It was assumed that experts possessed such knowledge. The whole procedure serves to develop sets of potential solution strategies that are literature based, reviewed by experts and accessible to the understanding of student teachers. The elaborate procedure will result in a measure for a survey on SD-relevant procedural knowledge of student teachers (Figure 1
According to common practice (cf. [27
]) two contexts were chosen to operationalize procedural knowledge: insects and pollination and peatland use. The solution strategies proposed in the items require interdisciplinary considerations that integrate environmental, socio-economic, and institutional issues. To broadly cover both exemplified SD challenges, items were deduced from eight overarching topics: environmental policy, ensuring the diversity of species, sustainable management, and ESD (see Section 3
During the process of questionnaire development (Figure 1
), a special questionnaire format was used. In the Delphi survey, the experts were asked to rate the effectiveness of given solution strategies on a four-point Likert scale, from “ineffective” to “very effective”. Each solution strategy was rated for three essential fields of action. For the insect and pollination context, the solution strategies were estimated concerning: (i) realization of sustainable land use, (ii) provision of ecosystem services, and (iii) biodiversity conservation. For the peatland use context, the solution strategies were equally reviewed regarding (i) and (ii), whereas (iii) consisted of the contribution to climate protection (Figure 2
). Each expert had to rate effectiveness in both contexts. In addition to the three answers per solution strategy, the experts gave information about how certain they were about their effectiveness estimations on a four-point Likert scale, from “absolutely uncertain” to “very certain” (Figure 2
). The evaluation of subjective certainty of estimations is a characteristic of Delphi surveys [59
In addition to the procedural knowledge assessment, the experts were asked to self-assess their knowledge regarding eight topics: biodiversity, bees and pollination, climate change, the importance of peatlands, sustainable development, sustainable land use, environmental policy, and ecosystem services. The experts could choose “unsatisfactory”, “satisfactory”, “good”, “very good”, or “excellent”. Each expert answered the questions once, even if they participated in both rounds. The entire procedure was conducted in German.
2.1. Sample Composition
Experts from different disciplines were invited to participate in the Delphi survey. Potential participants had expertise as scientists in subjects such as teacher education, biology, climatology, or agricultural sciences. The final participant list included 15 professors, four postdoctoral scientists and four persons with unknown academic degrees. They came from nine German universities as well as from four non-university institutions (Table 1
). All participants were people who deal with ESD, biodiversity, climate change or real-world problems of insects and pollination and peatland use in a wide range of fields.
For the first Delphi round, 27 experts were invited to participate. Among them, 19 answered the questionnaire in time (five female, 13 male, one not stated). The average age of the experts was 46.6 years (standard deviation (SD): 9.3). In the second Delphi round, 30 experts were invited, and 21 answered the questionnaire in the given deadline (five female, 15 male, one not stated). One person was subsequently excluded because of a lack of expertise (youngest scientific age and high self-assessment in contrast to the other experts). One person of the remaining experts only answered the peatland use context items. The average age of the 20 experts was 50.4 years (SD: 8.7). Three persons were between 31 and 40 years of age, nine were between 41 and 50 years, five were between 51 and 60 years, and three were older than 60 years. Sixteen of the 20 persons also participated in the first Delphi round. Table 1
displays the sample compositions with all participants of the first and second Delphi round. All experts of both Delphi rounds participated without any expense allowance.
The think-aloud study was conducted with nine student teachers: one bachelors and two masters students in biology, two bachelors and one masters student in geography, and two bachelors students and one student after his studies in political science.
2.2. Delphi Survey First Round
Based on the literature review, 41 solution strategies (Likert scale items) for insects and pollination (21 items) and for peatland use (20 items) were identified [58
]. Apart from estimating effectiveness in the three essential fields of action and the certainty of the effectiveness, the experts had the opportunity to comment on the solution strategies and make suggestions for further solution strategies in an open-ended format. The qualitative data generated by comments and suggestions were analyzed through qualitative content analysis according to Mayring [58
]. This analysis resulted in clarifying and optimizing items and determining further new solution strategies. Two items concerning agricultural subsidies originated from suggestions made by the experts in the first Delphi round (IP-8 and PU-10, see Section 3
The processing time of the questionnaire with all of its elements (estimations of effectiveness, estimation of certainty, comments, suggestions for solution strategies, and self-assessed knowledge) amounted to a maximum of 45 minutes. For the quantitative analysis, Excel (Microsoft Office Professional Plus 2013) was used. Statistical measures like means, medians and standard deviations were calculated with the quantitative data. Medians and percentage distribution within the response categories were used for providing graphically documented results to the participants of the second Delphi round (Figure 3
). In some cases, two boxes were marked for one rating scale. Instead of excluding these answers, it was assumed that the experts wished to have a scale with higher resolution. Therefore, the neighboring values were averaged.
Taking into account: (i) the results of analyzing the items from the questionnaire draft; (ii) the item difficulties; (iii) the comments and suggestions from the experts; and (iv) the representation of the eight overarching topics, a first revised version of the questionnaire with 27 items was created (Figure 1
). These 27 items cover 14 items in the insects and pollination context and 13 items in the peatland use context.
2.3. Think-aloud Study with Student Teachers
To develop a procedural knowledge test for teacher education purposes, the procedure requires testing with (prospective) teachers. Therefore, a think-aloud study was performed with nine student teachers (Figure 1
; for the method, see [62
]) with the first revised version of the questionnaire (27 solution strategies; Figure 1
). The think-aloud method is well suited to checking how items are perceived by subjects and a common method of item validation [63
]. The think-aloud protocols were transcribed and analyzed through qualitative content analysis according to Mayring [61
Like the experts, the students assessed the effectiveness of solution strategies on a four-point Likert scale in three fields of action: (i) realization of sustainable land use, (ii) provision of ecosystem services, and (iii) biodiversity conservation in the insect and pollination context or contribution to climate protection in the peatland use context (Figure 2
). In contrast to the experts, the students received background information on two scenarios of real-world problems before answering the questionnaire (Appendix A
and Appendix B
). The objective was to establish a homogeneous knowledge base on which procedural knowledge could be assessed.
The aim of the think-aloud study was to adapt the questionnaire material for student teachers. Thus, problems concerning the two scenarios, the corresponding solution strategies (items), and the three fields of action were identified. Problems in understanding items led to rewording of those items, e.g., several items were linguistically simplified. Sometimes, a supplement in the sentence was added to make the strategy presented more understandable for student teachers. For example, in the solution strategy “The government provides financial incentives for using bee-friendly bloomers as biomass in biogas power stations”, the phrase “as biomass” was appended after the think-aloud study. Several students were confused with the former formulation, e.g., Sandra mentioned, “I do not know if biogas power plants [...] are the areas where plants are grown that produce biogas? I have no idea at all” (134–136, translated from German). In addition to item revision and optimization, the think-aloud study provided indications for the potential removal of items, as we were aiming for a short measure of 20 items broadly covering both contexts. Furthermore, the think-aloud study revealed a need to explain the meanings of “sustainable land use”, “ecosystem services”, and “biodiversity” for student teachers. This resulted in editing a supplementary informational sheet containing brief definitions of these terms.
2.4. Delphi Survey Second Round
Controlled feedback and statistical aggregation of all participating experts from the first round responses are key characteristics of a Delphi survey [65
]. Therefore, in the second Delphi round, the experts received the graphically processed results of the first Delphi round, the supplementary information sheet containing the requested definitions of fields of action, and the second revised and condensed version of the questionnaire (20 items, Figure 1
). Ten items addressed solution strategies for insects and pollination and 10 items addressed solution strategies for peatland use. Again, the experts had to assess these strategies regarding their effectiveness concerning the three fields of action and indicate their certainty of their effectiveness assessment, as in the first round (Figure 2
). The processing time of the questionnaire took 15 to 20 minutes. The latter was due to the reduced item number, no further requested comments or suggestions and the self-assessment of knowledge was only necessary for those who did not participate in the first round.
The processing of the quantitative data of the second Delphi round finally aimed at establishing a benchmark for being able to evaluate procedural knowledge of student teachers. For the quantitative analysis, IBM SPSS Statistics 24 and Excel (Microsoft Office Professional Plus 2013) were used. The four-point Likert scales for effectiveness were coded from 1 (ineffective) to 4 (very effective). Like in the first Delphi round, if two boxes were marked, the neighboring values were averaged. Because of the two different contexts and due to differing expertise, the experts’ answers were weighted with the given certainty [66
] for more reliable results. Therefore, the function “weight cases” in SPSS was used. With the weighted values, means and standard deviations were calculated. These statistics were calculated separately into the three fields of action. Additionally, these measures were calculated for each solution strategy across the three fields of action. We call this the “total weighted effectivity with respect to SD challenge”, or in short version “SD effectivity”.
For each of the three fields of action, a reliability analysis was performed with the expert data on effectiveness estimations. Because of a lack of procedure in SPSS to weigh the items with different variables, the unweighted values were used for the analyses. In addition, a reliability analysis with weighting over averaged certainty estimation was performed. Cronbach’s alpha was computed across the 10 solution strategies for insects and pollination and peatland use and separated according to the three fields of action. Furthermore, analyses of significant differences between the fields of action of each solution strategy were tested by repeated measures ANOVA (rmANOVA) with unweighted values. To check the validity of the expert data, the experts’ self-assessment of knowledge was correlated with their averaged certainty for each context. Due to the data being not normally distributed, Spearman’s rho (rS) was used for correlation analysis.
Considering the quantitative data of the second Delphi round is essential for generating a benchmark for assessing procedural knowledge. Looking at the SD effectivity of the insects and pollination context, effectiveness corrected by certainty estimations (weighted) ranges from 2.21 to 3.71 (Table 2
). The range of weighted effectiveness for the 10 items of peatland use is from 2.54 to 3.50 (Table 3
). The mean SD effectivity across all solution strategies of each context amounts to 3.01 for insects and pollination and 3.01 for peatland use, too. The standard deviation in the insects and pollination context is about 0.10 lower (0.64) than in the peatland use context (0.74). The means and standard deviations of the individual fields of action are shown in Table 2
and Table 3
. The solution strategies are presented in ascending order with respect to weighted effectiveness.
In contrast to the presented results of weighted effectivity estimations, the following data analyses are conducted with unweighted effectivity estimations (see Section 2.4
)—if not indicated otherwise. In the following table, differences between the means of the single fields of action of sustainable land use, ecosystem services, and biodiversity conservation or climate protection are presented. The repeated measures ANOVA (partially with a Greenhouse–Geisser correction because of a lack of sphericity) reveals five statistically significant differences in solution strategies for the insects and pollination context and two for peatland use (Table 4
). The respective effect sizes (partial η²) indicate large effects (Table 4
; ≥0.01–<0.06: small, ≥0.06–<0.14: moderate, ≥0.14: large, cf. [67
]). Also, four solution strategies show tendencies (≤0.10) (Table 4
). The Bonferroni-adjusted post hoc analysis revealed eight statistically significant differences between the fields of action (marked in Figure 4
and Figure 5
, cf. Appendix C
For the insects and pollination context, the item “Align the breeding of honeybees for resistance to disease and parasites” (IP-1) is rated to be more effective for ecosystem services than for the realization of sustainable land use and biodiversity conservation (Figure 4
). The item “Realign agricultural subsidies to stop promoting conventional and intensive agriculture” (IM-8) is rated to be more effective concerning sustainable land use than the provision of ecosystem services and biodiversity conservation. The item “Design a cultural landscape to serve pollinators as food source and habitat” (IP-9) is rated to be more effective in biodiversity conservation than in ecosystem services. In contrast, the item “Strengthen the protection of wild bees and other pollinating insects” (IP-10) does not differ between biodiversity conservation and ecosystem services. However, IP-10 is rated more effective for biodiversity conservation than for sustainable land use, which does not account for IP-9 (Figure 4
For the peatland use context, the items “Cultivate peatlands without fertilizers and pesticides” (PU-6) and “Apply existing laws stricter, e.g., prohibit the converting of grassland into maize cultivation” (PU-9) are rated to be more effective in sustainable land use than in climate protection (Figure 5
In short, the results in the essential fields of action of sustainable land use, ecosystem services, and biodiversity conservation or climate protection differ partly. Thus, the individual consideration provides more information than the SD effectivity.
Reliabilities were calculated with unweighted effectivity estimations for consistency of data analysis. Cronbach’s α values for the insects and pollination context indicate satisfactory (biodiversity conservation = 0.758) to high reliabilities regarding fields of action (sustainable land use = 0.838, and ecosystem services = 0.887) (Table 5
). In the peatland use context, the reliability of the fields of action ecosystem services and climate protection is acceptable (0.763 and 0.764, respectively); the reliability of sustainable land use is relatively low (0.572). The reliability over both contexts in sustainable land use (0.852) and ecosystem services (0.906) indicate reliable scales (cf. [68
]). For the reliability analyses, a weighting with the averaged certainty of the 10 solution strategies per context is a feasible procedure. The results differ by less than 0.01 from the reported values—except for the peatland use context (see Cronbach’s α values in brackets in Table 5
with, e.g., 0.633 instead of 0.572 for sustainable land use).
For validation purposes, the certainty of estimations in the two contexts from the experts was correlated with their fitting self-assessed knowledge. Therefore, special variables were computed. Regarding the insects and pollination context, the mean of a variable combining self-assessed knowledge on biodiversity, bees and pollination, and sustainable development was calculated and correlated with the averaged certainty about all 10 solution strategies. The correlation is large and statistically significant (rS
= 0.73, p
< 0.001; r > 0.5: large (cf. [68
])). A large effect and significance could also be identified for the correlation of averaged certainty about all 10 solution strategy items from the peatland use context, with the mean of a variable consisting of self-assessment concerning the importance of peatlands, climate change, and sustainable development (rS
= 0.53, p
= 0.016). Thus, participants who assessed their knowledge as lower seemed to be less confident in answering the corresponding solution strategies. The results underline the validity of the expert certainty estimations.
Here we present a refined procedure to measure procedural knowledge with a multi-level Delphi approach. The two-round Delphi survey with an intermediate think-aloud study enabled us to develop a measure for procedural knowledge including a benchmark to evaluate such knowledge of student teachers in the future.
The procedure consisted of using solution strategies for SD challenges according to the literature [58
], estimations of such solution strategies and further suggestions from experts in the first Delphi round and item revisions due to the results of a think-aloud study with student teachers. The resulting measure contains 20 items reflecting the SD challenges of biodiversity loss and climate change. In the second Delphi round, the 20 solution strategies were estimated concerning effectivity with respect to three fields of action. Furthermore, the certainty of three effectivity estimations per solution strategy item was judged by the experts.
Regarding the sample composition of the Delphi study, experts from a broad range of expertise throughout Germany participated (Table 1
). In addition, non-university experts contributed (e.g., from the Ministry of Environment and from a research institute). The fields of expertise cover multiple perspectives with respect to ESD, SD, biodiversity, insect and pollination, climate change, and peatland use. Furthermore, the degree of expertise was very high, comprising 15 professors among the 20 participants of second Delphi round.
In contrast to an expert study, the two-round Delphi study allowed us, for example, to integrate new solutions strategies in the second Delphi round that were suggested by the first round participants. In the present study, the solution strategies “Realign agricultural subsidies to stop promoting conventional and intensive agriculture” (IP-8, see Table 2
) and “Provide agricultural subsidies only for sustainably managed peatlands” (PU-10, see Table 3
) were newly proposed in addition to the 41 solution strategies provided by the authors in the first Delphi round [58
]. These two solution strategies were integral components in the second Delphi round questionnaire. Furthermore, the Delphi procedure chosen includes that the participants in the second round can work with the expert knowledge from the first Delphi round, e.g., the graphically processed results of the questionnaire, judging upon revised solution strategies. In sum, the two-round Delphi procedure enriches the knowledge base for the assessment of—in the present study—SD-relevant interdisciplinary knowledge.
Strengths of the performed Delphi study are: (i) the great number of participants, (ii) the high degree of (scientific) expertise of the participants, (iii) the repeated questioning of the experts, and the possibility of the experts to suggest further solution strategies in contrast to other expert studies aiming at establishing benchmarks for assessing procedural knowledge [27
]. Another strength of the Delphi study presented is the diversity of the disciplines of the participants involved, which enriches the input given with respect to SD challenges. The expertise behind the measure of procedural knowledge for student teachers proposed in Table 2
and Table 3
, as well as the respective benchmark information for evaluating such knowledge, is deeply integrated in the measure. The presented measure resulted from a two-round Delphi study instead of a single data collection.
Another aspect of the Delphi study is the weighting of the experts’ answers. Despite being legitimated through the use of two different contexts [66
], the weighting can lead to a bias, because potentially self-critical persons chose a moderate confidence level despite high knowledge while self-confident persons chose a high confidence level [70
]. However, tests have shown that subjective certainty builds an indicator for the quality of estimations [59
]. An alternative method could be to only consider experts with high self-assessments [66
]. The latter would lead to different sample sizes in the two contexts because the self-assessment concerning biodiversity and bees is lower than that concerning climate change and peatlands. However, in the present study, weighting the estimations through certainty seems to be the best method for considering the individual knowledge of the experts. Besides, one participant with evidently biased responses was excluded from the analyses of the second Delphi round (see Section 2.1
). Applying the method of using weighted effectiveness estimations of solution strategies through considering certainty estimations is also supported by the fact that experts with lower self-assessed knowledge seem to be less confident in answering the solution strategies.
To determine the benchmark, it is recommended to exclude one item for each context. For the insect context, the item “The legislator prohibits application of neonicotinoids” (IP-6) should be excluded because, on 27 April 2018, the EU Commission prohibited the use of neonicotinoids based on the substances clothianidin, thiamethoxam, and imidacloprid. On the same day, the invitation to participate in the second Delphi round was sent to the experts. Because most abundant neonicotinoids are prohibited, this is no longer possible but rather an implemented strategy and therefore negligible. Solution strategy IP-6 was one of three items of the overarching topic “environmental policy”. Accordingly, this topic will still be represented.
Also, the item “Individuals purchase products only from sustainable peat extraction” (PU-2) should be excluded from the prospective benchmark. Contrasting all other items, solution strategy PU-2 does not derive from the literature review. The original objective was to create an ineffective item. Because of the very slow growth rate of peat, despite ideal conditions, sustainable extraction of peat is virtually impossible. The exclusion of PU-2 no longer represents the overarching topic of “sustainable consumption”. Contrasting, the relevance of sustainable consumption in the insects and pollination context, this topic could be neglected in the peatland use context, as the peat consumption is not as widespread as the use of agricultural products requiring pollination services. By eliminating items IP-6 and PU-2 from the benchmark, all remaining 18 items are valid regarding the actual legislation as well as the literature base. In addition, they cover a wide range of essential overarching SD-relevant topics. By extracting the two items, the reliability of expert ratings only slightly differ—except for sustainable land use in the peatland use context (Table 4
). All in all, for the measure of procedural knowledge, the reliabilities of the prospective data of student teachers is important.
As a result of the project, a final item set for a measure for interdisciplinary procedural knowledge concerning CK for ESD is provided. It includes two contexts concerning SD challenges. The insect and pollination and the peatland use contexts cover 18 items in total (Table 2
and Table 3
, without IP-6 and PU-2) for a differentiated assessment of procedural knowledge in three essential fields of action: the realization of sustainable land use, provision of ecosystem services, and biodiversity conservation or climate protection. These fields of action are all interdisciplinary and/or multidisciplinary. Within the fields of action, the items either have clear foci (e.g., PU-8: “Intensify the investigation of regenerative peat substitutes”) or cover broader ESD requirements beyond the assessed fields of action (e.g., IP-10: “Strengthen the protection of wild bees and other pollinating insects”). Thus, the two-round Delphi study, complemented by an intermediate think-aloud study with student teachers, turned out to be a suitable procedure for generating an instrument with an included benchmark for measuring SD-related procedural knowledge of student teachers.
The developed measure as well as the measurement instrument can likewise be applied to other target groups, such as students in higher education in academic fields such as biology, geography, and sustainability, as well as in secondary school education for students. The chosen contexts of insects and pollination and peatland use are suitable to European countries and even other continents. When students receive the questionnaire (without item IP-6 and PU-2), different evaluation procedures of the benchmark are possible: to correlate the answer profile for each student teacher with the profile of the experts (cf. [27
]) or to compare the ranking of effectiveness between experts and student teachers (cf. [69
]). Furthermore, the innovative procedure used to develop the measure can be applied to cope with the challenges attached to procedural knowledge for further contexts.
In the future, it would be conceivable to use a shorter, and thus time-efficient, instrument by, for example, recording only one of the three fields of action. However, it is too early in the process of instrument development to recommend such a condensed version. Regarding the experts, there are eight significant differences out of 60 answers to the three fields of action. This might be explained by the interdisciplinary knowledge needed. However, it is still unclear how student teachers assess the different fields of action. Therefore, in order not to lose important information, the differentiation between the three fields of action will be kept. In our research, the measure of procedural knowledge is one of three parts to evaluate interdisciplinary knowledge of student teachers concerning biodiversity and climate change issues (see Section 1.2
). The complete measurement instrument to evaluate situational, conceptual, and procedural knowledge will be applied in a survey with student teachers (starting in October 2018, n
= 300; cf. Figure 1
Based on the present study, recommendations on how to develop a measure for procedural knowledge, that reflects expert knowledge as well as student teacher perceptions, can be derived from the procedure demonstrated. The sophisticated procedure integrates a Delphi study and a think-aloud study. Furthermore, the relevant expert data for the benchmark (means, standard deviations, and ranking of effectiveness estimations) for using the presented measure for evaluating the procedural knowledge of student teachers are provided. Thus, the present study fills in the gap on how to assess SD-relevant procedural knowledge. In addition, to provide a refined procedure for developing a measure for procedural knowledge, the article specifically supplies a measure for procedural knowledge regarding the SD challenges biodiversity loss and climate change.