Next Article in Journal
Preparing an Organization for Sustainability Transitions—The Making of Boundary Spanners through Design Training
Next Article in Special Issue
Towards Sustainable Development in Education: Implementing a VET System for In-Service Teachers in Albania
Previous Article in Journal
Effects of Climate Change on Cultivation Patterns and Climate Suitability of Spring Maize in Inner Mongolia
Previous Article in Special Issue
School-Based Training for Sustainable Emotional Development in Chinese Preschoolers: A Quasi-Experiment Study
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:

Evaluating STEM-Based Sustainability Understanding: A Cognitive Mapping Approach

RAND Corporation, Behavioral and Policy Sciences Department, Arlington, VA 22202, USA
Arthur J. Bauenerfeind College of Business, Murray State University, Murray, KY 42071, USA
Department of Social Science & Cultural Studies, Montana State University Billings, Billings, MT 59101, USA
Geosciences Department, University of Arkansas, Fayetteville, AR 72701, USA
Authors to whom correspondence should be addressed.
Sustainability 2021, 13(14), 8074;
Received: 22 June 2021 / Revised: 13 July 2021 / Accepted: 15 July 2021 / Published: 20 July 2021
(This article belongs to the Special Issue Advances in Evaluation of Sustainable Educational Programs)


Management education holds promise for addressing deficiencies in interuniversity science, technology, engineering, and mathematics (STEM), as well as sustainability curricula. Accordingly, we designed, developed, implemented, and longitudinally evaluated interdisciplinary STEM-based curricula in the United States. Students in five sections of business management courses and two sections of STEM courses received a STEM-based sustainability intervention (i.e., an interdisciplinary STEM and sustainability module). To assess student outcomes following the intervention and examine the feasibility of cognitive mapping as a student learning assessment tool, we implemented a pre- and post-course modified cognitive mapping assessment in treatment and comparison courses. To interpret the results, we ran descriptives, correlations, paired sample t tests, and principal component analysis. The t tests suggest that when all coding categories are considered, those participating in curricular interventions listed significantly more sustainability terms. The principal component analysis results demonstrate that treatment courses improved variability explained by 7.23% between pre- and post-tests but declined by 8.22% for comparison courses. Overall, linkages became stronger between parent code categories for treatment courses and weaker for comparison courses. These findings add to existing research related to cognitive mapping and demonstrate the ability of the method to capture changes in student outcomes after exposure to STEM-based sustainability curriculum.

1. Introduction

Challenges remain in interdisciplinary sustainability education across business and science, technology, engineering, and math (STEM) disciplines [1,2]. The integration of STEM is rare in business courses, just as the integration of business concepts is rare in STEM courses [2]. Educators use sustainability as a concept to integrate STEM into curricula across disciplines [3]. However, many sustainability curricular initiatives involving multiple disciplines (e.g., management, STEM) result in multi-disciplinary curricula either by design or in part due to instructors developing curricula in disciplinary silos [3,4,5,6]. Multi-disciplinary curricula, as opposed to interdisciplinary, create challenges because students view sustainability as outside the normal realm of their own discipline and they cannot draw a clear connection to others’ perspectives about sustainability [5,7]. Yet, complex sustainability-related challenges (e.g., climate change) will require current and future leaders, regardless of disciplinary background, to demonstrate and apply an interconnected, interdisciplinary understanding of the challenges of using STEM competencies [8,9,10,11].
To address deficiencies in interuniversity STEM and sustainability curricula, we designed, developed, implemented, and longitudinally evaluated interdisciplinary STEM-based curricula at a western university in the United States. Students in five sections of business management courses and two sections of STEM courses received a STEM-based sustainability intervention (i.e., an interdisciplinary STEM and sustainability module). STEM-based sustainability integrates STEM competencies with and across component areas of sustainability [2]. To improve the validity of the evaluation, we included a comparison cohort of two business management course sections and one STEM course section in the evaluation. In total, 167 students completed pre- and post-tests to assess the impact of the intervention.
A challenge associated with sustainability curricula design and development is evaluation. Measurement tools that surpass self-reported knowledge are lacking, which prompts researchers to find new ways of measuring sustainability proficiency [12,13]. For example, one element of criticism is that students’ lack the capability to “integrate knowledge and apply it to complex problems” [14] (p. 6). One such method of assessing STEM-based sustainability understanding is through cognitive mapping [15]. Cognitive mapping entails diagraming individual mental models about a topic [16], producing graphical representations that demonstrate breadth (e.g., the number of terms students associate with sustainability) and depth of understanding (e.g., the strength of relationships between sustainability terms). A growing body of literature cites cognitive mapping, also referred to as “concept mapping,” as a useful tool for educational evaluation [17,18,19,20]. Accordingly, we build on cognitive mapping literature to explore the ability of the method to capture changes in student outcomes after exposure to STEM-based sustainability curricula. Next, we provide select literature related to STEM and sustainability education, and additionally to cognitive mapping. Following this, material and methods, results, discussion, and conclusion sections are provided.

1.1. STEM and Sustainability

Businesses will encounter increasing climate variability and shifting weather patterns in the future because of climate change. Business leaders may face challenges related to the degradation of the natural environment, market challenges, moral dilemmas, and other sociopolitical issues [21,22]. Considering the magnitude of climate change, future leaders need to navigate the complex impacts from these changes and respond appropriately to innovate, coordinate with other institutions, and change operations as needed.
Sustainability provides a lens through which business leaders can address grand challenges related to climate change [23]. Sustainability in business should explicitly integrate STEM competencies [1,2], yet STEM is often missing from business curricula [2]. With interdisciplinary STEM-based sustainability curricula, business students’ sustainability literacy [14], cognitive abilities [24], and affective outcomes [18] related to STEM and sustainability can improve. Understanding the interconnected relationships between economic, environmental, and social domains is essential to address the goals of doing business, protecting resources, and ensuring fairness across groups [25,26].
In recent years, government and non-profit organizations have increasingly supported STEM education [27,28]. Unfortunately, this momentum has not yet reached business schools. Relatedly, sustainability programs are often available to students across curricula, but tend to overlook business students [29]. In fact, only 179 out of the 2809 sustainability academic programs registered with AASHE as of November 2020 are in a business discipline [29]. We consider that taking on a STEM-based sustainability approach, integrating sustainability literacy (i.e., knowledge) and understanding (i.e., cognition) with the four STEM components (i.e., science, technology, engineering, math), is important to prepare business students for the future. Accordingly, we address gaps in research and practice that are absent from interdisciplinary STEM-based sustainability curricula.

Project Background

Three courses at a western university in the United States received curricular interventions (i.e., an interdisciplinary STEM and sustainability module) including two sections of Business and Environment (n = 32), three sections of International Business (n = 39), and two sections of Human Geography (n = 39). Business and Environment and International Business are upper-level requirements for business management majors and electives for other business majors. Human Geography is a STEM general education elective that fulfills a university cultural diversity requirement. Comparison courses, or those that did not receive the curricular intervention, were also evaluated including one section each of Business and Environment (n = 21), Human Geography (n = 15), and Management and Organization (n = 21).
We selected participating courses to introduce interdisciplinary (1) STEM and sustainability curricula in management education and (2) business and sustainability curricula in STEM education. The student outcome of interest, regardless of discipline, was sustainability cognition (i.e., the process of acquiring sustainability knowledge or understanding through the experience with the focal curriculum). To overcome challenges with designing, developing, and implementing interdisciplinary curricula [2,5] each of the instructors adapted course syllabi to include a common learning objective to: “define, explain, and apply economic, environmental, and social components of sustainability using STEM-based evidence” [30]. Module requirements included:
inclusion of the three sustainability dimensions;
inclusion of the four STEM dimensions; and
the design, development, and implementation of an original problem-based case study with teaching manual.
The cases for Business and Environment [31] and International Business [32] are published, and the case for Human Geography [33] is available as a full-length book. The module requirements ensured adaptation of the course-level common learning objective, interdisciplinary inclusion of STEM and sustainability, and consistent implementation. Below, pertinent literature about the evaluation method for student responses to the STEM-based sustainability curricular interventions is discussed.

1.2. Cognitive Mapping

In general, cognitive mapping has two direct applications: support for learners and evaluation for instructors. This study prioritizes cognitive maps as a sustainability education evaluation tool. Originally developed in the 1980s to characterize causal reasoning, cognitive maps are diagrams that represent the organization of knowledge [34,35]. Most of the literature examines cognitive mapping in K-12 and postsecondary educational settings with a small group of participants. The development of cognitive mapping in the education landscape originated in researchers’ interest in how children develop understanding of science concepts [36]. Cognitive mapping is a proven, useful tool for building and examining understanding of additional STEM and sustainability concepts [15,37].

1.2.1. Characteristics of Cognitive Maps

Three key inputs build a cognitive map: focus questions, nodes, and links [15]. Focus questions prompt participants to construct their maps. For example, focus questions may ask, “what is global climate change,” “what is the evidence [related to global climate change],” or “what are the consequences [of global climate change]” [38] (p. 359). Focus questions can also prompt participants to connect nodes, depicting the interconnectedness of relationships with linkages. In addition to asking questions, evaluators might also alternatively choose to propose a broad idea (e.g., sustainability, the environment, energy, etc.) to students. Thoughtful focus questions can enhance the richness of linkages and the complexity of concepts represented. A poor focus question can result in cognitive maps which do not fully answer the question, or which develop into off-topic responses [39].
By using focus questions to guide thinking, participants can express their thoughts with nodes to convey a list of related concepts, forming a preliminary map, absent linkages. In this step, students provide words that are used to clarify their understanding of the topic (i.e., guided by the focus question) [15]. During the cognitive mapping process, participants graph nodes (or terms), following the direction provided by a focus question. Nodes should relate to the main topic of the focus question and are generally one word or short phrases. Next, students work toward structuring or organizing their terms. This is traditionally done by participants during an in-person session with a researcher where they are asked to connect, or link, concepts to indicate relationships. The linkages between various concepts provide learners an opportunity to operationalize a hierarchical understanding of course curricula or learning experiences.
The next step in the process is the analysis of completed maps. Conceptual understandings of greater nuance correlate positively with map complexity [39]. Research indicates that early instruction or learning experiences may construct shallow cognitive maps that reflect “naïve theories” [40] (p. 52). Naïve theories are misconceptions born of little first-hand experience. Monitoring cognitive maps over time can enhance instructor understanding of the learning pathways students take from naïve theories to more complex knowledge structures [40]. Sellmann et al. [20] describe the importance of identifying preconceptions and how cognitive mapping, done prior to a given curricular intervention, can help shape its curriculum. Cognitive maps can reveal knowledge gaps and naïve theories to instructors, who in turn can specifically target these misconceptions through direct instruction. Other studies demonstrate the usefulness of this tool in measuring knowledge growth, particularly for academically underachieving students e.g., [41]. To maximize meaningful learning and knowledge growth, the maps should be continuously revised as new knowledge is acquired, assimilated, or modified, underscoring Novak’s long-held conception of learning as an ongoing process [39].

1.2.2. Cognitive Maps in STEM Settings

Cognitive mapping plays a particularly interesting role in the field of sustainability because the tool can effectively capture the field’s interdisciplinary nature. Lourdel et al. [15] conducted one of the first sustainability studies among undergraduate engineering students. They administered a mapping task to students at the end of a training session. Students first wrote down concepts related to the stimulus (i.e., sustainable development) and connected those concepts with arrows to signify relationships. During analysis, researchers coded student responses for semantic categories (e.g., social–cultural, environmental, multidimensional, economic/scientific/technological, procedural/political, and actors/stakeholders). Of these, social–cultural, environmental, and economic/scientific/technological categories “gather the nominal and concrete approaches of the concept” of sustainability [15] (p. 171). Analysis of these categories, then, determine a respondent’s ability to translate sustainability concepts into concrete ideas. The remaining categories, however, nonetheless accomplish different goals with respect to the sophistication of concepts. Multidimensional concepts concern “the capacity of abstraction from students” [15] (p. 172). For example, exposure to procedural and political content may promote the understanding of government actions or events related to sustainability. Finally, the actors/stakeholders category acknowledges the participatory dimension of sustainability [15]. In reviewing a respondent’s cognitive map and making an accounting thereof defined by these different categories, insights are revealed about respondents’ systematic vision of sustainability [15].
Linkages are analyzed, as holistic relations, by measuring the quantities of links to certain words. Our findings indicated students’ understanding of sustainable development became richer and more dynamic. However, the authors note limitations associated with this methodology. For example, data encoding may be subjective. Additionally, respondents were not asked to delineate the type of connection or relationship between nodes, as in other cognitive mapping activities; such identification of these relationships could lead to richer data analysis. Lastly, it is nearly impossible to create a standard map to compare against, which limits generalizability [15].
In a subsequent study, evaluating the feasibility of cognitive mapping in sustainability-focused engineering courses, Segalas et al. [42] expanded on Lourdel et al. [15] with a more substantive approach to data analysis. This study analyzed both the quantity of nodes and quantity of inter-linkages. Dimensions of data analysis included degree level, course, elective vs. compulsory enrollment, and pedagogies employed (e.g., lecturing, role play, workshops, distance education). Also, broader semantic categories were included, leading to a greater variety in student responses. Appendix A includes a comparison of the categories used for analysis in Lourdel et al. [15] and Segalas et al. [42]. In assessing pre-and post-instruction cognitive maps, Segalas et al. [42] noticed increased complexity in cognitive mapping of sustainability, which they concluded was due to an increased understanding of sustainability from a holistic and systemic perspective. Another follow-up study evaluated expert and student orientations towards sustainability using cognitive maps [37]. Students understood sustainability as a scientific issue solved by technological innovation, whereas experts understood sustainability in the context of the long-term effects. This mismatch suggests course curricula could benefit from being tailored toward the sociological contexts of sustainability.

1.3. Research Questions

This evaluation of STEM-based sustainability curricula using cognitive maps is guided by sustainability-related research and the cognitive mapping process [15,16,27,42]. Comparable to Segalas et al. [42], we used a pre- and post-test evaluation design around a sustainability-related curricular intervention, though the nature and scale (i.e., 167 students, in eight course sections, in different colleges, completing cognitive mapping exercises at two time points) of our interuniversity curricular interventions necessitated some adaptations. Specifically, brainstorming to generate nodes (i.e., terms) occurred via an online survey in coordination with quantitative evaluation items (see [2] for quantitative explanations and findings). The online format required introducing an alternative step in the process where: (1) nodes were structured into sustainability categories based on data coding (i.e., a qualitative method); and (2) linkages among categories were established using principal component analysis (i.e., a quantitative method). Unlike previous studies, this adapted method explores the strengths and directionality of linkages rather than the quantity of linkages alone. This is the first known study in which researchers, not participants, quantitatively established linkages. With respect to introducing STEM-based sustainability curricula in business management education and expanding the scale to include interuniversity curricular interventions, we pose the following research questions:
  • Research Question 1: Do the cognitive maps of sustainability of students enrolled in treatment courses significantly change after receiving STEM-based sustainability curricular interventions?
  • Research Question 2: How do the cognitive maps of sustainability of students enrolled in treatment courses change after receiving STEM-based sustainability curricular interventions?
  • Research Question 3: Is this adapted cognitive mapping process a viable method for evaluating interuniversity STEM-based sustainability curricular interventions?
In the following section, we discuss study design, data collection, and analysis.

2. Materials and Methods

We administered pre- and post-test surveys via Qualtrics before and after curricular interventions. We collected responses for business and environment and human geography courses in fall 2018. Data from international business courses, including relevant comparison courses, were collected in spring 2019. The curricula interventions (also referred to as modules) were anchored by original case studies [31,32,33]. Modules were delivered in courses over approximately a three-week period. Twenty-two students enrolled in both business and environment and international business courses and were omitted from the study. To ensure fidelity to the treatment courses, similar comparison courses did not receive a curricular intervention (see Table 1 for treatment and comparison cohort demographics). The comparison cohort helps demonstrate the sustainability knowledge outcomes for students not exposed to STEM-based sustainability content. This approach can serve as a viable alternative when it is not possible to completely randomize participants within a program for examining causal impact.

2.1. Cognitive Mapping Exercise

We adapted Lourdel et al.’s [15] initial focus question to prompt student brainstorming to list nodes. Students received the following prompt:
We are interested in how you think about the term “sustainability.” On the next page, you will have 2 min to list as many words as possible that you associate with the concept. At the end of 2 min, the survey will automatically go to the next question.
On the next page of the survey, students filled in as many words as they could related to sustainability. We pre-tested the focal question in fall 2017 and spring 2018 with 253 students at two U.S. universities. Only five students skipped the question or left the response prompt blank. Given the small amount of missing data, we made no changes to the question prompt or format. One notable difference between ours and Lourdel et al.’s [15] question is that we opted not to ask students to provide linkages between sustainability concepts. Initially, Lourdel et al. [15] used a small sample of 10 students to pilot the method via an in-person exercise. Analyzing visual diagrams by hand was not feasible due to our larger sample size. In the following sections, we describe qualitative and quantitative analyses that provide a viable approach for answering the proposed research questions.

2.2. Qualitative Coding

After pre-testing the instrument in fall 2017 and spring 2018, we developed a codebook for analyzing student responses (see Appendix B for a full codebook with definitions of parent and child codes). Our final categories of codes included (1) social–cultural, (2) economic, (3) environmental, (4) intrinsic, (5) actions, (6) multidimensional, and (7) catch-all (for content that was relevant but did not readily fit into the other existing categories). Catch-all codes were removed prior to analysis. Within each parent category, we created child codes for greater granularity in student responses. We relied on Ritchie and Spencer’s [43] framework analysis technique. The framework analysis technique allows for identification of both a priori issues, or those informed by the original research aims and established literature, along with emergent issues raised by the respondents along the way [43,44].
The authors met to discuss the codebook and clarify definitions. A small number of transcripts were coded to establish interrater reliability using the training function in Dedoose, a qualitative analysis software program [45]. The team tested code application for the codebook, calculating a pooled Cohen’s kappa coefficient and Cohen’s kappa for each code. The team met four times, discussed discrepancies, and established new tests until each team member exceeded a Pooled Cohen’s kappa coefficient of 0.7 or higher. While there are multiple ways to evaluate the significance of a Cohen’s kappa value, Landis and Koch [46] suggest a score of 0.61–0.8 is within a range of good agreement. With acceptable agreement in place, each author coded a subset of the student transcripts, with one author coding more than half of the total pre- and post-test student responses. Additional clarification of unique terms occurred in real-time, and the research team kept a running list of example STEM-based sustainability terms to further exemplify each parent code category.
Each student response was coded to determine the semantic categories used. Sorting into categories provided descriptive information surrounding the breadth and depth of student responses, documented by terms applied across categories in addition to the total number of terms applied. In turn, we can glean insights into changes pre- and post-curricular interventions, in addition to noting the impact to treatment courses in contrast with comparison courses.

2.3. Statistical Analysis

After coding student responses, we ran descriptives and correlations for all codes and for each parent code category, sorted by pre- and post-tests, for both treatment and comparison courses (see Table 2). To answer research question one, we used SPSS v. 25 to conduct paired sample t tests sorted by treatment and comparison courses (see Table 3). Paired sample t tests compared student means to determine whether there was statistical evidence that the mean difference between paired observations (i.e., between the pre- and post-test) was significantly different from zero. Maps in Figure 1 and Figure 2 depict changes in mean values in parent code categories from the pre- to post-tests for treatment and comparison courses.
To provide additional insight into research question one (i.e., whether or not the maps of those student enrolled in treatment courses significantly change) and to address research question two (i.e., how do the maps of those student enrolled in treatment courses change), we used SPSS v. 25 to conduct principal component analysis with varimax (orthogonal) rotation to develop linkages for the parent code categories, including relational maps (see Figure 3, Figure 4, Figure 5 and Figure 6). Principal component analysis is a form of multivariate analysis that seeks to determine which components explain the most variance among a set of data and to reduce components to a sub-set of factors highly representative of the set of data [47]. Varimax rotation redistributes the variance among the initially extracted factors to optimize the variance explained by each. PCA has applications “in many fields such as energy, multi-sensor data fusion, materials science, gas chromatographic analysis, ecology, video and image processing, agriculture, color coating, climate and automatic target recognition” [48] (p. XI). PCA has also been used in educational settings to assess the dimensionality of teacher and learner characteristics [49,50,51]. Given the broad applicability of the method and our focus on the dimensionality of student sustainability cognition, the use of PCA is appropriate.

3. Results

3.1. Paired Sample t Tests

We ran paired sample t tests for all codes and then sorted by parent code categories to explore how student cognitive maps for semantic categories changed from pre- to post-tests (see Table 3). For the treatment group (n = 110) there was a significant difference observed for all codes between pre- (m = 8.82) and post-tests (m = 11.06, t = 5.20, p = 0.000). For the comparison group (n = 57) there was no observed significant difference between pre- (m = 9.23) and post-tests (m = 9.28, t = 1.14, p = 0.259). The results demonstrated that, when all the coding categories were considered, those participating in curricular interventions listed significantly more sustainability terms than those who did not. There was no significant difference for the actions, environmental, or intrinsic parent code categories for the treatment group (n = 57). However, there were significant differences between the pre- and post-tests for economic (t = 3.57, p = 0.001), multidimensional (t = 2.64, p = 0.009), and socio-cultural (t = 5.04, p = 0.000) parent codes. There were no observed significant differences in the comparison group between the pre- and post-tests.

3.2. Principal Component Analysis

We ran principal component analysis using varimax rotation, sorting by pre- or post-test, for treatment and comparison courses, to develop linkages for parent code categories (see Table 4). For treatment courses, we reduced the six parent code categories (i.e., all codes) to two-factor solutions for pre- and post-tests, explaining 49.94% and 57.17% of the variance, respectively. For comparison courses, we reduced all codes to a three-factor solution for pre-tests and a two-factor solution for post-tests, explaining 67.90% and 59.68% of the variance, respectively. Solutions only included components with eigenvalues greater than one. The results demonstrate that treatment courses improved the variability explained by 7.23% from pre- to post-tests but declined by 8.22% for comparison courses. That is, overall linkages became stronger between parent code categories for treatment courses and weaker for comparison courses.
The factor coordinates in Table 5 represent correlations between a variable (i.e., a parent code category) and a factor axis. The coordinates are mapped in Figure 3, Figure 4, Figure 5 and Figure 6, depicting the linkages (i.e., correlations) that each parent code category shares with the factor. For treatment course pre-tests, Factor 1 represents 25.99% of all codes, where the codes economic and social–cultural are positively linked to the factor and the code multi-dimensional negatively linked. Factor 2 represents 23.95% of all codes, where codes environmental and action are positively linked to the factor and the code intrinsic negatively linked. On post-tests, Factor 1 improved, to represent 36.40% of all codes, where economic, environmental, and social–cultural are codes positively linked to the factor and intrinsic negatively linked. Factor 2 represents 27.11% of all codes, where actions and multidimensional are codes positively linked to the factor and, again, intrinsic is negatively linked.
For comparison group pre- pre-tests, Factor 1 represents 27.48% of all codes where economic, environmental, and social–cultural are positively linked to the factor. Factor 2 represents 20.27% of all codes where intrinsic and multidimensional are positively linked to the factor and environmental negatively linked. Factor 3 represents 20.16% of all codes where the action and multidimensional are codes positively linked to the factor. For post-tests, Factor 1 represents 42.16% of all codes, where economic, environmental, and social–cultural are codes positively linked to the factor, and intrinsic is negatively linked. Finally, Factor 2 represents 17.52% of all codes, where the codes action and multidimensional are positively linked to the factor.

4. Discussion

The complexities of sustainability-focused education in management education presents systems-level challenges [52] that STEM can address. However, the lack of communication and collaboration across disciplines hinders the implementation of interdisciplinary curricula [27] such as the STEM-based sustainability curriculum presented in this study. Organizations, both for-profit and non-profit, face complicated sustainability issues within and across the three key aspects of sustainability [8] that require a STEM- and sustainability-literate workforce. Yet, there are relatively few sustainability programs in business disciplines and an interdisciplinary STEM-focus is lacking from program offerings [29]. Business management content is also seldomly covered in STEM curricula [2]. The need for, and absence of, STEM-based sustainability education presents an opportunity for “an alternative vision of management education as a progressive educative practice: one that embraces our embeddedness in the natural world and our social relation to one another” [53] (p. 437). To address curricular gaps devoid of STEM and sustainability in management education, we developed, implemented, and longitudinally evaluated a STEM-based sustainability curriculum in three courses representing business management and STEM disciplines. We used cognitive maps to answer our three research questions.

4.1. Research Question 1: Do Student Cognitive Maps in Treatment Courses Significantly Change?

In the aggregate, findings from the all coded paired-sample t tests provide evidence that there is a significant change in cognitive maps from treatment courses (t(109) = 5.20, p = 0.000) between pre- and post-tests (see Figure 1 and Figure 2). These same changes did not occur in the comparison group that did not receive STEM-based sustainability curricula. There are significant changes exhibited for the parent code categories between pre- and post-tests for treatment students, including the economic (t(109) = 3.57, p = 0.001), multidimensional (t(109) = 2.64, p = 0.009), and social–cultural (t(109) = 5.04, p = 0.000) codes. Considering this, respondents demonstrated change in two of the concrete categories (i.e., economic and social–cultural) and one abstract category (i.e., multidimensional). Segalas et al. [42] used similar (though not identical) categories in their analysis and compared the categorical relevance (CR), or the distribution of concepts between categories, before and after sustainability-related engineering coursework. They, too, observed an overall improvement for the total number of concepts, though results were mixed and dependent on the coding category.
The lack of significant improvement for treatment students in the environmental code, which is a concrete sustainability concept, is a counter-intuitive finding, especially considering we explicitly integrated environmental sustainability into each module. Though, the finding is consistent with Segalas et al. [42], where student maps consisted of 24% environmental codes on the pre- pre-test but only 22.8% on the post-test. Many understand and operationalized sustainability in terms of the environment and environmental action (e.g., recycling, saving, composting) [13], so it may be that understanding of the environmental semantic category of sustainability was already well-developed among students. Furthermore, Kagawa [54] noted that students can perceive environmental components of sustainability as competitive with economic and social components, another possible explanation for improvement in these parent codes but not environmental.
Another counter-intuitive finding is that the mean number of intrinsic (i.e., values, attitudes, and beliefs) nodes listed did not significantly improve. Zwickle and Jones [13] found the link between sustainability knowledge and attitudes to be weak, thus, a possible explanation for the lack of change may be there was no clear link between knowledge and affect. Segalas et al. [42] similarly observed weak linkages between “soft” or abstract concepts (e.g., values) and sustainability on student cognitive maps. Unlike environmental sustainability, we did not adapt affective learning objectives in the implemented modules. Shephard [55] contends that instructors should target affective sustainability learning separate from cognitive learning—which has been the focus of our STEM-based sustainability curricula—and assign grades based on achievement of affective objectives. The adaptation of affective learning objectives, in coordination with co-curricular activities such as service-learning, holds promise for targeting improvement in and giving credit for affective sustainability learning [56,57]. Overall, tracking change in semantic categories clarified the overall ability to understand the systematic vision of sustainability. Further, we learned that treatment courses grasped some of the concrete and abstract aspects of sustainability but could still improve in other areas of sustainability.

4.2. Research Question 2: How Do Student Cognitive Maps from Treatment Courses Change?

We introduced an updated cognitive mapping process tailored for larger scales, featuring interuniversity curricular interventions where linkages between nodes were quantitively developed and explored by researchers, not participants. Unlike Lourdel et al. [15] and Segalas et al. [42], this update captured additional characteristics about linkages (e.g., strength and directionality of relationships) but not the quantity of those linkages. Using principal component analysis, we uncovered linkages between sustainability parent code categories and produced relational cognitive maps (Figure 3, Figure 4, Figure 5 and Figure 6). Inspecting maps and factor coordinates (see Table 5) between pre- and post-tests indicated that change had occurred for treatment and comparison students, and provided insight into those changes.
There are four key findings related to the linkages observed between the parent code categories. First, the overall strength of linkages shows a positive change induced by treatment courses and a negative change for comparison courses. This finding speaks directly to Research Question 1 and provides support for STEM-based sustainability curricula in business and STEM disciplines. Second, post-test maps from treatment courses demonstrate the strongest linkages among economic, environmental, and social–cultural categories. This suggests that respondents are making connections between the three concrete categories of sustainability. While the number of economic and social–cultural mean values significantly improved between pre- and post-tests for treatment students, the same was not true for environmental parent code categories. Unlike Kagawa’s [54] finding, however, treatment course factor coordinates demonstrate that environmental codes are not competing with the economic and social components on pre- pre-tests, rather they are unrelated. Factor coordinates on post-tests demonstrate that linkages changed where environmental codes became closely related to economic and social–cultural codes.
Third, the complexity of the post-test map for treatment courses increased, where abstract multidimensional codes transformed from negative linkage to positive linkage. Rather than competing with economic and social–cultural factors, as on pre-tests, post-test results showed multidimensional codes became complimentary with actions. In other words, respondents started to make connections between things like time, prevention, and future generations and activities, measures, or operations. Mean values of multidisciplinary codes increased (Figure 1) and the linkages shifted from negative to positive between pre- and post-tests for treatment students. Combined, the improvement in variability explained in addition to positive changes in environmental and multidimensional codes between pre- and post-tests for treatment courses suggest that meaningful learning occurred. Meaningful learning comprises the interrelationships of concepts, how they are applied, and how they are adapted requiring students to form active connections between course material [58,59].
And fourth, the intrinsic codes (i.e., values, attitudes, and beliefs) share negative relationships for both treatment and comparison courses on post-tests. Intrinsic codes are negatively linked to the three sustainability component factors for both treatment and comparison courses on post-tests and negatively linked to the factor including multidimensional and action codes for treatment courses. Ajzen et al. [60] contend that environmental affect is not related to environmental learning and Zwickle and Jones [13] note that sustainability knowledge is weakly related to sustainability attitudes. Our results suggest that cognitive learning, as a function of curricular interventions and/or learning that may have occurred from the cognitive mapping exercise, is negatively related to an affective connection with sustainability. These results reinforce the need to target affective learning about sustainability, independent of cognitive learning, to improve affective outcomes and facilitate a passion for sustainability [55,61].

4.3. Research Question 3: Using Cognitive Mapping to Assess Student Learning

Based on our results, we believe that our adapted cognitive mapping approach is a promising method to assess STEM-based sustainability student learning in business schools. As with any new or modified evaluation tool, any changes to an existing instrument should be tested to ensure retention of the utility of the instrument. The results from Research Questions 1 and 2 demonstrate that this method is capable of detecting differences both in the nodes provided by students and the strength and directionality of conceptual relationships. Therefore, our variation of the tool complements previous efforts from researchers [15,37] as a means of examining the understanding of STEM and sustainability concepts.
A strength of this study is that we explored ways of reducing burden, both to students and instructors who analyzed the data. For example, Lourdel et al. [15] (p. 172) acknowledge the challenge of “limited time” when collecting their data, and Rebich and Gautier [38] note that students in their study were given 45 min to complete their concept-mapping activity. Given that cognitive mapping may be one of many assessment items, instructors may not be able to devote limited class time to an extended cognitive mapping activity. However, even with limiting our procedure to 2 min, our results suggest that an abbreviated cognitive mapping approach is possible.
Additionally, our modified analysis strategies may be useful to other instructors who lack the resources to manually analyze cognitive maps by hand. For example, Lourdel et al. [15] examined the inter-linkages between categories and manually counted the total number of links between words produced by respondents. Other types of cognitive mapping software exist, which could be used to incorporate faster analysis of conceptual links, yet instructors may not have funds for additional software. Instructors could also face challenges, including research items that rely on specialized software for data collection otherwise not used in the evaluation of sibling items. Further, requiring students to access cognitive mapping software online creates barriers, particularly for students who may have limited internet access. Segalas et al. [42] and Segalas et al. [37] may have used software to collect cognitive map data from students, although this is not explicitly clarified. Using principal component analysis, we evaluated students in order to capture additional characteristics about linkages between sustainability parent code categories, without manually counting connections or relying on a separate data collection platform. Other instructors, in the future, could benefit from applying a similar approach to cognitive- or concept-mapping activities. Appendix C contains a one-page summary of cognitive mapping instructions, data preparation, data analysis, and interpretation to assist instructors interested in applying the exercise in the future.
Finally, as addressed by Lourdel at al. [15], classifying semantic terms can introduce subjectivity into processing data. We addressed this criticism by testing our codebook (Appendix A) by calculating a pooled Cohen’s kappa coefficient and Cohen’s kappa for each code. We also used the codebook to capture a priori categories for cognitive maps, which provided flexibility in capturing emergent issues [43,44]. For instance, unlike previous studies we identified the intrinsic category, noting individual affect has been previously linked to sustainability [62]. Our larger sample size (n = 167) afforded us the opportunity to build on these a priori categories and identify higher resolution parent and child codes which may be useful to educational evaluators in the future. Combined, we believe that our adapted cognitive mapping approach holds promise for assessing student learning, particularly among larger sample sizes, because it reduces the time and burden of data collection and data analysis, and it addresses a common criticism related to the subjectivity of semantic classification.

4.4. Limitations and Future Research

The study is not without limitation, warranting future research. First, while we expanded the sample to include multiple disciplines, the sample size was still limited (n = 167). Future studies should expand the use of cognitive maps for evaluation purposes across even larger samples. This would, over time, also assist with evaluating the reliability of the parent code categories. Additionally, future studies might employ the use of natural language processing software to reduce the burden of manual coding for evaluators. Additional testing of further automated analysis may be useful for further scaling up this approach in additional settings.
Also, there were curricular requirements to include STEM, business, and sustainability within an interdisciplinary context. However, the development and deployment of curricular materials was not systematic, and we were unable to assess changes based on curricular intervention type. Future development of curricular materials can use a more systematic approach, where changes in cognitive maps can be clearly linked (e.g., [42]). Future longitudinal studies should also implement mixed methodologies that include a combination of cognitive mapping, other qualitative methods (e.g., interviews), and quantitative STEM and sustainability knowledge/affect assessments. Future research could also use cognitive maps to evaluate and inform other STEM concepts and might explore differences concerning in-person versus online classes.
Lastly, there were notable differences in pre-test maps (Figure 3 and Figure 5) for treatment and comparison students. Unfortunately, the methods utilized in this study (e.g., t tests) preclude making causal inferences about why the differences emerged. Future researchers should consider using advanced methodology that controls for co-variates that could potentially influence differences in starting mental representations about sustainability.

5. Conclusions

Understanding the connections between economic, environmental, and social domains, and even beyond these, will be essential to address the goals of doing business, protecting resources, and ensuring fairness across groups in the future [25,26]. Complex sustainability-related challenges, such as climate change, require future decision-makers to demonstrate and apply an interconnected and interdisciplinary understanding of sustainability. Cognitive mapping is one technique to capture and evaluate concrete and abstract approaches to sustainability. As universities consider implementing interdisciplinary STEM-based curricula, using a cognitive mapping approach is one tool that could be used for educational evaluation. While this paper describes one interdisciplinary STEM-based curricula at a western university in the United States, educators can apply findings from this study to other efforts focused on improving STEM-based sustainability curricula, particularly in business and/or management programs. This study demonstrates how cognitive mapping can be employed to evaluate curricular interventions and suggests that the method is a useful tool for assessment.
This study found (Research Question 1) a significant change in treatment students’ maps pre- and post-test. Overall, we discovered that treatment students demonstrated more involvement with economic, multidimensional, and socio-cultural concepts but experienced no change with other sustainability categories. Further, (Research Question 2) we observed that the complexity of interrelationships between concepts improved for treatment students but declined for comparison students. Finally, (Research Question 3) we note that cognitive mapping shows how students conceptualize and organize knowledge in an open-ended format, offering an alternative to multiple choice or sheer memorization techniques and allowing students to integrate diverse higher-order constructs to develop metaphorical thinking [14,63]. Educators and evaluators can replicate this approach in future assessments of STEM-based sustainability curricula and may find additional ways of streamlining the method to reduce student and evaluator burden in the future.

Author Contributions

E.L.P.S. conceptualization, methodology, investigation, writing—original draft preparation, writing—review and editing, supervision, project administration, funding acquisition, validation; C.A.C. conceptualization, methodology, investigation, writing—original draft preparation, writing—review and editing, supervision, project administration, funding acquisition, validation, visualization; formal analysis; E.S. methodology, software, validation, formal analysis, investigation; G.G. methodology, software, formal analysis, investigation; S.G. conceptualization, writing—original draft preparation, writing—review and editing, supervision, project administration, funding acquisition; S.F. conceptualization, writing—review and editing, funding acquisition. All authors have read and agreed to the published version of the manuscript.


This research was funded by the National Science Foundation (United States): Grant Numbers 1726834, 1726278, 1726843, 1951290. Any opinions, findings, and conclusions or recommendations expressed in this article are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Review Board (or Ethics Committee) of RAND Corporation (protocol code 2017-0811-AM01, approved 27 September 2019) and Murray State University (protocol code 20-009, approved on 29 August 2019).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Data is not provided to maintain confidentiality of respondents in accordance with internal review board (IRB) policies at participating institutions.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

Appendix A

Table A1. Categories Used for Cognitive Map Analysis 1.
Table A1. Categories Used for Cognitive Map Analysis 1.
Lourdel et al. (2007)Segalas et al. (2008)
social–culturalresources scarcity
multidimensional approachessocial impact
economic, scientific, technologicalvalues
procedural and political approachesfuture generations (temporal)
actors and stakeholdersunbalances (spatial)
actors and stakeholders
1 Table adapted from Segalas et al. (2008).

Appendix B

Table A2. Sustainability Cognitive Mapping Codes.
Table A2. Sustainability Cognitive Mapping Codes.
social–cultural (parent)A set of beliefs, customs, practices and/or behaviors that exists within a population.language
politics Activities associated with the governance of a country or other area, especially the debate or conflict among individuals or parties having or hoping to achieve power.Republican Party
lawsAnything related to formal rules used to regulate local, state, or federal actors. This could be proposed law, current law, or past law. severance taxes
stakeholdersA stakeholder is any person, organization, or social group. The “who” that is involved with sustainability that encompasses individuals, groups of individuals, or organizations (both for-profit and not-for-profit)Exxon Mobile
religionBelief in or worship of a higher power; the organization of groups that share common beliefs and/or worship a higher powerreligion
higher power
economic (parent)Anything related to the production and management of material wealth.
This could include for-profit businesses. When coding businesses, including “Economic” code as well as the parent code “Social” and child code “Stakeholder.”
currency/dollars business/businesses
Exxon Mobile
resource, usage
environmental (parent)The “natural environment encompasses the atmosphere, hydrosphere, lithosphere, ecosystem processes, and all human and non-human lifeforms” where environmental refers to the interactions between the human and non-human (or wind energy, solar energy)
earth, food
forest, living
oil, non-GMO, green energy
scientific advancementScientific or technological advancement is the generation of information or the discovery of knowledge that advances the understanding of scientific relations or panels
wind turbines
recyclable (i.e., product characteristics)
transportationAn act, process, or instance of moving or being moved.electric car, mass transit, bicycle
intrinsic (parent)Encompasses an individual’s internal way of seeing, knowing, feeling, and/or understanding the term sustainability. The three most common intrinsic items will likely be values, attitudes, and beliefs.attitude: sad, happy, good, bad, satisfactory, useful, smart, strong, supported
value: happiness, important, green, healthy, green energy
belief: consensus, worthwhile, worthy, viable, functioning, operational, relevant, thoughtful, conservative, supportable
attitudesAttitudes: Positive or negative evaluations of something very specific. Unlike values, attitudes are more directed toward a specific item or event (i.e., towards sustainability).
values/beliefsValues: Concepts or beliefs about desirable end states or behaviors that go beyond specific situations to influence how we behave and evaluate behaviors. Values extend beyond a specific item or event.
Beliefs: Our understanding about the state of the world or the facts as we see them. This would also encompass synonyms for sustainability and personal opinions.
actions (parent)The fact or process of doing something, typically to achieve an aim. Please note that actions can fall within other categories. When coding, all action words should be included within this category.recycle
less packaging
walking, protect
save, manage
multidimensional(parent)See individual definitions below.
temporal Relating to time; future focused.continual, durable
longevity, stability
future, growth
renewable, survive
sustain/to sustain
length, cyclic
spatial Relating to space or dispersion.geographic
catch-allContent that does not fit into one of the other categories and may need additional review.

Appendix C

Cognitive Mapping Instructions and Analysis

Cognitive mapping has two direct applications: support for learners and evaluation for instructors. A cognitive mapping item, which can be included with other assessments items or as a stand-alone activity, is useful in the field of sustainability because the tool can effectively capture the field’s interdisciplinary nature. Cognitive maps include three core elements, including focus questions, nodes, and links. To begin the exercise, instructors will pose the following focus question to students:
We are interested in how you think about the term “sustainability.” On the next page, you will have 2 min to list as many words as possible that you associate with the concept. At the end of 2 min, the survey will automatically go to the next question.
We recommend completing the assignment using an online survey that allows the instructor to export each student’s responses into a Microsoft Word or Microsoft Excel file. Next, the steps instructors should take to prepare cognitive mapping data, analyze student responses, and interpret the results are described.
Coding: The instructor will use the codebook (see Appendix B) to code student responses. Parent categories of codes include (1) social–cultural, (2) economic, (3) environmental, (4) intrinsic, (5) actions, (6) and multidimensional. If possible, the instructor should work with a colleague or teaching assistant to establish interrater reliability and calculate a pooled Cohen’s kappa coefficient and Cohen’s kappa for each code. Once the coders establish good interrater reliability, the coders can review all the student responses and complete the coding process. Using qualitative coding software such as Dedoose, NVivo, Atlas.ti, etc. is recommended.
Analysis: Instructors should complete the following steps to analyze student data. SPSS v. 25 or other statistical packages may be used. Table A3 overviews key analytic steps and suggests how instructors should interpret the results.
Table A3. Analytic steps and interpretation.
Table A3. Analytic steps and interpretation.
and correlation
Run descriptive and correlations for all codes (parent and child codes), sorted by pre- and
post-test for treatment and comparison courses.
Descriptive information, such as the number of words provided, and the representation of semantic categories provide preliminary insight to student understanding of sustainable development. Higher word counts and
representation across semantic categories suggests greater student understanding. Correlations point to relationships between code categories.
paired-sample t testsNext, run paired-sample t tests to compare
student means to determine whether there is
statistical evidence that the mean difference
between paired observations (i.e., between pre- and post-tests) is significantly different from zero.
The paired-sample t tests will confirm if treatment courses are significantly different from the comparison courses. Significant findings will confirm the impact of the
curricular interventions in treatment courses.
principal component analysisConduct principal component analysis with
varimax (orthogonal) rotation to develop
linkages for the parent code categories, including relational maps.
Linkages between codes show strong and weak
connections between concepts and demonstrate changes in linkages over time via pre- and post-tests.
Conclusions: Cognitive mapping can assist instructors in tracking student progress with sustainability concepts and test the effectiveness of curricular interventions. The method suggests how students conceptualize and organize knowledge in an open-end format, offering an alternative to multiple choice or sheer memorization techniques and moving towards a method that allows students to integrate diverse higher-order constructs to develop metaphorical thinking.


  1. Bagley, C.E.; Sulkowski, A.J.; Nelson, J.S.; Waddock, S.; Shrivastava, P. A path to developing more insightful business school graduates: A systems-based, experiential approach to integrating law, strategy, and sustainability. Acad. Manag. Learn. Educ. 2020, 19, 541–568. [Google Scholar] [CrossRef]
  2. Petrun Sayers, E.L.; Craig, C.A.; Gilbertz, S.; Feng, S.; Karam, R.; Bohman, A. Advancing STEM-based business sustainability: Mending the curricular gap. Manag. Teach. Rev. 2020, 5, 82–93. [Google Scholar] [CrossRef]
  3. Rogers, M.; Pfaff, T.; Hamilton, J.; Erkan, A. Using sustainability themes and multidisciplinary approaches to enhance STEM education. Int. J. High. Educ. 2015, 16, 523–536. [Google Scholar] [CrossRef]
  4. Koehn, P.H.; Uitto, J.I. Evaluating sustainability education: Lessons from international development experience. High. Educ. 2015, 67, 621–635. [Google Scholar] [CrossRef]
  5. Kurland, N.B.; Michaud, K.E.H.; Best, M.; Wohldmann, E.; Cox, H.; Pontikis, K.; Vasishth, A. Overcoming silos: The role of an interdisciplinary course in shaping a sustainability network. Acad. Manag. Learn. Educ. 2010, 9, 457–476. [Google Scholar]
  6. Lidgren, A.; Rodhe, H.; Huisingh, D. A systematic approach to incorporate sustainability into universities and course curricula. J. Clean. Prod. 2006, 14, 797–809. [Google Scholar] [CrossRef]
  7. Hollander, R.; Arenberg, C.R. (Eds.) Ethics Education and Scientific Education Research: What’s Been Learned? What Should Be Done? Summary of a Workshop; National Academy of Engineering: Washington, DC, USA, 2009. [Google Scholar]
  8. Allen, M.W. Strategic Communication for Sustainable Organizations: Theory and Practice; Springer: New York, NY, USA, 2016. [Google Scholar]
  9. Hiezman, H.; Liu, H. Becoming green, becoming leaders: Identity narratives in sustainability leadership. Manag. Learn. 2017, 49, 40–58. [Google Scholar] [CrossRef]
  10. National Science Foundation (NSF). Improving Undergraduate STEM Education: Education and Human Resources (IUSE: ERH). Available online:\ (accessed on 3 March 2020).
  11. U.S. Department of Education. Science, Technology, Engineering, and Math Including Computer Science. Available online: (accessed on 4 April 2020).
  12. Zwickle, A.; Koontz, T.M.; Slagle, K.M.; Bruskotter, J.T. Assessing sustainability knowledge of a student population. Int. J. Sustain. High. Educ. 2014, 15, 375–389. [Google Scholar] [CrossRef]
  13. Zwickle, A.; Jones, K. Sustainability knowledge and attitudes-Assessing latent constructs. In Handbook of Sustainability and Social Science Research; Leal Filho, W., Marans, R.W., Callewaert, J., Eds.; Springer: New York, NY, USA, 2018; pp. 435–451. [Google Scholar]
  14. Somers, M.J.; Passerini, K.; Parhankangas, A.; Casal, J. Using mind maps to study how business school students and faculty organize and apply general business knowledge. Int. J. Manag. Educ. 2014, 12, 1–13. [Google Scholar] [CrossRef]
  15. Lourdel, N.; Gondran, N.; Laforest, V.; Debray, B.; Brodhag, C. Sustainable development cognitive map: A new method of evaluating student understanding. Int. J. Sustain. High. Educ. 2007, 8, 170–182. [Google Scholar] [CrossRef]
  16. Wood, M.D.; Bostrom, A.; Bridges, T.; Linkov, I. Cognitive mapping tools: Review and risk management needs. Risk Anal. 2012, 32, 1333–1348. [Google Scholar] [CrossRef] [PubMed]
  17. Gul, R.B.; Boman, J.A. Concept mapping: A strategy for teaching and evaluation in nursing education. Nurse Educ. Pract. 2006, 6, 199–206. [Google Scholar] [CrossRef] [PubMed]
  18. Martínez, G.; Pérez, A.L.; Suero, M.I.; Pardo, P.J. The effectiveness of concept maps in teaching physics concepts applied to engineering education: Experimental comparison of the amount of learning achieved with and without concept maps. J. Sci. Educ. Technol. 2013, 22, 204–214. [Google Scholar] [CrossRef]
  19. Omer, U.; Farooq, M.S.; Abid, A. Cognitive learning analytics using assessment data and concept map: A framework-based approach for sustainability of programming courses. Sustainability 2020, 12, 6990. [Google Scholar] [CrossRef]
  20. Sellmann, D.; Lieflander, A.K.; Bogner, F. Concept maps in the classroom: A new approach to reveal students’ conceptual change. J. Educ. Res. 2015, 108, 250–257. [Google Scholar] [CrossRef]
  21. Okereke, C.; Wittneben, B.; Bowen, F. Climate change: Challenging business, transforming politics. Bus. Soc. 2012, 51, 7–30. [Google Scholar] [CrossRef]
  22. Peoples, R. Preparing today for a sustainable future. J. Manag. Educ. 2009, 33, 376–383. [Google Scholar] [CrossRef]
  23. Eisenhardt, K.M.; Graebner, M.E.; Sonenshein, S. Grand challenges and inductive methods: Rigor without rigor mortis. Acad. Manag. J. 2016, 59, 1113–1123. [Google Scholar] [CrossRef]
  24. Henderson, C.; Mestre, J.P.; Slakey, L.L. Cognitive science research can improve undergraduate STEM instruction: What are the barriers? Policy Insights Beh. Brain Sci. 2015, 2, 51–60. [Google Scholar] [CrossRef]
  25. Norton, G.; Thompson, A. Ethics and sustainable development: The virtues of an adaptive approach to environmental choice. In Handbook of Sustainable Development, 2nd ed.; Atkinson, G., Dietz, S., Neumayer, E., Agarawal, M., Eds.; Edward Elgar: Northamptom, MA, USA, 2014; pp. 105–124. [Google Scholar]
  26. Twilley, N. Trailblazers: A New Plan to Solve California’s Fire Problem. New Yorker. 2019. Available online: (accessed on 10 November 2020).
  27. Beach, A.L.; Henderson, C.; Finkelstein, N. Facilitating change in undergraduate STEM education. Chang. Mag. High. Learn. 2012, 44, 52–59. [Google Scholar] [CrossRef]
  28. Laboy, J.B.; Singer, S.R.; George, M.D.; Schweingruber, H.A.; Hilton, M.L. Effective practices in undergraduate STEM education part 1: Examining the evidence. CBE-Life Sci. Educ. 2009, 8, 157–161. [Google Scholar] [CrossRef][Green Version]
  29. Association for the Advancement of Sustainability in Higher Education (AASHE). Campus Sustainability Hub: Academic Programs. Available online: (accessed on 9 November 2020).
  30. Craig, C.A.; Petrun Sayers, E.L.; Gilbertz, S.; Karam, R.; Feng, S. The Role of Stem-Based Sustainability in Business and Management Curricula: Exploring Cognitive and Affective Outcomes in University Students. J. Manag. Educ. (under review).
  31. Craig, C.A. Managinge wildfire disasters in the Rocky Mountains. CASE J. 2019, 15, 78–87. [Google Scholar] [CrossRef]
  32. Craig, C.A.; Petrun Sayers, E.L. Sustainable development in international markets: A metrics-based case study. SAGE Bus. Cases 2020. [Google Scholar] [CrossRef]
  33. Gilbertz, S.; Hall, D. Bringing Sustainability to the Ground Level: Competing Demands in the Yellowstone River Valley; Business Expert Press: New York, NY, USA, (forthcoming).
  34. Novak, J.D.; Gowin, D.B.; Johansen, G. The use of concept mapping and knowledge vee mapping with junior high school science students. Sci. Educ. 1983, 67, 625–645. [Google Scholar] [CrossRef]
  35. Novak, J.D. Clarify with concept maps: A tool for students and teachers alike. Sci. Teach. 1991, 58, 45–49. [Google Scholar]
  36. Novak, J.D.; Canas, A.J. The origins of the concept mapping tool and the continuing evolution of the tool. Inform. Visual. 2006, 5, 175–184. [Google Scholar] [CrossRef]
  37. Segalas, J.; Mulder, K.F.; Ferrer-Balas, D. What do EESD ‘experts’ think sustainability is? Which pedagogy is suitable to learn it? Results from interviews and Cmaps analysis gathered at EESD 2008. Int. J. Sustain. High. Educ. 2012, 13, 293–304. [Google Scholar] [CrossRef]
  38. Rebich, S.; Gautier, C. Concept mapping to reveal prior knowledge and conceptual change in a mock summit course on global climate change. J. Geosci. Educ. 2005, 53, 355–365. [Google Scholar] [CrossRef]
  39. Novak, J.D.; Canas, A.J. The Theory Underlying Concept Maps and How to Construct Them; Institute for Human and Machine Cognition: Ocala, FL, USA, 2006. [Google Scholar]
  40. Kinchin, I.M.; Hay, D.B.; Adams, A. How a qualitative approach to concept map analysis can be used to aid learning by illustrating patterns of conceptual development. Educ. Res. 2000, 42, 43–57. [Google Scholar] [CrossRef]
  41. Chen, J.; Wang, M.; Dede, C.; Grotzer, T.A. Design of a three-dimensional cognitive mapping approach to support inquiry learning. Educ. Technol. Soc. 2017, 20, 191–204. [Google Scholar]
  42. Segalas, J.; Ferrer-Balas, D.; Mulder, K.F. Conceptual maps: Measuring learning processes of engineering students concerning sustainable development. Eur. J. Eng. Educ. 2008, 33, 297–306. [Google Scholar] [CrossRef]
  43. Ritchie, J.; Spencer, L. Qualitative data analysis for applied policy research. In Analyzing Qualitative Data; Bryman, A., Burgess, R.G., Eds.; Routledge: New York, NY, USA, 1994; pp. 173–194. [Google Scholar]
  44. Iverson, J.O.; McPhee, R.D. Communicating knowing through communities of practice: Exploring internal communicative processes and differences among CoPs. J. Appl. Commun. Res. 2008, 36, 176–199. [Google Scholar] [CrossRef]
  45. Dedoose Software. Web Application for Managing, Analyzing, and Presenting Qualitative and Mixed Method Research Data. Available online: (accessed on 9 November 2020).
  46. Landis, J.R.; Koch, G.G. The measurement of observer agreement for categorical data. Biometrics 1977, 33, 159–174. [Google Scholar] [CrossRef] [PubMed][Green Version]
  47. Cozzolino, D.; Power, A.; Chapman, J. Interpreting and reporting principal component analysis in food science analysis and beyond. Food Anal. Methods 2019, 12, 2469–2473. [Google Scholar] [CrossRef]
  48. Sanguansat, P. (Ed.) Principal Component Analysis: Engineering Applications; IntechOpen: London, UK, 2012. [Google Scholar]
  49. Bakhru, K.A.; Sanghi, S.; Medury, Y. A principal component analysis of teaching competencies required for management education. Arth Prabhand A J. Econ. Manag. 2013, 2, 1–15. [Google Scholar]
  50. Erguven, M. Comparison of the Efficiency of Principal Component Analysis and Multiple Linear Regression to Determine Studetns’ Academic Achievement; IEEE: Piscataway, NJ, USA, 2013. [Google Scholar] [CrossRef]
  51. Yang, S.; Lu, O.; Huang, J.; Ogata, H.; Lin, A. Predicting students’ academic performance using multiple linear regression and principle component analysis. J. Inf. Process. 2018, 26, 170–176. [Google Scholar]
  52. Earle, A.G.; Layva-de la Hiz, D.I. The wicked problem of teaching about wicked problems: Design thinking and emerging technologies in sustainability education. Manag. Learn. 2020. [Google Scholar] [CrossRef]
  53. Kurucz, E.C.; Colbert, B.A.; Marcus, J. Sustainability as a provocation to rethink management education: Building a progressive education practice. Manag. Learn. 2013, 45, 437–457. [Google Scholar] [CrossRef]
  54. Kagawa, F. Dissonance in students’ perceptions of sustainable development and sustainability: Implications for curriculum change. Int. J. Sustain. High. Educ. 2007, 8, 317–338. [Google Scholar] [CrossRef]
  55. Shephard, K. Higher education for sustainability: Seeking affective learning outcomes. Int. J. Sustain. High. Educ. 2008, 9, 87–98. [Google Scholar] [CrossRef][Green Version]
  56. Brower, H.H. Sustainable development through service learning: A pedagogical framework and case example in a third world context. Acad. Manag. Learn. Educ. 2011, 10, 58–76. [Google Scholar] [CrossRef]
  57. Montiel, I.; Antolin-Lopez, R.; Gallo, P.J. Emotions and sustainability: A literary genre-based framework for environmental sustainability management education. Acad. Manag. Learn. Educ. 2018, 17, 155–183. [Google Scholar] [CrossRef]
  58. Ausubel, D.P. Educational Psychology: A Cognitive View; Rinehart and Winston: New York, NY, USA, 1968. [Google Scholar]
  59. Ausubel, D.P.; Novak, J.D.; Hanesian, H. Educational Psychology: A Cognitive View, 2nd ed.; Holt, Rinehart, and Winston: New York, NY, USA, 1978. [Google Scholar]
  60. Ajzen, I.; Joyce, N.; Sheikh, S.; Cote, N.G. Knowledge and the prediction of behavior: The role of information accuracy in the theory of planned behavior. Basic App. Soc. Psychol. 2011, 33, 101–117. [Google Scholar] [CrossRef]
  61. Shrivastava, P. Pedagogy of passion for sustainability. Acad. Manag. Learn. Educ. 2010, 9, 443–455. [Google Scholar] [CrossRef]
  62. Heeren, A.J.; Singh, A.S.; Zwickle, A.; Koontz, T.M.; Slagle, K.M.; McCreery, A.C. Is sustainability knowledge half the battle? An examination of sustainability knowledge, attitudes, norms, and efficacy to understand sustainable behaviours. Int. J. Sustain. High. Educ. 2016, 17, 613–632. [Google Scholar] [CrossRef]
  63. Mento, A.J.; Martinelli, P.; Jones, R.M. Mind mapping in executive education: Applications and outcomes. J. Manag. Dev. 1999, 18, 390–407. [Google Scholar] [CrossRef]
Figure 1. Treatment pre- and post-test mean value maps.
Figure 1. Treatment pre- and post-test mean value maps.
Sustainability 13 08074 g001
Figure 2. Comparison pre- and post-test mean value maps.
Figure 2. Comparison pre- and post-test mean value maps.
Sustainability 13 08074 g002
Figure 3. Treatment, pre-test: two-dimensional component plot map.
Figure 3. Treatment, pre-test: two-dimensional component plot map.
Sustainability 13 08074 g003
Figure 4. Treatment, post-test: two-dimensional component plot map.
Figure 4. Treatment, post-test: two-dimensional component plot map.
Sustainability 13 08074 g004
Figure 5. Comparison, pre-test: three-dimensional component plot map.
Figure 5. Comparison, pre-test: three-dimensional component plot map.
Sustainability 13 08074 g005
Figure 6. Comparison, post-test: two-dimensional component plot map.
Figure 6. Comparison, post-test: two-dimensional component plot map.
Sustainability 13 08074 g006
Table 1. Demographics.
Table 1. Demographics.
Treatment (n = 110)Comparison (n = 57)
female61 (55.5%)39 (68.4%)
male49 (44.5%)18 (31.6%)
Non-Hispanic American Indian or Alaskan Native7 (6.4%)2 (3.5%)
Non-Hispanic Asian3 (2.7%)1 (1.8%)
Non-Hispanic Black-2 (3.5%)
Non-Hispanic Hawaiian or Pacific Islander--
Non-Hispanic White105 (90.9%)52 (91.4%)
Student Classification
freshman9 (8.2%)4 (7.0%)
sophomore14 (14.5%)10 (17.5%)
junior39 (37.3%)14 (24.6%)
senior68 (38.2%)29 (49.1%)
graduate2 (1.8%)-
Latino, Hispanic, or Spanish Speaking5 (4.5%)3 (5.3%)
International5 (4.5%)1 (1.8%)
Employment status
full-time38 (34.5%)32 (56.1%)
part-time40 (36.4%)19 (33.3%)
unemployed32 (29.1%)6 (10.5%)
Table 2. Descriptives.
Table 2. Descriptives.
Treatment (n = 110)TestMeanSDSECorr.Sig.
all CodesPre8.824.230.400.530.000
Comparison (n = 57)TestMeanSDSECorr.Sig.
all codesPre9.234.150.550.620.000
Note. Significant correlations between pre- and post-tests support the use of the paired-sample t test method of analysis.
Table 3. Paired-sample t tests.
Table 3. Paired-sample t tests.
Treatment (n = 110)MSDSEtdfSig. (2-tail)
Comparison (n = 57)MSDSEtdfSig. (2-tail)
Table 4. Total variance of all codes explained by components.
Table 4. Total variance of all codes explained by components.
Initial EigenvaluesRotation Sums of Squared Loadings
Treatment Pre-Test (n = 110); χ2 = 54.024, df = 15, Sig. = 0.000, KMO = 0.541
Treatment Post-Test (n = 110); χ2 = 127.79, df = 15, Sig. = 0.000, KMO = 0.541
Comparison Pre-Test (n = 57); χ2 = 29.77, df = 15, Sig. = 0.013, KMO = 0.546
Comparison Post-Test (n = 57); χ2 = 68.77, df = 15, Sig. = 0.000, KMO = 0.714
Note. Extraction sums of squared loadings represent the same values as initial eigenvalues for each group and thus is omitted from the table. χ2 values represent Barlett’s test of sphericity and KMO represents the Kaiser–Meyer–Olkin Measure of Sampling Adequacy.
Table 5. Component matrix with factor coordinates.
Table 5. Component matrix with factor coordinates.
Treatment Pre-Test (n = 110)
environmental 0.71
action 0.74
intrinsic −0.53
Treatment Post-Test test (n = 110)
action 0.78
multidimensional 0.60
Comparison Pre-Test (n = 57)
action 0.81
multidimensional 0.350.68
intrinsic 0.89
Comparison Post-Test test (n = 57)
action 0.59
multidimensional 0.74
Note. Only correlations > 0.30 are reported.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Petrun Sayers, E.L.; Craig, C.A.; Skonicki, E.; Gahlon, G.; Gilbertz, S.; Feng, S. Evaluating STEM-Based Sustainability Understanding: A Cognitive Mapping Approach. Sustainability 2021, 13, 8074.

AMA Style

Petrun Sayers EL, Craig CA, Skonicki E, Gahlon G, Gilbertz S, Feng S. Evaluating STEM-Based Sustainability Understanding: A Cognitive Mapping Approach. Sustainability. 2021; 13(14):8074.

Chicago/Turabian Style

Petrun Sayers, Elizabeth L., Christopher A. Craig, Emily Skonicki, Grace Gahlon, Susan Gilbertz, and Song Feng. 2021. "Evaluating STEM-Based Sustainability Understanding: A Cognitive Mapping Approach" Sustainability 13, no. 14: 8074.

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop