Evaluating STEM-Based Sustainability Understanding: A Cognitive Mapping Approach

Abstract

Management education holds promise for addressing deficiencies in interuniversity science, technology, engineering, and mathematics (STEM), as well as sustainability curricula. Accordingly, we designed, developed, implemented, and longitudinally evaluated interdisciplinary STEM-based curricula in the United States. Students in five sections of business management courses and two sections of STEM courses received a STEM-based sustainability intervention (i.e., an interdisciplinary STEM and sustainability module). To assess student outcomes following the intervention and examine the feasibility of cognitive mapping as a student learning assessment tool, we implemented a pre- and post-course modified cognitive mapping assessment in treatment and comparison courses. To interpret the results, we ran descriptives, correlations, paired sample t tests, and principal component analysis. The t tests suggest that when all coding categories are considered, those participating in curricular interventions listed significantly more sustainability terms. The principal component analysis results demonstrate that treatment courses improved variability explained by 7.23% between pre- and post-tests but declined by 8.22% for comparison courses. Overall, linkages became stronger between parent code categories for treatment courses and weaker for comparison courses. These findings add to existing research related to cognitive mapping and demonstrate the ability of the method to capture changes in student outcomes after exposure to STEM-based sustainability curriculum.

1. Introduction

Challenges remain in interdisciplinary sustainability education across business and science, technology, engineering, and math (STEM) disciplines [1,2]. The integration of STEM is rare in business courses, just as the integration of business concepts is rare in STEM courses [2]. Educators use sustainability as a concept to integrate STEM into curricula across disciplines [3]. However, many sustainability curricular initiatives involving multiple disciplines (e.g., management, STEM) result in multi-disciplinary curricula either by design or in part due to instructors developing curricula in disciplinary silos [3,4,5,6]. Multi-disciplinary curricula, as opposed to interdisciplinary, create challenges because students view sustainability as outside the normal realm of their own discipline and they cannot draw a clear connection to others’ perspectives about sustainability [5,7]. Yet, complex sustainability-related challenges (e.g., climate change) will require current and future leaders, regardless of disciplinary background, to demonstrate and apply an interconnected, interdisciplinary understanding of the challenges of using STEM competencies [8,9,10,11].

To address deficiencies in interuniversity STEM and sustainability curricula, we designed, developed, implemented, and longitudinally evaluated interdisciplinary STEM-based curricula at a western university in the United States. Students in five sections of business management courses and two sections of STEM courses received a STEM-based sustainability intervention (i.e., an interdisciplinary STEM and sustainability module). STEM-based sustainability integrates STEM competencies with and across component areas of sustainability [2]. To improve the validity of the evaluation, we included a comparison cohort of two business management course sections and one STEM course section in the evaluation. In total, 167 students completed pre- and post-tests to assess the impact of the intervention.

A challenge associated with sustainability curricula design and development is evaluation. Measurement tools that surpass self-reported knowledge are lacking, which prompts researchers to find new ways of measuring sustainability proficiency [12,13]. For example, one element of criticism is that students’ lack the capability to “integrate knowledge and apply it to complex problems” [14] (p. 6). One such method of assessing STEM-based sustainability understanding is through cognitive mapping [15]. Cognitive mapping entails diagraming individual mental models about a topic [16], producing graphical representations that demonstrate breadth (e.g., the number of terms students associate with sustainability) and depth of understanding (e.g., the strength of relationships between sustainability terms). A growing body of literature cites cognitive mapping, also referred to as “concept mapping,” as a useful tool for educational evaluation [17,18,19,20]. Accordingly, we build on cognitive mapping literature to explore the ability of the method to capture changes in student outcomes after exposure to STEM-based sustainability curricula. Next, we provide select literature related to STEM and sustainability education, and additionally to cognitive mapping. Following this, material and methods, results, discussion, and conclusion sections are provided.

1.1. STEM and Sustainability

Businesses will encounter increasing climate variability and shifting weather patterns in the future because of climate change. Business leaders may face challenges related to the degradation of the natural environment, market challenges, moral dilemmas, and other sociopolitical issues [21,22]. Considering the magnitude of climate change, future leaders need to navigate the complex impacts from these changes and respond appropriately to innovate, coordinate with other institutions, and change operations as needed.

Sustainability provides a lens through which business leaders can address grand challenges related to climate change [23]. Sustainability in business should explicitly integrate STEM competencies [1,2], yet STEM is often missing from business curricula [2]. With interdisciplinary STEM-based sustainability curricula, business students’ sustainability literacy [14], cognitive abilities [24], and affective outcomes [18] related to STEM and sustainability can improve. Understanding the interconnected relationships between economic, environmental, and social domains is essential to address the goals of doing business, protecting resources, and ensuring fairness across groups [25,26].

In recent years, government and non-profit organizations have increasingly supported STEM education [27,28]. Unfortunately, this momentum has not yet reached business schools. Relatedly, sustainability programs are often available to students across curricula, but tend to overlook business students [29]. In fact, only 179 out of the 2809 sustainability academic programs registered with AASHE as of November 2020 are in a business discipline [29]. We consider that taking on a STEM-based sustainability approach, integrating sustainability literacy (i.e., knowledge) and understanding (i.e., cognition) with the four STEM components (i.e., science, technology, engineering, math), is important to prepare business students for the future. Accordingly, we address gaps in research and practice that are absent from interdisciplinary STEM-based sustainability curricula.

Project Background

Three courses at a western university in the United States received curricular interventions (i.e., an interdisciplinary STEM and sustainability module) including two sections of Business and Environment (n = 32), three sections of International Business (n = 39), and two sections of Human Geography (n = 39). Business and Environment and International Business are upper-level requirements for business management majors and electives for other business majors. Human Geography is a STEM general education elective that fulfills a university cultural diversity requirement. Comparison courses, or those that did not receive the curricular intervention, were also evaluated including one section each of Business and Environment (n = 21), Human Geography (n = 15), and Management and Organization (n = 21).

We selected participating courses to introduce interdisciplinary (1) STEM and sustainability curricula in management education and (2) business and sustainability curricula in STEM education. The student outcome of interest, regardless of discipline, was sustainability cognition (i.e., the process of acquiring sustainability knowledge or understanding through the experience with the focal curriculum). To overcome challenges with designing, developing, and implementing interdisciplinary curricula [2,5] each of the instructors adapted course syllabi to include a common learning objective to: “define, explain, and apply economic, environmental, and social components of sustainability using STEM-based evidence” [30]. Module requirements included:

(1)

inclusion of the three sustainability dimensions;

(2)

inclusion of the four STEM dimensions; and

(3)

the design, development, and implementation of an original problem-based case study with teaching manual.

The cases for Business and Environment [31] and International Business [32] are published, and the case for Human Geography [33] is available as a full-length book. The module requirements ensured adaptation of the course-level common learning objective, interdisciplinary inclusion of STEM and sustainability, and consistent implementation. Below, pertinent literature about the evaluation method for student responses to the STEM-based sustainability curricular interventions is discussed.

1.2. Cognitive Mapping

In general, cognitive mapping has two direct applications: support for learners and evaluation for instructors. This study prioritizes cognitive maps as a sustainability education evaluation tool. Originally developed in the 1980s to characterize causal reasoning, cognitive maps are diagrams that represent the organization of knowledge [34,35]. Most of the literature examines cognitive mapping in K-12 and postsecondary educational settings with a small group of participants. The development of cognitive mapping in the education landscape originated in researchers’ interest in how children develop understanding of science concepts [36]. Cognitive mapping is a proven, useful tool for building and examining understanding of additional STEM and sustainability concepts [15,37].

1.2.1. Characteristics of Cognitive Maps

Three key inputs build a cognitive map: focus questions, nodes, and links [15]. Focus questions prompt participants to construct their maps. For example, focus questions may ask, “what is global climate change,” “what is the evidence [related to global climate change],” or “what are the consequences [of global climate change]” [38] (p. 359). Focus questions can also prompt participants to connect nodes, depicting the interconnectedness of relationships with linkages. In addition to asking questions, evaluators might also alternatively choose to propose a broad idea (e.g., sustainability, the environment, energy, etc.) to students. Thoughtful focus questions can enhance the richness of linkages and the complexity of concepts represented. A poor focus question can result in cognitive maps which do not fully answer the question, or which develop into off-topic responses [39].

By using focus questions to guide thinking, participants can express their thoughts with nodes to convey a list of related concepts, forming a preliminary map, absent linkages. In this step, students provide words that are used to clarify their understanding of the topic (i.e., guided by the focus question) [15]. During the cognitive mapping process, participants graph nodes (or terms), following the direction provided by a focus question. Nodes should relate to the main topic of the focus question and are generally one word or short phrases. Next, students work toward structuring or organizing their terms. This is traditionally done by participants during an in-person session with a researcher where they are asked to connect, or link, concepts to indicate relationships. The linkages between various concepts provide learners an opportunity to operationalize a hierarchical understanding of course curricula or learning experiences.

The next step in the process is the analysis of completed maps. Conceptual understandings of greater nuance correlate positively with map complexity [39]. Research indicates that early instruction or learning experiences may construct shallow cognitive maps that reflect “naïve theories” [40] (p. 52). Naïve theories are misconceptions born of little first-hand experience. Monitoring cognitive maps over time can enhance instructor understanding of the learning pathways students take from naïve theories to more complex knowledge structures [40]. Sellmann et al. [20] describe the importance of identifying preconceptions and how cognitive mapping, done prior to a given curricular intervention, can help shape its curriculum. Cognitive maps can reveal knowledge gaps and naïve theories to instructors, who in turn can specifically target these misconceptions through direct instruction. Other studies demonstrate the usefulness of this tool in measuring knowledge growth, particularly for academically underachieving students e.g., [41]. To maximize meaningful learning and knowledge growth, the maps should be continuously revised as new knowledge is acquired, assimilated, or modified, underscoring Novak’s long-held conception of learning as an ongoing process [39].

1.2.2. Cognitive Maps in STEM Settings

Cognitive mapping plays a particularly interesting role in the field of sustainability because the tool can effectively capture the field’s interdisciplinary nature. Lourdel et al. [15] conducted one of the first sustainability studies among undergraduate engineering students. They administered a mapping task to students at the end of a training session. Students first wrote down concepts related to the stimulus (i.e., sustainable development) and connected those concepts with arrows to signify relationships. During analysis, researchers coded student responses for semantic categories (e.g., social–cultural, environmental, multidimensional, economic/scientific/technological, procedural/political, and actors/stakeholders). Of these, social–cultural, environmental, and economic/scientific/technological categories “gather the nominal and concrete approaches of the concept” of sustainability [15] (p. 171). Analysis of these categories, then, determine a respondent’s ability to translate sustainability concepts into concrete ideas. The remaining categories, however, nonetheless accomplish different goals with respect to the sophistication of concepts. Multidimensional concepts concern “the capacity of abstraction from students” [15] (p. 172). For example, exposure to procedural and political content may promote the understanding of government actions or events related to sustainability. Finally, the actors/stakeholders category acknowledges the participatory dimension of sustainability [15]. In reviewing a respondent’s cognitive map and making an accounting thereof defined by these different categories, insights are revealed about respondents’ systematic vision of sustainability [15].

Linkages are analyzed, as holistic relations, by measuring the quantities of links to certain words. Our findings indicated students’ understanding of sustainable development became richer and more dynamic. However, the authors note limitations associated with this methodology. For example, data encoding may be subjective. Additionally, respondents were not asked to delineate the type of connection or relationship between nodes, as in other cognitive mapping activities; such identification of these relationships could lead to richer data analysis. Lastly, it is nearly impossible to create a standard map to compare against, which limits generalizability [15].

In a subsequent study, evaluating the feasibility of cognitive mapping in sustainability-focused engineering courses, Segalas et al. [42] expanded on Lourdel et al. [15] with a more substantive approach to data analysis. This study analyzed both the quantity of nodes and quantity of inter-linkages. Dimensions of data analysis included degree level, course, elective vs. compulsory enrollment, and pedagogies employed (e.g., lecturing, role play, workshops, distance education). Also, broader semantic categories were included, leading to a greater variety in student responses. Appendix A includes a comparison of the categories used for analysis in Lourdel et al. [15] and Segalas et al. [42]. In assessing pre-and post-instruction cognitive maps, Segalas et al. [42] noticed increased complexity in cognitive mapping of sustainability, which they concluded was due to an increased understanding of sustainability from a holistic and systemic perspective. Another follow-up study evaluated expert and student orientations towards sustainability using cognitive maps [37]. Students understood sustainability as a scientific issue solved by technological innovation, whereas experts understood sustainability in the context of the long-term effects. This mismatch suggests course curricula could benefit from being tailored toward the sociological contexts of sustainability.

1.3. Research Questions

This evaluation of STEM-based sustainability curricula using cognitive maps is guided by sustainability-related research and the cognitive mapping process [15,16,27,42]. Comparable to Segalas et al. [42], we used a pre- and post-test evaluation design around a sustainability-related curricular intervention, though the nature and scale (i.e., 167 students, in eight course sections, in different colleges, completing cognitive mapping exercises at two time points) of our interuniversity curricular interventions necessitated some adaptations. Specifically, brainstorming to generate nodes (i.e., terms) occurred via an online survey in coordination with quantitative evaluation items (see [2] for quantitative explanations and findings). The online format required introducing an alternative step in the process where: (1) nodes were structured into sustainability categories based on data coding (i.e., a qualitative method); and (2) linkages among categories were established using principal component analysis (i.e., a quantitative method). Unlike previous studies, this adapted method explores the strengths and directionality of linkages rather than the quantity of linkages alone. This is the first known study in which researchers, not participants, quantitatively established linkages. With respect to introducing STEM-based sustainability curricula in business management education and expanding the scale to include interuniversity curricular interventions, we pose the following research questions:

• Research Question 1: Do the cognitive maps of sustainability of students enrolled in treatment courses significantly change after receiving STEM-based sustainability curricular interventions?
• Research Question 2: How do the cognitive maps of sustainability of students enrolled in treatment courses change after receiving STEM-based sustainability curricular interventions?
• Research Question 3: Is this adapted cognitive mapping process a viable method for evaluating interuniversity STEM-based sustainability curricular interventions?

In the following section, we discuss study design, data collection, and analysis.

2. Materials and Methods

We administered pre- and post-test surveys via Qualtrics before and after curricular interventions. We collected responses for business and environment and human geography courses in fall 2018. Data from international business courses, including relevant comparison courses, were collected in spring 2019. The curricula interventions (also referred to as modules) were anchored by original case studies [31,32,33]. Modules were delivered in courses over approximately a three-week period. Twenty-two students enrolled in both business and environment and international business courses and were omitted from the study. To ensure fidelity to the treatment courses, similar comparison courses did not receive a curricular intervention (see

Table 1. Demographics.

	Treatment (n = 110)	Comparison (n = 57)
Gender
female	61 (55.5%)	39 (68.4%)
male	49 (44.5%)	18 (31.6%)
other	-	-
Race/Ethnicity
Hispanic	-	-
Non-Hispanic American Indian or Alaskan Native	7 (6.4%)	2 (3.5%)
Non-Hispanic Asian	3 (2.7%)	1 (1.8%)
Non-Hispanic Black	-	2 (3.5%)
Non-Hispanic Hawaiian or Pacific Islander	-	-
Non-Hispanic White	105 (90.9%)	52 (91.4%)
Student Classification
freshman	9 (8.2%)	4 (7.0%)
sophomore	14 (14.5%)	10 (17.5%)
junior	39 (37.3%)	14 (24.6%)
senior	68 (38.2%)	29 (49.1%)
graduate	2 (1.8%)	-
Latino, Hispanic, or Spanish Speaking	5 (4.5%)	3 (5.3%)
International	5 (4.5%)	1 (1.8%)
Employment status
full-time	38 (34.5%)	32 (56.1%)
part-time	40 (36.4%)	19 (33.3%)
unemployed	32 (29.1%)	6 (10.5%)

for treatment and comparison cohort demographics). The comparison cohort helps demonstrate the sustainability knowledge outcomes for students not exposed to STEM-based sustainability content. This approach can serve as a viable alternative when it is not possible to completely randomize participants within a program for examining causal impact.

2.1. Cognitive Mapping Exercise

We adapted Lourdel et al.’s [15] initial focus question to prompt student brainstorming to list nodes. Students received the following prompt:

We are interested in how you think about the term “sustainability.” On the next page, you will have 2 min to list as many words as possible that you associate with the concept. At the end of 2 min, the survey will automatically go to the next question.

On the next page of the survey, students filled in as many words as they could related to sustainability. We pre-tested the focal question in fall 2017 and spring 2018 with 253 students at two U.S. universities. Only five students skipped the question or left the response prompt blank. Given the small amount of missing data, we made no changes to the question prompt or format. One notable difference between ours and Lourdel et al.’s [15] question is that we opted not to ask students to provide linkages between sustainability concepts. Initially, Lourdel et al. [15] used a small sample of 10 students to pilot the method via an in-person exercise. Analyzing visual diagrams by hand was not feasible due to our larger sample size. In the following sections, we describe qualitative and quantitative analyses that provide a viable approach for answering the proposed research questions.

2.2. Qualitative Coding

After pre-testing the instrument in fall 2017 and spring 2018, we developed a codebook for analyzing student responses (see Appendix B for a full codebook with definitions of parent and child codes). Our final categories of codes included (1) social–cultural, (2) economic, (3) environmental, (4) intrinsic, (5) actions, (6) multidimensional, and (7) catch-all (for content that was relevant but did not readily fit into the other existing categories). Catch-all codes were removed prior to analysis. Within each parent category, we created child codes for greater granularity in student responses. We relied on Ritchie and Spencer’s [43] framework analysis technique. The framework analysis technique allows for identification of both a priori issues, or those informed by the original research aims and established literature, along with emergent issues raised by the respondents along the way [43,44].

The authors met to discuss the codebook and clarify definitions. A small number of transcripts were coded to establish interrater reliability using the training function in Dedoose, a qualitative analysis software program [45]. The team tested code application for the codebook, calculating a pooled Cohen’s kappa coefficient and Cohen’s kappa for each code. The team met four times, discussed discrepancies, and established new tests until each team member exceeded a Pooled Cohen’s kappa coefficient of 0.7 or higher. While there are multiple ways to evaluate the significance of a Cohen’s kappa value, Landis and Koch [46] suggest a score of 0.61–0.8 is within a range of good agreement. With acceptable agreement in place, each author coded a subset of the student transcripts, with one author coding more than half of the total pre- and post-test student responses. Additional clarification of unique terms occurred in real-time, and the research team kept a running list of example STEM-based sustainability terms to further exemplify each parent code category.

Each student response was coded to determine the semantic categories used. Sorting into categories provided descriptive information surrounding the breadth and depth of student responses, documented by terms applied across categories in addition to the total number of terms applied. In turn, we can glean insights into changes pre- and post-curricular interventions, in addition to noting the impact to treatment courses in contrast with comparison courses.

2.3. Statistical Analysis

After coding student responses, we ran descriptives and correlations for all codes and for each parent code category, sorted by pre- and post-tests, for both treatment and comparison courses (see

Table 2. Descriptives.

Treatment (n = 110)	Test	Mean	SD	SE	Corr.	Sig.
all Codes	Pre	8.82	4.23	0.40	0.53	0.000
	Post	11.06	4.99	0.48
actions	Pre	0.70	0.98	0.09	0.24	0.012
	Post	0.81	1.21	0.12
economic	Pre	0.81	1.15	0.11	0.37	0.000
	Post	1.35	1.58	0.15
environmental	Pre	2.13	3.03	0.29	0.46	0.000
	Post	2.35	2.53	0.24
intrinsic	Pre	2.85	2.66	0.25	0.63	0.000
	Post	3.04	3.11	0.30
multidimensional	Pre	1.85	1.64	0.16	0.32	0.001
	Post	2.35	1.75	0.17
social–cultural	Pre	0.48	0.92	0.09	0.33	0.000
	Post	1.18	1.47	0.14
Comparison (n = 57)	Test	Mean	SD	SE	Corr.	Sig.
all codes	Pre	9.23	4.15	0.55	0.62	0.000
	Post	9.79	4.31	0.57
actions	Pre	1.23	1.58	0.21	0.40	0.002
	Post	0.93	1.41	0.19
economic	Pre	0.74	0.81	0.11	0.47	0.000
	Post	0.72	0.92	0.12
environmental	Pre	2.25	2.87	0.38	0.53	0.000
	Post	2.51	3.31	0.44
intrinsic	Pre	2.35	2.26	0.30	0.65	0.000
	Post	2.67	2.36	0.31
multidimensional	Pre	2.28	1.69	0.22	0.23	0.082
	Post	2.51	1.72	0.23
social–cultural	Pre	0.39	0.70	0.09	0.36	0.006
	Post	0.46	0.85	0.11

Note. Significant correlations between pre- and post-tests support the use of the paired-sample t test method of analysis.

). To answer research question one, we used SPSS v. 25 to conduct paired sample t tests sorted by treatment and comparison courses (see

Table 3. Paired-sample t tests.

Treatment (n = 110)	M	SD	SE	t	df	Sig. (2-tail)
total	2.25	4.53	0.43	5.20	109	0.000
action	0.11	1.36	0.13	0.84	109	0.403
economic	0.54	1.58	0.15	3.57	109	0.001
environmental	0.22	2.91	0.28	0.79	109	0.434
intrinsic	0.18	2.51	0.24	0.76	109	0.450
multidimensional	0.50	1.99	0.19	2.64	109	0.009
social–cultural	0.70	1.46	0.14	5.04	109	0.000
Comparison (n = 57)	M	SD	SE	t	df	Sig. (2-tail)
total	0.56	3.72	0.49	1.14	56	0.259
action	−0.30	1.65	0.22	−1.37	56	0.177
economic	−0.02	0.90	0.12	−0.15	56	0.883
environmental	0.26	3.01	0.40	0.66	56	0.512
intrinsic	0.32	1.93	0.26	1.24	56	0.222
multidimensional	0.23	2.11	0.28	0.82	56	0.419
social–cultural	0.07	0.88	0.12	0.60	56	0.551

). Paired sample t tests compared student means to determine whether there was statistical evidence that the mean difference between paired observations (i.e., between the pre- and post-test) was significantly different from zero. Maps in Figure 1 and Figure 2 depict changes in mean values in parent code categories from the pre- to post-tests for treatment and comparison courses.

To provide additional insight into research question one (i.e., whether or not the maps of those student enrolled in treatment courses significantly change) and to address research question two (i.e., how do the maps of those student enrolled in treatment courses change), we used SPSS v. 25 to conduct principal component analysis with varimax (orthogonal) rotation to develop linkages for the parent code categories, including relational maps (see Figure 3, Figure 4, Figure 5 and Figure 6). Principal component analysis is a form of multivariate analysis that seeks to determine which components explain the most variance among a set of data and to reduce components to a sub-set of factors highly representative of the set of data [47]. Varimax rotation redistributes the variance among the initially extracted factors to optimize the variance explained by each. PCA has applications “in many fields such as energy, multi-sensor data fusion, materials science, gas chromatographic analysis, ecology, video and image processing, agriculture, color coating, climate and automatic target recognition” [48] (p. XI). PCA has also been used in educational settings to assess the dimensionality of teacher and learner characteristics [49,50,51]. Given the broad applicability of the method and our focus on the dimensionality of student sustainability cognition, the use of PCA is appropriate.

3. Results

3.1. Paired Sample t Tests

We ran paired sample t tests for all codes and then sorted by parent code categories to explore how student cognitive maps for semantic categories changed from pre- to post-tests (see Table 3). For the treatment group (n = 110) there was a significant difference observed for all codes between pre- (m = 8.82) and post-tests (m = 11.06, t = 5.20, p = 0.000). For the comparison group (n = 57) there was no observed significant difference between pre- (m = 9.23) and post-tests (m = 9.28, t = 1.14, p = 0.259). The results demonstrated that, when all the coding categories were considered, those participating in curricular interventions listed significantly more sustainability terms than those who did not. There was no significant difference for the actions, environmental, or intrinsic parent code categories for the treatment group (n = 57). However, there were significant differences between the pre- and post-tests for economic (t = 3.57, p = 0.001), multidimensional (t = 2.64, p = 0.009), and socio-cultural (t = 5.04, p = 0.000) parent codes. There were no observed significant differences in the comparison group between the pre- and post-tests.

3.2. Principal Component Analysis

We ran principal component analysis using varimax rotation, sorting by pre- or post-test, for treatment and comparison courses, to develop linkages for parent code categories (see

Table 4. Total variance of all codes explained by components.

	Initial Eigenvalues			Rotation Sums of Squared Loadings
Component	Total	%Variance	%Cumulative	Total	%Variance	%Cumulative
Treatment Pre-Test (n = 110); χ2 = 54.024, df = 15, Sig. = 0.000, KMO = 0.541
1	1.76	29.29	29.29	1.56	25.99	25.99
2	1.24	20.65	49.94	1.44	23.95	49.94
3	0.96	16.05	65.99
4	0.92	15.25	81.24
5	0.62	10.36	91.60
6	0.50	8.40	100
Treatment Post-Test (n = 110); χ2 = 127.79, df = 15, Sig. = 0.000, KMO = 0.541
1	2.18	36.40	36.40	2.18	36.40	36.40
2	1.25	20.77	57.17	1.25	20.77	57.17
3	0.96	16.03	73.20
4	0.73	12.24	85.43
5	0.62	10.39	95.83
6	0.25	4.17	100
Comparison Pre-Test (n = 57); χ2 = 29.77, df = 15, Sig. = 0.013, KMO = 0.546
1	1.81	30.11	30.11	1.65	27.48	27.48
2	1.19	19.89	50.00	1.22	20.27	47.74
3	1.07	17.90	67.90	1.21	20.16	67.90
4	0.82	13.60	81.50
5	0.63	10.51	92.01
6	0.48	7.99	100
Comparison Post-Test (n = 57); χ2 = 68.77, df = 15, Sig. = 0.000, KMO = 0.714
1	2.53	42.16	42.16	2.53	42.16	42.16
2	1.05	17.52	59.68	1.05	17.52	59.68
3	0.89	14.76	74.44
4	0.73	12.20	86.64
5	0.44	7.32	93.96
6	0.36	6.04	100

Note. Extraction sums of squared loadings represent the same values as initial eigenvalues for each group and thus is omitted from the table. χ² values represent Barlett’s test of sphericity and KMO represents the Kaiser–Meyer–Olkin Measure of Sampling Adequacy.

). For treatment courses, we reduced the six parent code categories (i.e., all codes) to two-factor solutions for pre- and post-tests, explaining 49.94% and 57.17% of the variance, respectively. For comparison courses, we reduced all codes to a three-factor solution for pre-tests and a two-factor solution for post-tests, explaining 67.90% and 59.68% of the variance, respectively. Solutions only included components with eigenvalues greater than one. The results demonstrate that treatment courses improved the variability explained by 7.23% from pre- to post-tests but declined by 8.22% for comparison courses. That is, overall linkages became stronger between parent code categories for treatment courses and weaker for comparison courses.

The factor coordinates in

Table 5. Component matrix with factor coordinates.

Component	1	2	3
Treatment Pre-Test (n = 110)
economic	0.78
environmental		0.71
social–cultural	0.72
action		0.74
multidimensional	−0.59
intrinsic		−0.53
Treatment Post-Test test (n = 110)
economic	0.82
environmental	0.75
social–cultural	0.86
action		0.78
multidimensional		0.60
intrinsic	−0.43	−0.51
Comparison Pre-Test (n = 57)
economic	0.72
environmental	0.66	−0.40
social–cultural	0.83
action			0.81
multidimensional		0.35	0.68
intrinsic		0.89
Comparison Post-Test test (n = 57)
economic	0.81
environmental	0.80
social–cultural	0.73
action		0.59
multidimensional		0.74
intrinsic	−0.67

Note. Only correlations > 0.30 are reported.

represent correlations between a variable (i.e., a parent code category) and a factor axis. The coordinates are mapped in Figure 3, Figure 4, Figure 5 and Figure 6, depicting the linkages (i.e., correlations) that each parent code category shares with the factor. For treatment course pre-tests, Factor 1 represents 25.99% of all codes, where the codes economic and social–cultural are positively linked to the factor and the code multi-dimensional negatively linked. Factor 2 represents 23.95% of all codes, where codes environmental and action are positively linked to the factor and the code intrinsic negatively linked. On post-tests, Factor 1 improved, to represent 36.40% of all codes, where economic, environmental, and social–cultural are codes positively linked to the factor and intrinsic negatively linked. Factor 2 represents 27.11% of all codes, where actions and multidimensional are codes positively linked to the factor and, again, intrinsic is negatively linked.

For comparison group pre- pre-tests, Factor 1 represents 27.48% of all codes where economic, environmental, and social–cultural are positively linked to the factor. Factor 2 represents 20.27% of all codes where intrinsic and multidimensional are positively linked to the factor and environmental negatively linked. Factor 3 represents 20.16% of all codes where the action and multidimensional are codes positively linked to the factor. For post-tests, Factor 1 represents 42.16% of all codes, where economic, environmental, and social–cultural are codes positively linked to the factor, and intrinsic is negatively linked. Finally, Factor 2 represents 17.52% of all codes, where the codes action and multidimensional are positively linked to the factor.

4. Discussion

The complexities of sustainability-focused education in management education presents systems-level challenges [52] that STEM can address. However, the lack of communication and collaboration across disciplines hinders the implementation of interdisciplinary curricula [27] such as the STEM-based sustainability curriculum presented in this study. Organizations, both for-profit and non-profit, face complicated sustainability issues within and across the three key aspects of sustainability [8] that require a STEM- and sustainability-literate workforce. Yet, there are relatively few sustainability programs in business disciplines and an interdisciplinary STEM-focus is lacking from program offerings [29]. Business management content is also seldomly covered in STEM curricula [2]. The need for, and absence of, STEM-based sustainability education presents an opportunity for “an alternative vision of management education as a progressive educative practice: one that embraces our embeddedness in the natural world and our social relation to one another” [53] (p. 437). To address curricular gaps devoid of STEM and sustainability in management education, we developed, implemented, and longitudinally evaluated a STEM-based sustainability curriculum in three courses representing business management and STEM disciplines. We used cognitive maps to answer our three research questions.

4.1. Research Question 1: Do Student Cognitive Maps in Treatment Courses Significantly Change?

In the aggregate, findings from the all coded paired-sample t tests provide evidence that there is a significant change in cognitive maps from treatment courses (t(109) = 5.20, p = 0.000) between pre- and post-tests (see Figure 1 and Figure 2). These same changes did not occur in the comparison group that did not receive STEM-based sustainability curricula. There are significant changes exhibited for the parent code categories between pre- and post-tests for treatment students, including the economic (t(109) = 3.57, p = 0.001), multidimensional (t(109) = 2.64, p = 0.009), and social–cultural (t(109) = 5.04, p = 0.000) codes. Considering this, respondents demonstrated change in two of the concrete categories (i.e., economic and social–cultural) and one abstract category (i.e., multidimensional). Segalas et al. [42] used similar (though not identical) categories in their analysis and compared the categorical relevance (CR), or the distribution of concepts between categories, before and after sustainability-related engineering coursework. They, too, observed an overall improvement for the total number of concepts, though results were mixed and dependent on the coding category.

The lack of significant improvement for treatment students in the environmental code, which is a concrete sustainability concept, is a counter-intuitive finding, especially considering we explicitly integrated environmental sustainability into each module. Though, the finding is consistent with Segalas et al. [42], where student maps consisted of 24% environmental codes on the pre- pre-test but only 22.8% on the post-test. Many understand and operationalized sustainability in terms of the environment and environmental action (e.g., recycling, saving, composting) [13], so it may be that understanding of the environmental semantic category of sustainability was already well-developed among students. Furthermore, Kagawa [54] noted that students can perceive environmental components of sustainability as competitive with economic and social components, another possible explanation for improvement in these parent codes but not environmental.

Another counter-intuitive finding is that the mean number of intrinsic (i.e., values, attitudes, and beliefs) nodes listed did not significantly improve. Zwickle and Jones [13] found the link between sustainability knowledge and attitudes to be weak, thus, a possible explanation for the lack of change may be there was no clear link between knowledge and affect. Segalas et al. [42] similarly observed weak linkages between “soft” or abstract concepts (e.g., values) and sustainability on student cognitive maps. Unlike environmental sustainability, we did not adapt affective learning objectives in the implemented modules. Shephard [55] contends that instructors should target affective sustainability learning separate from cognitive learning—which has been the focus of our STEM-based sustainability curricula—and assign grades based on achievement of affective objectives. The adaptation of affective learning objectives, in coordination with co-curricular activities such as service-learning, holds promise for targeting improvement in and giving credit for affective sustainability learning [56,57]. Overall, tracking change in semantic categories clarified the overall ability to understand the systematic vision of sustainability. Further, we learned that treatment courses grasped some of the concrete and abstract aspects of sustainability but could still improve in other areas of sustainability.

4.2. Research Question 2: How Do Student Cognitive Maps from Treatment Courses Change?

We introduced an updated cognitive mapping process tailored for larger scales, featuring interuniversity curricular interventions where linkages between nodes were quantitively developed and explored by researchers, not participants. Unlike Lourdel et al. [15] and Segalas et al. [42], this update captured additional characteristics about linkages (e.g., strength and directionality of relationships) but not the quantity of those linkages. Using principal component analysis, we uncovered linkages between sustainability parent code categories and produced relational cognitive maps (Figure 3, Figure 4, Figure 5 and Figure 6). Inspecting maps and factor coordinates (see Table 5) between pre- and post-tests indicated that change had occurred for treatment and comparison students, and provided insight into those changes.

There are four key findings related to the linkages observed between the parent code categories. First, the overall strength of linkages shows a positive change induced by treatment courses and a negative change for comparison courses. This finding speaks directly to Research Question 1 and provides support for STEM-based sustainability curricula in business and STEM disciplines. Second, post-test maps from treatment courses demonstrate the strongest linkages among economic, environmental, and social–cultural categories. This suggests that respondents are making connections between the three concrete categories of sustainability. While the number of economic and social–cultural mean values significantly improved between pre- and post-tests for treatment students, the same was not true for environmental parent code categories. Unlike Kagawa’s [54] finding, however, treatment course factor coordinates demonstrate that environmental codes are not competing with the economic and social components on pre- pre-tests, rather they are unrelated. Factor coordinates on post-tests demonstrate that linkages changed where environmental codes became closely related to economic and social–cultural codes.

Third, the complexity of the post-test map for treatment courses increased, where abstract multidimensional codes transformed from negative linkage to positive linkage. Rather than competing with economic and social–cultural factors, as on pre-tests, post-test results showed multidimensional codes became complimentary with actions. In other words, respondents started to make connections between things like time, prevention, and future generations and activities, measures, or operations. Mean values of multidisciplinary codes increased (Figure 1) and the linkages shifted from negative to positive between pre- and post-tests for treatment students. Combined, the improvement in variability explained in addition to positive changes in environmental and multidimensional codes between pre- and post-tests for treatment courses suggest that meaningful learning occurred. Meaningful learning comprises the interrelationships of concepts, how they are applied, and how they are adapted requiring students to form active connections between course material [58,59].

And fourth, the intrinsic codes (i.e., values, attitudes, and beliefs) share negative relationships for both treatment and comparison courses on post-tests. Intrinsic codes are negatively linked to the three sustainability component factors for both treatment and comparison courses on post-tests and negatively linked to the factor including multidimensional and action codes for treatment courses. Ajzen et al. [60] contend that environmental affect is not related to environmental learning and Zwickle and Jones [13] note that sustainability knowledge is weakly related to sustainability attitudes. Our results suggest that cognitive learning, as a function of curricular interventions and/or learning that may have occurred from the cognitive mapping exercise, is negatively related to an affective connection with sustainability. These results reinforce the need to target affective learning about sustainability, independent of cognitive learning, to improve affective outcomes and facilitate a passion for sustainability [55,61].

4.3. Research Question 3: Using Cognitive Mapping to Assess Student Learning

Based on our results, we believe that our adapted cognitive mapping approach is a promising method to assess STEM-based sustainability student learning in business schools. As with any new or modified evaluation tool, any changes to an existing instrument should be tested to ensure retention of the utility of the instrument. The results from Research Questions 1 and 2 demonstrate that this method is capable of detecting differences both in the nodes provided by students and the strength and directionality of conceptual relationships. Therefore, our variation of the tool complements previous efforts from researchers [15,37] as a means of examining the understanding of STEM and sustainability concepts.

A strength of this study is that we explored ways of reducing burden, both to students and instructors who analyzed the data. For example, Lourdel et al. [15] (p. 172) acknowledge the challenge of “limited time” when collecting their data, and Rebich and Gautier [38] note that students in their study were given 45 min to complete their concept-mapping activity. Given that cognitive mapping may be one of many assessment items, instructors may not be able to devote limited class time to an extended cognitive mapping activity. However, even with limiting our procedure to 2 min, our results suggest that an abbreviated cognitive mapping approach is possible.

Additionally, our modified analysis strategies may be useful to other instructors who lack the resources to manually analyze cognitive maps by hand. For example, Lourdel et al. [15] examined the inter-linkages between categories and manually counted the total number of links between words produced by respondents. Other types of cognitive mapping software exist, which could be used to incorporate faster analysis of conceptual links, yet instructors may not have funds for additional software. Instructors could also face challenges, including research items that rely on specialized software for data collection otherwise not used in the evaluation of sibling items. Further, requiring students to access cognitive mapping software online creates barriers, particularly for students who may have limited internet access. Segalas et al. [42] and Segalas et al. [37] may have used software to collect cognitive map data from students, although this is not explicitly clarified. Using principal component analysis, we evaluated students in order to capture additional characteristics about linkages between sustainability parent code categories, without manually counting connections or relying on a separate data collection platform. Other instructors, in the future, could benefit from applying a similar approach to cognitive- or concept-mapping activities. Appendix C contains a one-page summary of cognitive mapping instructions, data preparation, data analysis, and interpretation to assist instructors interested in applying the exercise in the future.

Finally, as addressed by Lourdel at al. [15], classifying semantic terms can introduce subjectivity into processing data. We addressed this criticism by testing our codebook (Appendix A) by calculating a pooled Cohen’s kappa coefficient and Cohen’s kappa for each code. We also used the codebook to capture a priori categories for cognitive maps, which provided flexibility in capturing emergent issues [43,44]. For instance, unlike previous studies we identified the intrinsic category, noting individual affect has been previously linked to sustainability [62]. Our larger sample size (n = 167) afforded us the opportunity to build on these a priori categories and identify higher resolution parent and child codes which may be useful to educational evaluators in the future. Combined, we believe that our adapted cognitive mapping approach holds promise for assessing student learning, particularly among larger sample sizes, because it reduces the time and burden of data collection and data analysis, and it addresses a common criticism related to the subjectivity of semantic classification.

4.4. Limitations and Future Research

The study is not without limitation, warranting future research. First, while we expanded the sample to include multiple disciplines, the sample size was still limited (n = 167). Future studies should expand the use of cognitive maps for evaluation purposes across even larger samples. This would, over time, also assist with evaluating the reliability of the parent code categories. Additionally, future studies might employ the use of natural language processing software to reduce the burden of manual coding for evaluators. Additional testing of further automated analysis may be useful for further scaling up this approach in additional settings.

Also, there were curricular requirements to include STEM, business, and sustainability within an interdisciplinary context. However, the development and deployment of curricular materials was not systematic, and we were unable to assess changes based on curricular intervention type. Future development of curricular materials can use a more systematic approach, where changes in cognitive maps can be clearly linked (e.g., [42]). Future longitudinal studies should also implement mixed methodologies that include a combination of cognitive mapping, other qualitative methods (e.g., interviews), and quantitative STEM and sustainability knowledge/affect assessments. Future research could also use cognitive maps to evaluate and inform other STEM concepts and might explore differences concerning in-person versus online classes.

Lastly, there were notable differences in pre-test maps (Figure 3 and Figure 5) for treatment and comparison students. Unfortunately, the methods utilized in this study (e.g., t tests) preclude making causal inferences about why the differences emerged. Future researchers should consider using advanced methodology that controls for co-variates that could potentially influence differences in starting mental representations about sustainability.

5. Conclusions

Understanding the connections between economic, environmental, and social domains, and even beyond these, will be essential to address the goals of doing business, protecting resources, and ensuring fairness across groups in the future [25,26]. Complex sustainability-related challenges, such as climate change, require future decision-makers to demonstrate and apply an interconnected and interdisciplinary understanding of sustainability. Cognitive mapping is one technique to capture and evaluate concrete and abstract approaches to sustainability. As universities consider implementing interdisciplinary STEM-based curricula, using a cognitive mapping approach is one tool that could be used for educational evaluation. While this paper describes one interdisciplinary STEM-based curricula at a western university in the United States, educators can apply findings from this study to other efforts focused on improving STEM-based sustainability curricula, particularly in business and/or management programs. This study demonstrates how cognitive mapping can be employed to evaluate curricular interventions and suggests that the method is a useful tool for assessment.

This study found (Research Question 1) a significant change in treatment students’ maps pre- and post-test. Overall, we discovered that treatment students demonstrated more involvement with economic, multidimensional, and socio-cultural concepts but experienced no change with other sustainability categories. Further, (Research Question 2) we observed that the complexity of interrelationships between concepts improved for treatment students but declined for comparison students. Finally, (Research Question 3) we note that cognitive mapping shows how students conceptualize and organize knowledge in an open-ended format, offering an alternative to multiple choice or sheer memorization techniques and allowing students to integrate diverse higher-order constructs to develop metaphorical thinking [14,63]. Educators and evaluators can replicate this approach in future assessments of STEM-based sustainability curricula and may find additional ways of streamlining the method to reduce student and evaluator burden in the future.