Comparing Measures of Student Sustainable Design Skills Using a Project-Level Rubric and Surveys

Abstract

Civil engineers are poised to impact sustainable development. Consequently, there is a need for curricular materials to scaffold students in developing sustainable design skills. Previously, a sustainability module, based on Kolb’s learning cycle, was integrated into a civil engineering capstone course in the United States. The purpose of this study was to analyze the extent to which students engaging in the module (intervention cohort) were able to improve their sustainable design skills, as compared to a group of capstone students not participating in the module (control cohort). A Sustainable Design Rubric was used to assess students’ sustainable design performance, as captured in capstone reports. In addition, students reflected on their confidence related to several sustainable design competencies via a survey. Based on an evaluation of capstone design reports, improvement in the intervention teams’ consideration of sustainable design criteria was somewhat limited, as they more extensively addressed only 2 of the 16 sustainable design criteria compared to control teams. Intervention students reported improved confidence in more sustainable design competencies than control students (10 of 12 for intervention students; 1 of 12 for control students). For future implementations, clearer and more extensive sustainable design expectations need to be set by instructors and project sponsors to increase the execution of sustainable design and close the gap between students’ perceptions of improved skills and teams’ actual application of sustainable design criteria.

1. Introduction

1.1. Sustainable Design

Design is fundamental to engineering practice and is even considered to be “the essence of engineering” [1] (p. 1). Dym et al. [2] provide a thoughtful narrative on engineering design, describing it as: “a systematic, intelligent process in which designers generate, evaluate, and specify concepts for devices, systems, or processes whose form and function achieve clients’ objectives or users’ needs while satisfying a specified set of constraints” (p. 104). Even so, ABET, the body that accredits undergraduate engineering programs in the United States (US), defines design as “a process of devising a system, component, or process to meet desired needs and specifications within constraints” [3]. While conceptualizations of engineering design are numerous, many converge on the ideas that design (1) requires decision-making regarding trade-offs among constraints and (2) should address the needs of stakeholders.

Sustainable design, as several propose (e.g., [4,5]), is not a new paradigm, but rather requires a more holistic scope and valuation of the criteria and stakeholders to be considered during decision making. More specifically, sustainable design criteria encompass the short-term and long-term effects that a solution could have on the economy, environment, and society [6]. In sustainable design, stakeholders who may reap benefits or experience hardships as a result of a design, both immediately and in the future, are considered and actively engaged (if possible) [7]. In contrast, short-term economic impacts on immediate clients and users often drive decision-making during traditional design. While the traditional design process and sustainable design are not mutually exclusive, Mulder [8] cautions that sustainability is not an “add-on” and must be incorporated throughout design to achieve impact. Indeed, Abraham [9] provides engineers with a set of Nine Principles of Sustainable Engineering that provide “overarching concepts” that can be integrated throughout the design process to guide sustainable decisions (

Table 1. Nine Principles of Sustainable Engineering [9].

No.	Principle
1	Engineer processes and products holistically using system analysis.
2	Conserve and improve natural ecosystems while protecting human health and wellbeing.
3	Use life cycle thinking in all engineering activities.
4	Ensure that all material and energy inputs and outputs are as inherently safe and benign as possible.
5	Minimize depletion of natural resources.
6	Strive to prevent waste.
7	Develop and apply engineering solutions, while considering local circumstances and cultures.
8	Create engineering solutions beyond current or dominant technologies.
9	Actively engage communities and stakeholders in development of engineering solutions.

). Gagnon et al. [10] propose that engineers practice along a spectrum ranging from traditional to sustainable design. Indeed, engineers have a critical role to play in the pursuit of sustainable development, as they are the designers of products, processes, services, and infrastructure that will have lasting impacts on people, prosperity, and the planet.

1.2. Sustainable Design Education in Civil Engineering

The American Society of Civil Engineers (ASCE) has called for reform of engineering practice and subsequently engineering education as well. Recently, ASCE published a five-year sustainability roadmap to “transform the profession” [11]. In addition, the ASCE Code of Ethics mandates that engineers “strive to comply with the principles of sustainable development in the performance of their professional duties” [12]. Regardless of discipline, ABET specifies that all undergraduate students demonstrate “an ability to apply engineering design to produce solutions that meet specified needs with consideration of…global, cultural, social, environmental, and economic factors” [3]. Consequently, there is a need to develop curricular materials and educational experiences to scaffold students in developing and applying sustainable design skills so that they can revolutionize engineering practice in the future.

Within civil engineering curricula, the capstone design course is a common target for innovation related to sustainability. In capstone courses, teams of senior engineering students typically develop designs in response to a real-world problem. Brunell [13] reports on a civil engineering capstone design course where students were introduced to the United Nations Sustainable Development Goals (UNSDGs) and asked to choose one or more to address in their civil engineering capstone projects. Delatte and Hatley [14] and Burian [15] used established rating systems, such as LEED^® and/or Envision^®, to scaffold civil engineering students in learning about and integrating sustainability into their capstone projects. Alternatively, Payne and Aidoo [16] developed a community engagement module that students completed concurrently with their capstone course. Student teams were required to develop economically, socially, and environmentally focused design alternatives and then synthesize alternatives into a final capstone design. Common among these interventions is the need for instructors to address limitations in students’ prior knowledge about sustainably prior to design. As Delatte and Hatley [14] summarize, students’ prior experiences with sustainability can vary significantly even within a single capstone course.

1.3. Learning-Cycle-Based Sustainability Module

A sustainability module was previously developed to guide civil engineering students in learning about and applying sustainability principles during design [17,18,19,20]. The module was designed to lead students through activities that were structured to follow Kolb’s learning cycle [21,22]. In Session 1, students reflect on their prior sustainability knowledge using concept maps. In Session 2, students prepare and participate in peer lectures to learn about sustainable development and related themes: environmental, economic, and social sustainability, as well as sustainable design tools. In Session 3, students analyze case studies to provide examples of what sustainable design “looks like” in real-world projects. During Session 4, students apply their knowledge by completing a preliminary sustainability analysis of their semester design project. Finally, in Session 5, students reflect again on their sustainability knowledge using concept maps.

The sustainability module has been implemented at two different institutions in the US with students across academic years in civil engineering [17,19] and interdisciplinary engineering [18,23] programs. In previous studies, the impacts of the module on students’ conceptual understanding of sustainability were assessed using concept maps and student self-report surveys. Concept maps universally showed that students improved the breadth, depth, and connectedness of their sustainability knowledge. Content analysis of concept maps generally showed an emphasis on environmental concepts prior to module completion and a more balanced understanding of sustainability across areas after module completion. However, even after module completion, the economic area tended to be somewhat under-represented in concept maps [24]. Survey results generally aligned with concept map findings, although civil engineering students completing the module as part of a cornerstone design course perceived greater gains in their sustainability knowledge than students completing the module as part of a capstone design course [19]. While multiple studies support the efficacy of the learning-cycle-based module for improving sustainability knowledge among undergraduate engineering students, additional work is needed to capture impacts on sustainable design skills.

1.4. Sustainable Design Rubric

Previously, a three-phase process was used to develop a Sustainable Design Rubric for undergraduate-level design projects [25,26]. In the first phase, professional rating systems (e.g., LEED^® and Envision^®) were explored, but they were deemed too complex and/or narrow in scope for typical student courses that span a semester or academic year. However, the Nine Principles of Sustainable Engineering [9] (Table 1) were identified as a set of broad, accessible guidelines that might be applicable to a range of student engineering projects. In the second phase, a pilot rubric, including 16 sustainable design criteria, was created based on the Nine Principles (

Table 2. Mapping of sustainable design frameworks used for this study.

Sustainable Design Areas	Nine Principles of Sustainable Engineering [9]	Sustainable Design Rubric Criteria and Sustainable Design Competencies *
Environmental	No. 5	Minimizes natural resource depletion *
	No. 6	Prevents waste *
	No. 2	Protects natural ecosystems *
	No. 2, 5, 4	Uses renewable energy sources *
	No. 4	Uses inherently safe/benign materials (to environment) *
Social	No. 9	Addresses community and stakeholder requests *
	No. 7	Considers local circumstances and cultures *
	No. 2	Protects human health and well-being *
	No. 4	Uses inherently safe/benign materials (to humans) *
Design Tools	No. 3	Incorporates life cycle analysis *
	No. 3	Incorporates environmental impact assessment (EIA) tools *
	No. 1	Incorporates systems analysis *
	No. 8	Uses innovative technologies *
Economic b	-	Considers economic impacts of environmental design criteria
	-	Considers economic impacts of social design criteria
	-	Conducts a cost and/or cost-benefit analysis

* Criteria comprise the 12 sustainable design competencies for which students rated their confidence level. ^b Economic considerations not included in the Nine Principles [9].

). To measure achievement of the criteria, two four-point rating scales were created, with the “potential” scale capturing the extent to which each criterion is applicable to a given project and the “earned” scale capturing the extent to which a student or team addresses each criterion in their design. Based on a repository of civil engineering capstone projects, examples of how each criterion might be applied were summarized to assist with project scoring. In the third phase, the rubric was reviewed and refined by an expert panel to establish content validity. The final Sustainable Design Rubric is composed of 16 criteria across four categories [25,26].

The Sustainable Design Rubric was used at a research-intensive institution in the US to evaluate a sample of civil and environmental engineering (CEE) capstone design projects completed over a ten-year period [26]. Researchers found that students’ sustainable design skills, as evidenced in the final reports, only changed slightly over the period for some economic design criteria. In general, researchers found that social design criteria were most frequently addressed, specifically “protect human health and well-being” and “address community and stakeholder requests.” The Economic category was the second most emphasized, especially “conduct a cost and/or cost-benefit analysis”, which is a required project component at the institution. Criteria from the Environmental and Design Tools categories were applied to a lesser extent. Overall, the Rubric proved to be a useful tool for systematically evaluating the extent to which sustainability is considered in undergraduate CEE design projects.

1.5. Project Scope

The goal of the current study is to examine the impact of the learning-cycle-based sustainability module on students’ sustainable design skills. In a previous study [17], students participating in a civil engineering capstone design course with the embedded module (intervention cohort) showed greater gains in sustainability knowledge, as compared to students participating in the capstone course alone (control cohort). Student sustainable design skills will be assessed and compared between the intervention and control cohorts through a self-report survey of sustainable design competencies and application of the Rubric to team project reports (Table 2). The following questions will be addressed: (1) To what extent does participation in the module impact teams’ tendencies to apply sustainable design criteria in their capstone projects? (2) Between cohorts, how do teams’ tendencies to apply sustainable design criteria in their capstone projects compare to sponsors’ expectations? (3) Which sustainable design competencies are most impacted by module participation, according to students’ individual perceptions? Ultimately, the results will be used to inform the improvement in future module implementations, especially related to the development of students’ sustainable design skills.

2. Materials and Methods

2.1. Study Site

The study was conducted at a large, research-intensive institution in the Southeastern US with students enrolled in a CEE capstone design course. Students were seniors who had completed all prerequisite design courses in CEE sub-disciplines. Most capstone participants were male (78.4%) civil engineering (79.1%) students from the US (83.6%). The course was project-based, with student teams (3–6 students) working with instructors and engineering professionals (“sponsors”) to complete an authentic design. One of the primary deliverables was a comprehensive report that presented the final design to sponsors. Incorporating sustainability into the projects was not a course requirement, although sponsors may have conveyed sustainability expectations to teams. Most students’ primary exposure to sustainability concepts was during their second academic year when taking a civil engineering systems course, which is intended to introduce students to sustainability from a systems perspective [27,28].

2.2. Experimental Design

A quasi-experimental design [17] was used to investigate the impacts of the module on students’ sustainable design skills. Students enrolled in a traditional, unamended capstone design course served as the control cohort. Students enrolled in the capstone design course the following semester participated in module activities and served as the intervention cohort. Students in the intervention cohort were required to complete all module assignments, which accounted for 5% of their overall course grade. Other than participation in the module, elements of the two capstone courses were similar, including the deliverables and co-instructors.

Sustainable design skills were assessed via a student survey and application of the Sustainable Design Rubric [25,26] to team project reports. The survey was administered to both cohorts at the beginning and end of the semester to capture their perceptions of several sustainable design competencies, which were composed based on the Nine Principles of Sustainable Engineering [9] (Table 1 and Table 2). Of the 67 students in the control cohort, 47 completed design skills surveys (70% participation). Of the 100 students in the intervention cohort, 84 completed design skills surveys (84% participation). Students in the control cohort were organized into 14 groups, while students in the intervention cohort were organized into 20 groups. Each group composed a final report that was assessed as part of the study.

2.3. Analysis of Team Design Reports

Sustainable design skills across the control and intervention cohorts were assessed by applying the previously developed Sustainable Design Rubric to team design projects, as documented in the final reports. Scoring encompassed assigning “potential” and “earned” points, for each of the 16 sustainable design criteria, to capture sponsors’ sustainable design expectations and groups’ sustainable design performance, respectively. Potential and earned points were assessed on a four-point scale ranging from no expectation/performance to extensive expectations/performance (

Table 3. Earned points scale to capture sustainable design expectations [25,26].

Earned Score	Descriptor	Dimension Description
0	Unacceptable	Criterion not at all considered in project report.
1	Developing	Criterion mentioned or discussed in the project report, but not applied in design process.
2	Competent	Project report shows evidence that the criterion was adequately applied in design process (1–2 instances of criterion application).
3	Exemplary	Project report shows evidence that the criterion was extensively applied in the design process (3 or more instances of criterion application).

and

Table 4. Potential points scale to capture sustainable design expectations [25,26].

Potential Score	Descriptor	Dimension Description
0	Inapplicable	The criterion is not at all valid for the project.
1	Valid	Although the sponsor does not require application of the criterion, it is still applicable to the project.
2	Required	The sponsor requires some application of the criterion in the project (1–2 instances of requiring criterion application).
3	Critical	The sponsor requires extensive application of the criterion in the project (3 or more instances of requiring criterion application).

).

Consistent with prior studies [25,26], three expert judges used a three-step process to score design reports. First, judges collectively scored a sample of projects that were outside the scope of this study (i.e., produced by capstone groups not in the control or intervention cohorts) to establish scoring conventions (

Table 5. Sustainable Design Rubric, including scoring conventions for potential and earned points used in this study [25,26].

Design Criteria	Potential Points	Earned Points
Environmental Design Criteria
Minimizes natural resource depletion	1–3	0–3
Prevents waste	1–3	0–3
Protects natural ecosystems	1–3	0–3
Uses renewable energy sources	1–3	0–3
Uses inherently safe/benign materials (to environment)	1–3	0–3
Social Design Criteria
Addresses community and stakeholder requests	1–3	0–3
Considers local circumstances and cultures	1–3	0–3
Protects human health and well-being	3	0–3
Uses inherently safe/benign materials (to humans)	1–3	0–3
Sustainable Design Tools Design Criteria
Incorporates life cycle analysis	1–3	0–3
Incorporates environmental impact assessment (EIA) tools	1–3	0–3
Incorporates systems analysis	1–3	0–3
Uses innovative technologies	1–3	0–3
Economic Design Criteria
Considers economic impacts of environmental criteria	1–3	0–3
Considers economic impacts of social criteria	1–3	0–3
Conducts a cost and/or cost-benefit analysis	2	0–3

). Each criterion was deemed applicable to all student projects, leading to a minimum “potential” score of 1. Two criteria, “protects human health and well-being” and “conducts a cost and/or cost-benefit analysis”, received higher minimum potential scores, 3 and 2, respectively, because of across-the-board project requirements. Second, two judges individually applied the Rubric to each design report resulting from the two cohorts. Each project report was scored by the lead judge and one additional judge. Individual scores were recorded and discrepancies were discussed to reach a set of consensus scores, as per Besterfield-Sacre et al. [29]. Using consensus scores, Sustainable Design Indexes were calculated for each Rubric category as the difference between average potential and earned scores across criteria. Indexes range from −3 to 3, with values near 3 indicating high expectations and low sustainable design performance and values near −3 indicating low expectations and high sustainable design performance.

Statistical analyses were conducted using IBM SPSS Statistics. Based on judges’ individual scores, interrater reliability was quantified by Krippendorff’s alpha [30]. Using an SPSS macro [31], Krippendorff’s alpha for potential and earned scores for all criteria were calculated as 0.784 and 0.782, respectively. These values are within the range deemed “acceptable for exploratory research” [30]. Judge’s consensus scores were used for all subsequent statistical analyses. One-way Analysis of Variances (ANOVAs) were used to detect any significant differences between Sustainable Design Indexes, potential scores, and earned scores for the two cohorts.

2.4. Analysis of Students’ Perceptions

Students in both cohorts were asked to reflect on their sustainable design skills at the beginning and end of the semester using a seven-point, Likert-type survey. Specifically, students were asked to rate their confidence to demonstrate 12 sustainable design competencies adapted from the Nine Principles of Sustainable Engineering [9]. Pre- and post-survey responses were compared within each cohort using McNemar tests to analyze percentages of students who rated their confidence as a six or higher (π_6–7).

3. Results

Impacts of the learning-cycle-based sustainability module on students’ sustainable design skills were assessed through application of the Sustainable Design Rubric to capstone design reports and statistical comparison of earned points, potential points, and overall Sustainable Design Indexes between control and intervention cohorts. In addition, students’ perceptions of their sustainable design skills were captured using a Likert-type survey.

3.1. Sustainable Design Performance-Earned Scores

Earned scores, which capture sustainable design performance, differed for some criteria between cohorts (

Table 6. Comparison between earned scores for design projects completed by students enrolled in a traditional capstone design course (control cohort, n = 14) and a capstone course with an integrated sustainability module (intervention cohort, n = 20).

	Control	Intervention	ANOVA
	M (SD)	M (SD)	F (1, 32)	p
Environmental Design Criteria
Minimizes natural resource depletion	0.6 (0.9)	1.4 (1.1)	4.441	0.043 *
Prevents waste	0.4 (0.6)	0.7 (1.0)	0.758	0.391
Protects natural ecosystems	1.6 (1.2)	1.8 (1.1)	0.204	0.655
Uses renewable energy sources	0.0 (0.0)	0.1 (0.3)	1.464	0.235
Uses inherently safe/benign materials (to env.)	0.2 (0.6)	0.2 (0.5)	0.006	0.941
Average for Environmental Design Criteria	0.6 (0.3)	0.8 (0.6)	2.365	0.134
Social Design Criteria
Addresses community and stakeholder requests	2.2 (0.9)	2.6 (0.5)	2.587	0.118
Considers local circumstances and cultures	0.6 (0.8)	1.4 (1.1)	4.149	0.050 *
Protects human health and well-being	2.8 (0.4)	2.9 (0.7)	0.100	0.754
Uses inherently safe/benign materials (to humans)	0.0 (0.0)	0.10 (0.4)	0.693	0.411
Average for Social Design Criteria	1.4 (0.4)	1.7 (0.3)	7.305	0.011 *
Sustainable Design Tools Design Criteria
Incorporates life cycle analysis	0.5 (0.7)	0.4 (0.7)	0.184	0.671
Incorporates EIA tools	0.4 (0.7)	0.5 (0.8)	0.266	0.610
Incorporates systems analysis	1.4 (1.0)	1.8 (0.7)	1.771	0.193
Uses innovative technologies	0.7 (1.0)	1.1 (1.3)	0.878	0.356
Average for Sustainable Design Tools Criteria	0.7 (0.5)	0.9 (0.6)	1.093	0.304
Economic Design Criteria
Considers economic impacts of env. criteria	0.4 (0.7)	0.5 (0.8)	0.266	0.610
Considers economic impacts of social criteria	0.4 (0.9)	0.6 (0.9)	0.159	0.692
Conducts a cost and/or cost-benefit analysis	1.9 (0.7)	1.4 (0.6)	3.849	0.059
Average for Economic Design Criteria	0.9 (0.4)	0.8 (0.6)	0.139	0.712
Average for all Sustainable Design Criteria	0.9 (0.3)	1.1 (0.3)	3.397	0.075

* p ≤ 0.05.

). For instance, the mean earned score for “minimizes natural resource depletion” was significantly higher (p ≤ 0.05) for intervention projects (M = 1.4) as compared to control projects (M = 0.6). In addition, the mean earned score for “considers local circumstances and cultures” was also statistically greater (p ≤ 0.05) for intervention projects (M = 1.4) than for control projects (M = 0.6). However, overall earned scores for all 16 criteria were approximately 1.0 for both cohorts, indicating that student sustainable design capabilities were still “developing” (Table 3).

3.2. Sustainable Design Expectations–Potential Points

Potential scores, which capture sponsors’ sustainable design expectations, were similar for all projects. In fact, no statistical differences between potential scores for any of the 16 criteria were identified based on cohort (

Table 7. Comparison between potential scores for design projects completed by students enrolled in a traditional capstone design course (control cohort, n = 14) and a capstone course with an integrated sustainability module (intervention cohort, n = 20).

	Control	Intervention	ANOVA
	M (SD)	M (SD)	F (1, 32)	p
Environmental Design Criteria
Minimizes natural resource depletion	1.2 (0.4)	1.2 (0.4)	0.010	0.922
Prevents waste	1.0 (0.0)	1.0 (0.0)	-1	-
Protects natural ecosystems	1.4 (0.5)	1.5 (0.6)	0.012	0.915
Uses renewable energy sources	1.0 (0.0)	1.0 (0.0)	-	-
Uses inherently safe/benign materials (to env.)	1.0 (0.0)	1.0 (0.0)	-	-
Average for Environmental Design Criteria	1.1 (0.1)	1.1 (0.2)	0.001	0.979
Social Design Criteria
Addresses community and stakeholder requests	1.9 (0.5)	2.0 (0.5)	0.194	0.663
Considers local circumstances and cultures	1.0 (0.0)	1.2 (0.4)	3.294	0.079
Protects human health and well-being	3.0 (0.0)	3.0 (0.0)	-	-
Uses inherently safe/benign materials (to humans)	1.0 (0.0)	1.0 (0.0)	-	-
Average for Social Design Criteria	1.7 (0.1)	1.8 (0.2)	1.601	0.215
Sustainable Design Tools Design Criteria
Incorporates life cycle analysis	1.0 (0.0)	1.0 (0.0)	-	-
Incorporates EIA tools	1.1 (0.3)	1.1 (0.2)	0.064	0.801
Incorporates systems analysis	1.2 (0.4)	1.2 (0.4)	0.222	0.641
Uses innovative technologies	1.1 (0.3)	1.1 (0.3)	0.079	0.781
Average for Sustainable Design Tools Criteria	1.1 (0.1)	1.1 (0.1)	0.091	0.764
Economic Design Criteria
Considers economic impacts of env. criteria	1.1 (0.3)	1.0 (0.0)	1.448	0.238
Considers economic impacts of social criteria	1.0 (0.0)	1.1 (0.2)	0.693	0.411
Conducts a cost and/or cost-benefit analysis	2.0 (0.0)	2.0 (0.0)	-	-
Average for Economic Design Criteria	1.4 (0.1)	1.3 (0.1)	0.064	0.801
Average for all Sustainable Design Criteria	1.3 (0.1)	1.3 (0.1)	0.190	0.666

¹ Potential scores identical for control and intervention cohorts.

). Overall, the mean potential scores for each the control and intervention projects were 1.3 out of a maximum 3.0 points, indicating that sustainable design criteria were “valid” although not “required” by project sponsors (Table 3). As a result, student groups, regardless of cohort, could have met sustainable design criteria, even without encouragement from project sponsors.

3.3. Sustainable Design Indexes

Sustainable Design Indexes, which consider both student performance and sponsor expectations, were somewhat impacted by participation in the sustainability module (

Table 8. Comparison between sustainable design indexes [M_potential − M_earned] for design projects completed by teams enrolled in a traditional capstone design course (control cohort, n = 14) and a capstone course with an integrated sustainability module (intervention cohort, n = 20).

	Control	Intervention	ANOVA
	M (SD)	M (SD)	F (1, 32)	p
Environmental Design Criteria	0.6 (0.3)	0.3 (0.5)	3.126	0.087
Social Design Criteria	0.3 (0.3)	0.1 (0.3)	5.220	0.029 *
Sustainable Design Tools Criteria	0.4 (0.5)	0.1 (0.6)	1.390	0.247
Economic Design Criteria	0.5 (0.4)	0.5 (0.5)	0.123	0.728
All Sustainable Design Criteria	0.4 (0.2)	0.2 (0.3)	3.637	0.066

* p ≤ 0.05.

). While indexes tended to be lower for the intervention cohort, as compared to the control cohort, this relationship was only statistically significant for the Social Design Criteria (p ≤ 0.05). As indexes are calculated as the difference between mean potential and earned scores, a decrease in values is desirable because it suggests a sustainable design performance that exceeds sponsor expectations.

3.4. Student Perceptions

Of the 12 sustainable design competencies that students reflected on, intervention students showed improved confidence for ten, as compared to one for control students (

Table 9. Confidence related to sustainable design competencies before and after a traditional capstone design course (control cohort) and a capstone course with an integrated sustainability module (intervention cohort).

Prompt: Statements Below are Related to Sustainable Design. Indicate How Confident You are in Your Ability to Develop Designs that Meet the Criteria.	Control Cohort (n = 47, df = 1) [π6–7] (%)			Intervention Cohort (n = 84, df = 1) [π6–7] (%)
	Pre	Post	1 χ2	Pre	Post	1 χ2
Addresses community/stakeholder requests	59.6	55.3	0.20	38.1	58.3	7.12 **
Considers local circumstances and cultures	55.3	46.8	0.80	36.9	52.4	4.67 *
Incorporates life cycle analysis	36.2	40.4	0.29	23.8	52.4	17.65 ***
Incorporates EIA tools	27.7	31.9	0.25	26.2	50.0	11.06 ***
Incorporates systems analysis	29.8	42.6	2.00	29.8	52.4	8.84 **
Uses innovative technologies	25.5	34.0	1.14	23.8	44.0	8.75 **
Minimizes natural resource depletion	21.3	44.7	5.26 *	34.5	69.0	19.34 ***
Prevents waste	27.7	42.6	3.27	36.9	51.2	4.60 *
Protects natural ecosystems	44.7	53.2	0.67	39.3	53.6	4.34 *
Protects human health and well-being	59.6	57.4	0.07	51.2	48.8	0.07
Uses inherently safe and benign materials	53.2	59.6	0.53	44.0	58.3	3.72
Uses renewable energy sources	40.4	42.6	0.07	29.8	53.6	11.06 ***

* p ≤ 0.05; ** p < 0.01; *** p ≤ 0.001; ¹ McNemar χ-Calculated with Yates correction factor.

). The most significant improvements among intervention students were for “incorporates life cycle analysis,” “incorporates environmental impact assessment tools,” “minimizes natural resource depletion,” and “uses renewable energy sources.” For control students, the only significant improvement was for “minimizes natural resource depletion.”

4. Discussion

4.1. To What Extent Does Participation in the Module Impact Teams’ Tendencies to Apply Sustainable Design Criteria in Their Capstone Projects?

The expectation was that teams in the intervention cohort would demonstrate a greater execution of sustainable design in their capstone projects compared to the control cohort. Application of the Rubric to capstone project reports showed that intervention teams addressed two sustainable design criteria more frequently than control teams (Table 6). The two criteria—“minimizes natural resource depletion” and “considers local circumstances and cultures”—represented two different sustainability areas–environmental and social sustainability, respectively. Thus, student abilities to address two of three major sustainability areas (economic, environmental, social) were improved. Furthermore, consideration of “minimizes natural resource depletion” was twice as frequent among intervention teams, as compared to control teams, which is significant because the environmental area was the least addressed by teams by both cohorts. In addition, consideration of “considers local circumstances and cultures” was twice as frequent among intervention teams, as compared to control teams, which is significant because it was among the least considered Social Design Criteria by both cohorts. Overall, the difference between sustainable design performance for the intervention and control cohorts was not as large as anticipated, indicating a need to assist students with translating their knowledge gains into action.

4.2. Between Cohorts, How Do Teams’ Tendencies to Apply Sustainable Design Criteria in Their Capstone Projects Compare to Sponsors’ Expectations?

Sponsors’ sustainability expectations, as captured by the potential points scale, did not differ between the control and intervention cohorts (Table 7). Expectations were highest for the Social Design Criteria and lowest for the Environmental Design and Sustainable Design Tools Criteria across both cohorts. Consequently, the control and intervention teams had similar opportunities to integrate sustainability into their capstone projects.

When comparing sustainable design performance (earned points) to sponsor expectations (potential points), both similarities and differences were observed between the cohorts (Table 8). First, for both cohorts, the Indexes [M_potential − M_earned] for Economic Design Criteria were farthest from zero in the positive direction, which means that teams least effectively addressed economic sustainability in their designs. Second, for both cohorts, the Indexes for Social Design Criteria were closest to zero, which means that teams most effectively met sponsor expectations concerning social sustainability in their designs. The only statistically different Index between the cohorts was for the Social Design Criteria. Specifically, the Index for Social Design Criteria was lowest for the intervention cohort, which supports the idea that intervention teams addressed social design expectations more completely than control groups. Overall, the module had the greatest impact on teams’ abilities to address social sustainability considerations during design. However, the module did not address the gap in economic design expectations and performance.

4.3. Which Sustainable Design Competencies are Most Impacted by Module Participation, According to Students’ Individual Perceptions?

Intervention students’ confidences improved for almost all sustainable design competencies, while control students’ confidences improved for only one competency (Table 9). Indeed, only intervention students showed improved confidence in their ability to consider social sustainability during design and apply sustainable design tools. Both cohorts showed improved confidence in their ability to consider environmental sustainability during design; however, the intervention cohort reported improved confidence related to more environmental design competencies than the control cohort. It is also important to note that students’ initial confidence (indicated by the “pre” condition in Table 9) differed across criteria and in some cases, such as the social design competencies, the control cohort actually started with substantially greater confidence than the intervention cohort. While data analysis does not explain why such differences existed between cohorts, the finding may have practical implications for how instructors could use incoming capstone students’ differing confidences to target sustainable design instruction at building confidence in particular areas.

Some alignment was observed between analysis of project reports and students’ perceptions of sustainable design skills. Students in the intervention cohort felt more confident in their social design competencies (Table 9) and also demonstrated more frequent application of socially oriented criteria in their team project reports (Table 6 and Table 8). In particular, application of the Rubric showed greater consideration of “considers local circumstances and cultures” and “minimizes natural resource depletion” in intervention project reports, as compared to control reports (Table 6). Indeed, intervention students reported a highly significant increase in their confidence to “consider local circumstances and cultures,” while control students did not (Table 9). Both cohorts reported an increase in their confidence to “minimize natural resource depletion,” although the increase was more substantial for the intervention cohort, as compared to the control cohort (Table 9).

4.4. Insights for Future Module Implementations

While earlier studies clearly show that participation in the module helped students to develop a richer understanding of sustainability [17,19,23], the impacts on sustainable design skills are somewhat mixed. Students participating in the module felt more confident in their ability to meet ten out of twelve sustainable design competencies, as compared to only one for students not participating in the module (Table 9). Thus, the module had a substantial impact on students’ perceptions of their sustainable design skills. However, when experts evaluated team reports using the Sustainable Design Rubric, intervention teams only addressed two criteria (out of 16) more extensively than control teams (Table 6). While improved consideration of the two criteria—“minimizes natural resource depletion” and “considers local circumstances and cultures”—demonstrated mastery across sustainability areas (environmental and social), there was much more opportunity for the intervention teams to further improve skills related to the other 14 criteria. The modest improvement in sustainable design performance may indicate that although students learned new sustainability concepts and felt more confident in their abilities after completing the module, they did not necessarily translate those new concepts to their capstone projects. From this study, one important question emerged that will guide future module implementations: Why did students’ heightened sustainable design confidences not manifest as improved sustainable design performance in their capstone projects?

It is possible that while students were more confident in their sustainable design skills, teams were not motivated to address sustainability during design because of a lack of sponsor and/or course requirements. The potential scores for most criteria were close to “1,” which indicates that sponsors did not explicitly require that teams address them (Table 7). Indeed, the criteria with the highest potential scores—“protects human health and well-being,” “conducts a cost and/or cost-benefit analysis,” “addresses community and stakeholder requests,” and “protects natural ecosystems”—also had the highest earned scores. Thus, as teams tended to address those criteria that were emphasized by sponsors, explicit sustainable design expectations may be instituted alongside future module implementations to incentivize the application of acquired skills. For example, sponsors and/or instructors could use the Rubric to collaboratively set sustainable design expectations at the beginning of the semester. In addition, sustainable design performance could be incorporated into grading of capstone projects using the Rubric or other measures. The Rubric could be used as a formative assessment to help students identify areas for improvement prior to submission of their final report. Using the Rubric formatively could reinforce the new concepts that students learned during the sustainable design module and from their prior related coursework.

4.5. Study Limitations

Previously, limitations to the study design and module content were discussed [19]. In short, while quasi-experimental designs can suffer from low internal validity, true experimental designs are often not feasible in educational research due to ethical and operational issues associated with randomly assigning students to groups [32]. Efforts were made to limit threats to internal validity by ensuring similar conditions between the control and intervention cohorts. For example, the same two instructors administered both courses. In addition, scaffolding and requirements for the capstone project were similar, including the process for assignment to teams and selection of projects. Related to content, it is acknowledged that the module may not encompass the full realm of sustainability; however, emphasis on the three fundamental pillars (economy, environment, society) provide engineering students with the basic information needed to practice operating under a sustainable design paradigm.

Additional limitations related to assessments used in this study are acknowledged. First, only written design reports were reviewed to evaluate sustainable design performance (earned points) and expectations (potential points). It is possible that teams did not properly capture sponsors’ expectations; however, inclusion of design requirements was a mandated element of reports. Those implementing the module in the future might consider using the Rubric as a pedagogical tool to facilitate discussions between teams and their sponsors to scaffold sustainable design and make assignment of potential points more straightforward. Similarly, it is possible that teams considered criteria but did not include related work in their reports. While reports might not contain all sustainable design activities, they likely include those that were most integral to the design process. Overall, project reports proved to be artifacts that allowed researchers to review the most pertinent elements of sustainable design expectations and performance.

Finally, self-report surveys were used as one tool to capture changes in students’ sustainable design skills during the courses. Other authors have shown that students often over-rate their own knowledge and skills [33]. Indeed, the discrepancy between Rubric scores and students’ perceptions may be because students over-estimated their sustainable design competencies. Even so, the belief that one can complete a task (i.e., self-efficacy) can impact future success in some domains (e.g., [34]). Thus, even if students’ skills are less than indicated, their belief that they can demonstrate sustainable design competencies might be sufficient for them to at least attempt to do so in future module implementations that incorporate a source of external motivation.

5. Conclusions

A study was conducted to analyze how participation in a learning-cycle-based sustainability module, already shown to improve conceptual understanding, impacts students’ sustainable design skills. Civil and environmental engineering students enrolled in a capstone design course participated in the study, with one cohort completing module activities (intervention cohort) and another cohort only completing the traditional course (control cohort). Team design reports were analyzed using the Sustainable Design Rubric to compare sustainable design performance between cohorts. In addition, students reflected on their sustainable design skills via a Likert-type survey at the beginning and end of the semester. The following conclusions were made based on the results:

• Based on evaluation of capstone design reports, improvement in intervention teams’ consideration of sustainable design criteria was somewhat limited, as they more extensively addressed only 2 of 16 sustainable design compared to control teams;
• For both control and intervention cohorts, sustainable design expectations were similar, and teams generally addressed those criteria that were emphasized by sponsors;
• Intervention students reported improved confidence in more sustainable design competencies than control students (10 of 12 for intervention students; 1 of 12 for control students);
• For future implementations, explicit sustainable design expectations should be set for project teams, and performance should be reflected in grading schemes. Clearer and more extensive expectations might close the gap between students’ perceptions of improved skills and teams’ actual application of sustainable design criteria.