Comparing Measures of Student Sustainable Design Skills Using a Project-Level Rubric and Surveys

: Civil engineers are poised to impact sustainable development. Consequently, there is a need for curricular materials to sca ﬀ old students in developing sustainable design skills. Previously, a sustainability module, based on Kolb’s learning cycle, was integrated into a civil engineering capstone course in the United States. The purpose of this study was to analyze the extent to which students engaging in the module (intervention cohort) were able to improve their sustainable design skills, as compared to a group of capstone students not participating in the module (control cohort). A Sustainable Design Rubric was used to assess students’ sustainable design performance, as captured in capstone reports. In addition, students reﬂected on their conﬁdence related to several sustainable design competencies via a survey. Based on an evaluation of capstone design reports, improvement in the intervention teams’ consideration of sustainable design criteria was somewhat limited, as they more extensively addressed only 2 of the 16 sustainable design criteria compared to control teams. Intervention students reported improved conﬁdence in more sustainable design competencies than control students (10 of 12 for intervention students; 1 of 12 for control students). For future implementations, clearer and more extensive sustainable design expectations need to be set by instructors and project sponsors to increase the execution of sustainable design and close the gap between students’ perceptions of improved skills and teams’ actual application of sustainable design criteria.


Sustainable Design
Design is fundamental to engineering practice and is even considered to be "the essence of engineering" [1] (p. 1). Dym et al. [2] provide a thoughtful narrative on engineering design, describing it as: "a systematic, intelligent process in which designers generate, evaluate, and specify concepts for devices, systems, or processes whose form and function achieve clients' objectives or users' needs while satisfying a specified set of constraints" (p. 104). Even so, ABET, the body that accredits undergraduate engineering programs in the United States (US), defines design as "a process of devising Table 1. Nine Principles of Sustainable Engineering [9].

No. Principle
1 Engineer processes and products holistically using system analysis. 2 Conserve and improve natural ecosystems while protecting human health and wellbeing. 3 Use life cycle thinking in all engineering activities. 4 Ensure that all material and energy inputs and outputs are as inherently safe and benign as possible. 5 Minimize depletion of natural resources. 6 Strive to prevent waste. 7 Develop and apply engineering solutions, while considering local circumstances and cultures. 8 Create engineering solutions beyond current or dominant technologies. 9 Actively engage communities and stakeholders in development of engineering solutions.

Sustainable Design Education in Civil Engineering
The American Society of Civil Engineers (ASCE) has called for reform of engineering practice and subsequently engineering education as well. Recently, ASCE published a five-year sustainability roadmap to "transform the profession" [11]. In addition, the ASCE Code of Ethics mandates that engineers "strive to comply with the principles of sustainable development in the performance of their professional duties" [12]. Regardless of discipline, ABET specifies that all undergraduate students demonstrate "an ability to apply engineering design to produce solutions that meet specified needs with consideration of . . . global, cultural, social, environmental, and economic factors" [3]. Consequently, there is a need to develop curricular materials and educational experiences to scaffold students in developing and applying sustainable design skills so that they can revolutionize engineering practice in the future.
Within civil engineering curricula, the capstone design course is a common target for innovation related to sustainability. In capstone courses, teams of senior engineering students typically develop designs in response to a real-world problem. Brunell [13] reports on a civil engineering capstone design course where students were introduced to the United Nations Sustainable Development Goals (UNSDGs) and asked to choose one or more to address in their civil engineering capstone projects. Delatte and Hatley [14] and Burian [15] used established rating systems, such as LEED ® and/or Envision ® , to scaffold civil engineering students in learning about and integrating sustainability into their capstone projects. Alternatively, Payne and Aidoo [16] developed a community engagement module that students completed concurrently with their capstone course. Student teams were required to develop economically, socially, and environmentally focused design alternatives and then synthesize alternatives into a final capstone design. Common among these interventions is the need for instructors to address limitations in students' prior knowledge about sustainably prior to design. As Delatte and Hatley [14] summarize, students' prior experiences with sustainability can vary significantly even within a single capstone course.

Learning-Cycle-Based Sustainability Module
A sustainability module was previously developed to guide civil engineering students in learning about and applying sustainability principles during design [17][18][19][20]. The module was designed to lead students through activities that were structured to follow Kolb's learning cycle [21,22]. In Session 1, students reflect on their prior sustainability knowledge using concept maps. In Session 2, students prepare and participate in peer lectures to learn about sustainable development and related themes: environmental, economic, and social sustainability, as well as sustainable design tools. In Session 3, students analyze case studies to provide examples of what sustainable design "looks like" in real-world projects. During Session 4, students apply their knowledge by completing a preliminary sustainability analysis of their semester design project. Finally, in Session 5, students reflect again on their sustainability knowledge using concept maps.
The sustainability module has been implemented at two different institutions in the US with students across academic years in civil engineering [17,19] and interdisciplinary engineering [18,23] programs. In previous studies, the impacts of the module on students' conceptual understanding of sustainability were assessed using concept maps and student self-report surveys. Concept maps universally showed that students improved the breadth, depth, and connectedness of their sustainability knowledge. Content analysis of concept maps generally showed an emphasis on environmental concepts prior to module completion and a more balanced understanding of sustainability across areas after module completion. However, even after module completion, the economic area tended to be somewhat under-represented in concept maps [24]. Survey results generally aligned with concept map findings, although civil engineering students completing the module as part of a cornerstone design course perceived greater gains in their sustainability knowledge than students completing the module as part of a capstone design course [19]. While multiple studies support the efficacy of the learning-cycle-based module for improving sustainability knowledge among undergraduate engineering students, additional work is needed to capture impacts on sustainable design skills.

Sustainable Design Rubric
Previously, a three-phase process was used to develop a Sustainable Design Rubric for undergraduate-level design projects [25,26]. In the first phase, professional rating systems (e.g., LEED ® and Envision ® ) were explored, but they were deemed too complex and/or narrow in scope for typical student courses that span a semester or academic year. However, the Nine Principles of Sustainable Engineering [9] (Table 1) were identified as a set of broad, accessible guidelines that might be applicable to a range of student engineering projects. In the second phase, a pilot rubric, including 16 sustainable design criteria, was created based on the Nine Principles (Table 2). To measure achievement of the criteria, two four-point rating scales were created, with the "potential" scale capturing the extent to which each criterion is applicable to a given project and the "earned" scale capturing the extent to which a student or team addresses each criterion in their design. Based on a repository of civil engineering capstone projects, examples of how each criterion might be applied were summarized to assist with project scoring. In the third phase, the rubric was reviewed and refined by an expert panel to establish content validity. The final Sustainable Design Rubric is composed of 16 criteria across four categories [25,26]. The Sustainable Design Rubric was used at a research-intensive institution in the US to evaluate a sample of civil and environmental engineering (CEE) capstone design projects completed over a ten-year period [26]. Researchers found that students' sustainable design skills, as evidenced in the final reports, only changed slightly over the period for some economic design criteria. In general, researchers found that social design criteria were most frequently addressed, specifically "protect human health and well-being" and "address community and stakeholder requests." The Economic category was the second most emphasized, especially "conduct a cost and/or cost-benefit analysis", which is a required project component at the institution. Criteria from the Environmental and Design Tools categories were applied to a lesser extent. Overall, the Rubric proved to be a useful tool for systematically evaluating the extent to which sustainability is considered in undergraduate CEE design projects.

Project Scope
The goal of the current study is to examine the impact of the learning-cycle-based sustainability module on students' sustainable design skills. In a previous study [17], students participating in a civil engineering capstone design course with the embedded module (intervention cohort) showed greater gains in sustainability knowledge, as compared to students participating in the capstone course alone (control cohort). Student sustainable design skills will be assessed and compared between the intervention and control cohorts through a self-report survey of sustainable design competencies and application of the Rubric to team project reports ( Table 2). The following questions will be addressed: (1) To what extent does participation in the module impact teams' tendencies to apply sustainable design criteria in their capstone projects? (2) Between cohorts, how do teams' tendencies to apply sustainable design criteria in their capstone projects compare to sponsors' expectations? (3) Which sustainable design competencies are most impacted by module participation, according to students' individual perceptions? Ultimately, the results will be used to inform the improvement in future module implementations, especially related to the development of students' sustainable design skills.

Study Site
The study was conducted at a large, research-intensive institution in the Southeastern US with students enrolled in a CEE capstone design course. Students were seniors who had completed all prerequisite design courses in CEE sub-disciplines. Most capstone participants were male (78.4%) civil engineering (79.1%) students from the US (83.6%). The course was project-based, with student teams (3-6 students) working with instructors and engineering professionals ("sponsors") to complete an authentic design. One of the primary deliverables was a comprehensive report that presented the final design to sponsors. Incorporating sustainability into the projects was not a course requirement, although sponsors may have conveyed sustainability expectations to teams. Most students' primary exposure to sustainability concepts was during their second academic year when taking a civil engineering systems course, which is intended to introduce students to sustainability from a systems perspective [27,28].

Experimental Design
A quasi-experimental design [17] was used to investigate the impacts of the module on students' sustainable design skills. Students enrolled in a traditional, unamended capstone design course served as the control cohort. Students enrolled in the capstone design course the following semester participated in module activities and served as the intervention cohort. Students in the intervention cohort were required to complete all module assignments, which accounted for 5% of their overall course grade. Other than participation in the module, elements of the two capstone courses were similar, including the deliverables and co-instructors.
Sustainable design skills were assessed via a student survey and application of the Sustainable Design Rubric [25,26] to team project reports. The survey was administered to both cohorts at the beginning and end of the semester to capture their perceptions of several sustainable design competencies, which were composed based on the Nine Principles of Sustainable Engineering [9] (Tables 1 and 2). Of the 67 students in the control cohort, 47 completed design skills surveys (70% participation). Of the 100 students in the intervention cohort, 84 completed design skills surveys (84% participation). Students in the control cohort were organized into 14 groups, while students in the intervention cohort were organized into 20 groups. Each group composed a final report that was assessed as part of the study.

Analysis of Team Design Reports
Sustainable design skills across the control and intervention cohorts were assessed by applying the previously developed Sustainable Design Rubric to team design projects, as documented in the final reports. Scoring encompassed assigning "potential" and "earned" points, for each of the 16 sustainable design criteria, to capture sponsors' sustainable design expectations and groups' sustainable design performance, respectively. Potential and earned points were assessed on a four-point scale ranging from no expectation/performance to extensive expectations/performance (Tables 3 and 4). Table 3. Earned points scale to capture sustainable design expectations [25,26].

Unacceptable
Criterion not at all considered in project report.
1 Developing Criterion mentioned or discussed in the project report, but not applied in design process.

Competent
Project report shows evidence that the criterion was adequately applied in design process (1-2 instances of criterion application).

Exemplary
Project report shows evidence that the criterion was extensively applied in the design process (3 or more instances of criterion application). Table 4. Potential points scale to capture sustainable design expectations [25,26].

Inapplicable
The criterion is not at all valid for the project.
1 Valid Although the sponsor does not require application of the criterion, it is still applicable to the project.

Required
The sponsor requires some application of the criterion in the project (1-2 instances of requiring criterion application).

Critical
The sponsor requires extensive application of the criterion in the project (3 or more instances of requiring criterion application).
Consistent with prior studies [25,26], three expert judges used a three-step process to score design reports. First, judges collectively scored a sample of projects that were outside the scope of this study (i.e., produced by capstone groups not in the control or intervention cohorts) to establish scoring conventions (Table 5). Each criterion was deemed applicable to all student projects, leading to a minimum "potential" score of 1. Two criteria, "protects human health and well-being" and "conducts a cost and/or cost-benefit analysis", received higher minimum potential scores, 3 and 2, respectively, because of across-the-board project requirements. Second, two judges individually applied the Rubric to each design report resulting from the two cohorts. Each project report was scored by the lead judge and one additional judge. Individual scores were recorded and discrepancies were discussed to reach a set of consensus scores, as per Besterfield-Sacre et al. [29]. Using consensus scores, Sustainable Design Indexes were calculated for each Rubric category as the difference between average potential and earned scores across criteria. Indexes range from −3 to 3, with values near 3 indicating high expectations and low sustainable design performance and values near −3 indicating low expectations and high sustainable design performance. Table 5. Sustainable Design Rubric, including scoring conventions for potential and earned points used in this study [25,26].
Judge's consensus scores were used for all subsequent statistical analyses. One-way Analysis of Variances (ANOVAs) were used to detect any significant differences between Sustainable Design Indexes, potential scores, and earned scores for the two cohorts.

Analysis of Students' Perceptions
Students in both cohorts were asked to reflect on their sustainable design skills at the beginning and end of the semester using a seven-point, Likert-type survey. Specifically, students were asked to rate their confidence to demonstrate 12 sustainable design competencies adapted from the Nine Principles of Sustainable Engineering [9]. Pre-and post-survey responses were compared within each cohort using McNemar tests to analyze percentages of students who rated their confidence as a six or higher (π 6-7 ).

Results
Impacts of the learning-cycle-based sustainability module on students' sustainable design skills were assessed through application of the Sustainable Design Rubric to capstone design reports and statistical comparison of earned points, potential points, and overall Sustainable Design Indexes between control and intervention cohorts. In addition, students' perceptions of their sustainable design skills were captured using a Likert-type survey.

Sustainable Design Performance-Earned Scores
Earned scores, which capture sustainable design performance, differed for some criteria between cohorts (Table 6). For instance, the mean earned score for "minimizes natural resource depletion" was significantly higher (p ≤ 0.05) for intervention projects (M = 1.4) as compared to control projects (M = 0.6). In addition, the mean earned score for "considers local circumstances and cultures" was also statistically greater (p ≤ 0.05) for intervention projects (M = 1.4) than for control projects (M = 0.6). However, overall earned scores for all 16 criteria were approximately 1.0 for both cohorts, indicating that student sustainable design capabilities were still "developing" (Table 3). Table 6. Comparison between earned scores for design projects completed by students enrolled in a traditional capstone design course (control cohort, n = 14) and a capstone course with an integrated sustainability module (intervention cohort, n = 20).

Control
Intervention ANOVA

Sustainable Design Expectations-Potential Points
Potential scores, which capture sponsors' sustainable design expectations, were similar for all projects. In fact, no statistical differences between potential scores for any of the 16 criteria were identified based on cohort (Table 7). Overall, the mean potential scores for each the control and intervention projects were 1.3 out of a maximum 3.0 points, indicating that sustainable design criteria were "valid" although not "required" by project sponsors ( Table 3). As a result, student groups, regardless of cohort, could have met sustainable design criteria, even without encouragement from project sponsors. Table 7. Comparison between potential scores for design projects completed by students enrolled in a traditional capstone design course (control cohort, n = 14) and a capstone course with an integrated sustainability module (intervention cohort, n = 20).

Control
Intervention

Sustainable Design Indexes
Sustainable Design Indexes, which consider both student performance and sponsor expectations, were somewhat impacted by participation in the sustainability module (Table 8). While indexes tended to be lower for the intervention cohort, as compared to the control cohort, this relationship was only statistically significant for the Social Design Criteria (p ≤ 0.05). As indexes are calculated as the difference between mean potential and earned scores, a decrease in values is desirable because it suggests a sustainable design performance that exceeds sponsor expectations.

Student Perceptions
Of the 12 sustainable design competencies that students reflected on, intervention students showed improved confidence for ten, as compared to one for control students ( Table 9). The most significant improvements among intervention students were for "incorporates life cycle analysis," "incorporates environmental impact assessment tools," "minimizes natural resource depletion," and "uses renewable energy sources." For control students, the only significant improvement was for "minimizes natural resource depletion." Table 9. Confidence related to sustainable design competencies before and after a traditional capstone design course (control cohort) and a capstone course with an integrated sustainability module (intervention cohort).

To What Extent Does Participation in the Module Impact Teams' Tendencies to Apply Sustainable Design Criteria in Their Capstone Projects?
The expectation was that teams in the intervention cohort would demonstrate a greater execution of sustainable design in their capstone projects compared to the control cohort. Application of the Rubric to capstone project reports showed that intervention teams addressed two sustainable design criteria more frequently than control teams ( Table 6). The two criteria-"minimizes natural resource depletion" and "considers local circumstances and cultures"-represented two different sustainability areas-environmental and social sustainability, respectively. Thus, student abilities to address two of three major sustainability areas (economic, environmental, social) were improved. Furthermore, consideration of "minimizes natural resource depletion" was twice as frequent among intervention teams, as compared to control teams, which is significant because the environmental area was the least addressed by teams by both cohorts. In addition, consideration of "considers local circumstances and cultures" was twice as frequent among intervention teams, as compared to control teams, which is significant because it was among the least considered Social Design Criteria by both cohorts. Overall, the difference between sustainable design performance for the intervention and control cohorts was not as large as anticipated, indicating a need to assist students with translating their knowledge gains into action.

Between Cohorts, How Do Teams' Tendencies to Apply Sustainable Design Criteria in Their Capstone Projects Compare to Sponsors' Expectations?
Sponsors' sustainability expectations, as captured by the potential points scale, did not differ between the control and intervention cohorts (Table 7). Expectations were highest for the Social Design Criteria and lowest for the Environmental Design and Sustainable Design Tools Criteria across both cohorts. Consequently, the control and intervention teams had similar opportunities to integrate sustainability into their capstone projects.
When comparing sustainable design performance (earned points) to sponsor expectations (potential points), both similarities and differences were observed between the cohorts (Table 8). First, for both cohorts, the Indexes [M potential − M earned ] for Economic Design Criteria were farthest from zero in the positive direction, which means that teams least effectively addressed economic sustainability in their designs. Second, for both cohorts, the Indexes for Social Design Criteria were closest to zero, which means that teams most effectively met sponsor expectations concerning social sustainability in their designs. The only statistically different Index between the cohorts was for the Social Design Criteria. Specifically, the Index for Social Design Criteria was lowest for the intervention cohort, which supports the idea that intervention teams addressed social design expectations more completely than control groups. Overall, the module had the greatest impact on teams' abilities to address social sustainability considerations during design. However, the module did not address the gap in economic design expectations and performance.

Which Sustainable Design Competencies are Most Impacted by Module Participation, According to Students' Individual Perceptions?
Intervention students' confidences improved for almost all sustainable design competencies, while control students' confidences improved for only one competency (Table 9). Indeed, only intervention students showed improved confidence in their ability to consider social sustainability during design and apply sustainable design tools. Both cohorts showed improved confidence in their ability to consider environmental sustainability during design; however, the intervention cohort reported improved confidence related to more environmental design competencies than the control cohort. It is also important to note that students' initial confidence (indicated by the "pre" condition in Table 9) differed across criteria and in some cases, such as the social design competencies, the control cohort actually started with substantially greater confidence than the intervention cohort. While data analysis does not explain why such differences existed between cohorts, the finding may have practical implications for how instructors could use incoming capstone students' differing confidences to target sustainable design instruction at building confidence in particular areas.
Some alignment was observed between analysis of project reports and students' perceptions of sustainable design skills. Students in the intervention cohort felt more confident in their social design competencies (Table 9) and also demonstrated more frequent application of socially oriented criteria in their team project reports (Tables 6 and 8). In particular, application of the Rubric showed greater consideration of "considers local circumstances and cultures" and "minimizes natural resource depletion" in intervention project reports, as compared to control reports (Table 6). Indeed, intervention students reported a highly significant increase in their confidence to "consider local circumstances and cultures," while control students did not (Table 9). Both cohorts reported an increase in their confidence to "minimize natural resource depletion," although the increase was more substantial for the intervention cohort, as compared to the control cohort (Table 9).

Insights for Future Module Implementations
While earlier studies clearly show that participation in the module helped students to develop a richer understanding of sustainability [17,19,23], the impacts on sustainable design skills are somewhat mixed. Students participating in the module felt more confident in their ability to meet ten out of twelve sustainable design competencies, as compared to only one for students not participating in the module (Table 9). Thus, the module had a substantial impact on students' perceptions of their sustainable design skills. However, when experts evaluated team reports using the Sustainable Design Rubric, intervention teams only addressed two criteria (out of 16) more extensively than control teams (Table 6). While improved consideration of the two criteria-"minimizes natural resource depletion" and "considers local circumstances and cultures"-demonstrated mastery across sustainability areas (environmental and social), there was much more opportunity for the intervention teams to further improve skills related to the other 14 criteria. The modest improvement in sustainable design performance may indicate that although students learned new sustainability concepts and felt more confident in their abilities after completing the module, they did not necessarily translate those new concepts to their capstone projects. From this study, one important question emerged that will guide future module implementations: Why did students' heightened sustainable design confidences not manifest as improved sustainable design performance in their capstone projects?
It is possible that while students were more confident in their sustainable design skills, teams were not motivated to address sustainability during design because of a lack of sponsor and/or course requirements. The potential scores for most criteria were close to "1," which indicates that sponsors did not explicitly require that teams address them (Table 7). Indeed, the criteria with the highest potential scores-"protects human health and well-being," "conducts a cost and/or cost-benefit analysis," "addresses community and stakeholder requests," and "protects natural ecosystems"-also had the highest earned scores. Thus, as teams tended to address those criteria that were emphasized by sponsors, explicit sustainable design expectations may be instituted alongside future module implementations to incentivize the application of acquired skills. For example, sponsors and/or instructors could use the Rubric to collaboratively set sustainable design expectations at the beginning of the semester. In addition, sustainable design performance could be incorporated into grading of capstone projects using the Rubric or other measures. The Rubric could be used as a formative assessment to help students identify areas for improvement prior to submission of their final report. Using the Rubric formatively could reinforce the new concepts that students learned during the sustainable design module and from their prior related coursework.

Study Limitations
Previously, limitations to the study design and module content were discussed [19]. In short, while quasi-experimental designs can suffer from low internal validity, true experimental designs are often not feasible in educational research due to ethical and operational issues associated with randomly assigning students to groups [32]. Efforts were made to limit threats to internal validity by ensuring similar conditions between the control and intervention cohorts. For example, the same two instructors administered both courses. In addition, scaffolding and requirements for the capstone project were similar, including the process for assignment to teams and selection of projects. Related to content, it is acknowledged that the module may not encompass the full realm of sustainability; however, emphasis on the three fundamental pillars (economy, environment, society) provide engineering students with the basic information needed to practice operating under a sustainable design paradigm.
Additional limitations related to assessments used in this study are acknowledged. First, only written design reports were reviewed to evaluate sustainable design performance (earned points) and expectations (potential points). It is possible that teams did not properly capture sponsors' expectations; however, inclusion of design requirements was a mandated element of reports. Those implementing the module in the future might consider using the Rubric as a pedagogical tool to facilitate discussions between teams and their sponsors to scaffold sustainable design and make assignment of potential points more straightforward. Similarly, it is possible that teams considered criteria but did not include related work in their reports. While reports might not contain all sustainable design activities, they likely include those that were most integral to the design process. Overall, project reports proved to be artifacts that allowed researchers to review the most pertinent elements of sustainable design expectations and performance.
Finally, self-report surveys were used as one tool to capture changes in students' sustainable design skills during the courses. Other authors have shown that students often over-rate their own knowledge and skills [33]. Indeed, the discrepancy between Rubric scores and students' perceptions may be because students over-estimated their sustainable design competencies. Even so, the belief that one can complete a task (i.e., self-efficacy) can impact future success in some domains (e.g., [34]). Thus, even if students' skills are less than indicated, their belief that they can demonstrate sustainable design competencies might be sufficient for them to at least attempt to do so in future module implementations that incorporate a source of external motivation.

Conclusions
A study was conducted to analyze how participation in a learning-cycle-based sustainability module, already shown to improve conceptual understanding, impacts students' sustainable design skills. Civil and environmental engineering students enrolled in a capstone design course participated in the study, with one cohort completing module activities (intervention cohort) and another cohort only completing the traditional course (control cohort). Team design reports were analyzed using the Sustainable Design Rubric to compare sustainable design performance between cohorts. In addition, students reflected on their sustainable design skills via a Likert-type survey at the beginning and end of the semester. The following conclusions were made based on the results:

1.
Based on evaluation of capstone design reports, improvement in intervention teams' consideration of sustainable design criteria was somewhat limited, as they more extensively addressed only 2 of 16 sustainable design compared to control teams; 2.
For both control and intervention cohorts, sustainable design expectations were similar, and teams generally addressed those criteria that were emphasized by sponsors; 3.
Intervention students reported improved confidence in more sustainable design competencies than control students (10 of 12 for intervention students; 1 of 12 for control students);

4.
For future implementations, explicit sustainable design expectations should be set for project teams, and performance should be reflected in grading schemes. Clearer and more extensive expectations might close the gap between students' perceptions of improved skills and teams' actual application of sustainable design criteria.