Engineering Projects in Community Service (EPICS) in High Schools: Subtle but Potentially Important Student Gains Detected from Human-Centered Curriculum Design

Abstract

A major goal in Engineering training in the U.S. is to continue to both grow and diversify the field. Project- and service-based forms of experiential, problem-based learning are often implemented with this as a goal, and Engineering Projects in Community Service (EPICS) High is one of the more well-regarded and widely implemented. Yet, the evidence based on if and how participation in such programs shapes student intentions and commitment to STEM pathways is currently limited, most especially for pre-college programming. This study asks: How do high school students’ engineering mindsets and their views of engineering/engineers change as they participate in project–service learning (as implemented through an EPICS High curriculum)? This study employed a mixed method design, combining pre- and post-test survey data that were collected from 259 matched students (63% minority, 43% women) enrolling in EPICS High (total of 536 completed pre-tests, 375 completed post-tests) alongside systematic ethnographic analysis of participant observation data conducted in the same 13 socioeconomically diverse schools over a two-year period. Statistical analyses showed that participants score highly on engineering-related concepts and attitudes at both pre- and post-test. These did not change significantly as a result of participation. However, we detected nuanced but potentially important changes in student perspectives and meaning, such as shifting perceptions of engineering and gaining key transversal skills. The value of participation to participants was connected to changes in the meaning of commitments to pursue engineering/STEM.

1. Introduction

Professional Engineering training is not currently meeting perceived national needs for competitiveness, meaning there is a push to recruit, and then retain to graduation, larger cohorts of suitable students [1]. In addition, even though relative gains have been made, the goal of equitable representation of women and historically-underrepresented minority (URM) students in the Science, Technology, Engineering, and Math (STEM) fields in the U.S. has also not yet been achieved. For example, even though women now earn the majority of bachelor’s degrees overall, they represent only 20% of those awarded in Engineering. Similarly, African-American students graduate with 9.5% of the undergraduate degrees in the U.S. but represent only 3.8% of those in Engineering [2].

This underrepresentation in STEM begins prior to university entrance, being clearly evident in high school. For example, Asian and White students are much more likely to successfully complete calculus by the 12th grade compared to URM students [3]. Thus, there has been a significant national push to find the best means to both engage and build the relevant skills for diverse students, and students more generally, as potential STEM graduates while they are still in the K-12 system. A fundamental goal of this broader agenda is to integrate pedagogies that focus on collaborative forms of community service, practical application, and project-based learning that articulate solutions for real-world problems. Innovative high school mathematics and science curricula have been developed along these lines to address this challenge, both aiming to orient students towards and prepare them better for advanced study in these fields [4,5]. One of the most widely applied is EPICS (Engineering Projects in Community Service) High [6].

EPICS is one of the more widely applied means to meet this goal. The EPICS strategy is based on connecting students interested in engineering and computing design with local community partners to solve practical problems [7,8], including over extended semesters [9]. This strategy of “designing for others” [10] was originally intended for engaging college students, with the assumption that project-based and service-learning modalities would spark a broader interest in and commitment to STEM careers. The underpinning philosophy for Engineering, for example, is that highlighting and engaging Engineering’s concern for people, local communities, and broader societal welfare would better draw women and URM students into the field [9]. The approach of EPICS is consistent with an array of published research that indicates that use of dynamic curricula that incorporate service-learning and/or project-based learning predicts improved student outcomes in potentially advancing minority and female students in engineering/STEM at the college level [7,11,12,13,14,15].

In 2006, responding to this recognition that attrition from the STEM training pipeline was occurring earlier in students’ education and needed to be addressed sooner, the program developed a high school curriculum [16]. This fits with research showing that students who excel in mathematics in high school (specifically 10th- through 12th-grade math) and have self-belief in their mathematic abilities are more likely to pursue STEM degrees in college [14]. Early exposure to mathematics and science related courses seems to be key but should be done in a way that engages students and makes learning these subjects enjoyable [17,18,19]. Project/problem-based learning, particularly in early grades, can act to foster creativity through problem-solving, a much-needed skill in engineering [18]. Thus, earlier education is considered central to instilling the necessary skills and knowledge to create successful STEM pathways for future practitioners [19].

The EPICS curriculum was subsequently deployed in high schools across the country (“EPICS High”). At the high school level, the technical capacities of students to engage in human-centered engineering solutions to real work may be less [12], but the underlying assumptions remains the same: That such programs should: (a) Fundamentally change students’ perceptions of engineering by making the engineering design process fully human-centered and (b) that the greatest positive shifts in perception should be for URM and female students.

Operationally, EPICS High programming is integrated into existing classrooms and school arrangements. During the summer, math/science teachers are trained on the EPICS High curriculum, which is then incorporated into their STEM or career and technical education (CTE) classes, or in afterschool clubs. Usually, projects seek to engage with the needs of partners local to the specific school. The EPICS High approach fits within a broader trend in STEM education toward supporting innovations in STEM-project-based learning (PBL) [12,14,20]. Unlike problem-based learning—where students are given a hypothetical, real-world problem to solve through thinking through different steps [20]—project-based learning includes actual projects that have clients and real applications [11,12,14]. Specifically, PBL mimics professional work in that the projects take longer to complete, it applies knowledge rather than simply acquiring it, and students must self-direct their time and energies as they work on a team [14]. In general, student outcomes from participation in PBL include improved teamwork, greater communication skills, a better understanding of the complexity of professional problem-solving and the knowledge needed in order to apply solutions, as well as increased motivation to continue on [11,12]. Conversely, because of the project–solution focus, students may be deficient in understanding the fundamentals of engineering concepts [12]. Much of the research suggests the gains are seen in all participants, not just URM and female students.

The EPICS High approach also purposefully focuses student activity in service, not just project learning. As a form of pedagogy, service-learning is experiential education that educators design as structured opportunities where students partake in activities that address community needs in order to foster learning and development [21] (p. 5). Within engineering, service-learning can help students to build skills useful to the prolonged professional practice of engineering while also fostering civic responsibilities [15]. Service-learning is quite distinctive because, when done most effectively, it gives students a voice to choose what project they want to complete and it involves community partners—both factors that help to motivate students to gain the best learning outcomes as well as make the experience meaningful [22,23]. For example, students who fully engage in service-learning report enjoying school more and becoming more civically inclined [23]. Furthermore, students who engage in service-learning that incorporates STEM problem-based learning techniques increase their academic engagement, increase achievement in science, and become more resilient and civically engaged [13].

The college-based Engineering Projects in Community Service (EPICS) program typically integrates both service-learning and project-based learning into one program that spans one or more semesters. College students’ reflections on EPICS programs suggest students believe they gained skills in teamwork, communication, project planning, leadership, and that engineering could be viewed as a “caring profession” [6,9,24]. Moreover, Purdue University enrollment data over a decade suggest female students in Mechanical and Electrical and Computer Engineering majors were 170% more likely than male students to participate [9,25]. Using a case-study approach, Huff et al. (2012) [24] suggested benefits in this engaged learning (in international settings) also had value for advancing basic engineering competencies.

EPICS-related gains at high school levels are even less well studied, but early signs suggest that impacts might be hampered by a lack of basic skills (such as mathematics). In a pilot study comparing EPICS High program participants to those in another program without the service-learning components, Kelly et al. (2010) [26] reported participants in both programs exhibited significant challenges in solving basic problems; differences by gender or ethnicity were not studied (the sample was small) [26]. Zoltowski, Oakes, and Cardella (2012) [27] used a qualitative–phenomenological approach to identify the ways that 33 students, enrolled in EPICS and in several similar programs, understood and reacted to the idea of engineering as a human-centered design. They concluded that students understood the needs of the end-user and were able to integrate those needs into their designs. Although the sample was selected for diversity, the role of this diversity in shaping perceptions was not an interpretive focus.

Considering the broader potential of EPICS and similar instructional modalities for changing the engineering pipeline at the high school level, the evidence base must include a systematic assessment of how these students might be differently impacted; many of the available studies are preliminary in nature. Based on the demographics of students participating in 50 schools in three states, Oakes et al. (2012) [28] concluded relative participation by girls (44%) and URM students and a high percentage of students eligible for free or reduced school lunches (46%) were markers of success. In addition, of the students who said at the beginning of the program they would “not at all” be interested in a STEM major, 53% of girls and 47% of boys had subsequently changed their response to “a lot”.

To our knowledge, however, there are currently no detailed studies focused on how EPICS High (or similarly designed project- and service-learning curricula) might differently impact women and minorities compared to other students—i.e., those that the fields of Engineering and STEM more generally are seeking to advance into careers. Our goal in this study was to do just that, as a first step in establishing a solid evidence base for identifying which strategies should best help to meet the much larger goal of supporting and diversifying the student pipeline in Engineering and related STEM fields. Our research question is: How do high school students’ engineering mindsets and their views of engineering/engineers change as they participate in project-service learning (as implemented through an EPICS High curriculum)?

To answer this, we used pre- and post-test data to identify if the degree of change after program completion was predicted by low beginning scores. Since the theory behind EPICS High curricula is of value to enhancing a diverse STEM pipeline because project–service-based learning changes students perceptions, our research strategy engaged in collecting and analyzing qualitative data on student perceptions alongside quantitative ratings of pre- and post-test mindsets.

2. Materials and Methods

Arizona State University (ASU) began delivering the EPICS college program in 2009 and then collaborated to deliver an EPICS High curriculum with local high schools that could act as potential pipelines to their Engineering degrees. Now the ASU program serves as one of three EPICS High hubs, meaning that EPICS High schools in the large metropolitan area work directly with our university program, whereas other EPICS High programs work with Purdue University. As the largest hub, the ASU program currently serves just over 800 high school students within 32 schools in the Phoenix Metro area, and the EPICS programming specifically makes it a point to partner with more diverse student bodies. For example, out of the 32 schools, 13 are classified as serving low-income, high ethnic-minority schools. The core curriculum mirrors Purdue’s EPICS High (curriculum can be found at https://engineering.purdue.edu/EPICS/k12), defined by student engagement in the design process (Figure 1). EPICS High at ASU is a highly coordinated endeavor with dedicated staff to implement the program and support the high school teachers throughout the year. Teachers are given access to the curriculum via Purdue University’s website portal, provided a week-long summer training, have dedicated college student mentors who visit the classrooms regularly, are offered a funding competition to support project development throughout the school year, and host a showcase as a culminating experience at the end of the spring semester. The EPICS High model is integrated into existing classroom frameworks, either through their STEM or career and technical education (CTE) classes or in afterschool clubs. The curriculum is grounded in design education and service-learning pedagogies and seeks to promote engineering as a force for social good.

By pairing meaningful community service with engineering instruction, EPICS High seeks to provide a conduit for students to engage in project-based learning to master course content while fostering greater civic responsibility and community engagement. Moreover, the curriculum incorporates human-centered design—focusing on the needs and uses for the end-stakeholder—and key engineering processes to foster engineering habits of mind, such as systems thinking, optimism, and ethical consideration in engineering, as well as entrepreneurial mindsets such as the three Cs (curiosity, connections, and creation of value) from the KEEN Framework [29]. Across EPICS High programming, students continuously explore, at increasing levels of sophistication, solutions to problems identified by their community partners by applying skills they are learning in the classroom. Ultimately, students work with members of the community to create engineering solutions to address real-world problems (see

Table 1. Sample selection of student team projects. STEM: Science, Technology, Engineering, and Math.

Community Partner	Project Goals
Audubon Society	Redesign and renovate the seating and shading structure of the Butterfly Garden, a space used to host community environmental events and school field trips
County Animal Care and Control	Create a durable, hygienic, and inexpensive dog bed for animal shelters
Student’s Own High School	Rebuild their school hypnotherapy garden to make it more accessible for those with physical disabilities
Local STEM Outreach Program	Teach children in foster homes about STEM to create better opportunities
Neighboring Elementary School	Create an app that is tailored to the curriculum to enhance student reading skills
Horse Rescue	Create a water catchment system to avoid stalls becoming flooded when it rains
Family-centered not-for-profit organization	Improve soundproofing of the community room in a low-income family housing complex
Students’ Own High School Community	Build a community garden to provide fresh fruits and vegetables to members of the community (the high school is located in a food desert)
Homeless Youth Organization	Assess the location most in need of help for the organization to expand
Housing/Health/Community Service Organization	Design a robotics curriculum for a summer program for homeless children aged 5 to 12

for selection of projects).

Data for this study were collected during the 2016–2017 and the 2017–2018 school years, covering two separate sequential implementations of the EPICS High program. Human subjects’ approval for this study was acquired through the ASU Institutional Review Board (STUDY00004523), and each school also provided individual administrative approval.

2.1. Scalar Data Collection and Analysis

We began data collection with an online survey targeting all participating students, deployed at the beginning of each school year. Surveys (N = 838) were completed by high school students from 15 schools (Wave 1 N = 361, Wave 2 N = 344). The surveys included a set of items for pre-testing of outcomes (see “scale construction” below). The second wave survey also presented open-ended narrative responses to questions to be assessed through systematic qualitative analysis. These questions included their goals after high school and their views of engineering, pre- and post-test. Post-test surveys were given at the end of each spring semester. We excluded from post-tests any students who proved from the pre-test to be outside of the grade range for the EPICS High program (6th and 7th graders, N = 10), as well as all students from the one high school that failed to implement the EPICS program in full during the semester (N = 121 students). After these exclusions, we had a total of N = 705 unique students across both waves. Unforeseen issues with the unique identifiers (required for ethics protection), and follow-up response rates, resulted in additional students being excluded from the post-study pool. This resulted in a final matched pre/post-test sample of 259 (

Table 2. Participant sample size and demographics.

	Pre-test (N = 578)		Post-test (N = 386)		Matched (N = 259)
Grade	N	%	N	%	N	%
8th	35	6.1	15	3.9	9	3.5
9th	44	7.6	39	10.1	26	10
10th	184	31.8	110	28.5	73	28.2
11th	205	35.5	122	31.6	89	34.4
12th	94	16.3	90	23.3	62	23.9
Missing	16	2.8	10	2.6	0	0
Gender
Male	312	54	218	56.5	142	54.8
Female	225	38.9	143	37	110	42.5
Prefer not to respond	21	3.6	15	3.9	7	2.7
Missing	20	3.5	10	2.6	0	0
Ethnicity
White	226	39.1	144	37.3	96	37.1
Latino(a)	200	34.6	137	35.5	98	37.8
Asian	48	8.3	45	11.7	34	13.1
URM (Latinx, African American)	43	7.4	23	6.0	17	6.6
Missing/Refused	61	10.6	37	9.6	14	5.4
Parent College Graduate
None	345	59.7	196	50.8	126	48.6
One Parent	118	20.4	92	23.8	60	23.2
Both Parents	115	19.9	98	25.4	73	28.2
Parents Engineer
Parent an engineer	94	16.3	65	16.8	50	19.3
Title I
Non-Title I School	217	37.5	170	44	134	51.7
Title I School	361	62.5	216	56	125	48.3
Total	578	100	386	100	259	100

). Students self-identified ethnicity and gender as part of the survey demographics.

The surveys contained 23 items for pre/post-test using scale items directly related to the learning outcomes for EPICS High (

Table 3. Scale reliability and the original contributing scale items.

	Pre-test Scale Reliability		Post-test Scale Reliability
	N	Alpha	N	Alpha
Improve Ideas	562	0.79	375	0.83
Importance of Feedback	563	0.83	370	0.86
Growth Mindset	557	0.73	372	0.71
Social Responsibility	560	0.82	371	0.85
Importance of Multiple Perspectives	562	0.79	372	0.82
Attitudes towards Engineering	559	0.72	375	0.63

) and adapted from scales developed to assess the KEEN Framework’s 3Cs [30,31]. These items were selected to assess status (and thus growth) in the following domains: Attitudes towards engineering (learning about engineering, considering studying engineering, understanding the importance of engineering); improving ideas (inventing new ways of doing things); importance of feedback (identifying needs of stakeholders, seeking input, incorporating feedback into designs); growth mindset (seeing obstacles as opportunities, not giving up on difficult tasks, seeing failure as a chance to improve); social responsibility (contributing to the good of society, seeking opportunities to improve lives of others); and importance of multi-perspectives (putting self in other’s shoes, incorporating different expertise/ideas). The item responses were on 5-point Likert scale ranging from strongly agree (5) to strongly disagree (1). These scales demonstrated a high reliability across both the pre- and post-test (Table 3). Students overall scored highly on all scale items, resulting in a left-skewed distribution for all scales at both pre-test and post-test.

Given we were only able to match 259 students across pre-test and post-test surveys, we performed more liberal statistical analyses on the scalar responses for the full analytic sample, comparing all pre-test scores with all post-test scores using the Mann–Whitney U test for binary groups and the Kruskal–Wallis test for tests among multiple groups. We then focused a second set of more conservative analyses on the matched sample and tests for growth within students from pre-test to post-test for those surveyed twice using the Wilcoxon signed-test.

2.2. Qualitative Data Collection and Analysis

Qualitative data collection, in the form of open-ended questions to students, was applied via two different methods. First, in the wave 2 (pre- and post-test) surveys, we added open-ended elicitations, asking students to reflect on the program. Examples of the questions were: “What are your views of engineers?” and “What are your plans after high school?” Second, over the two years of the study, we conducted extended participant observation—overseen by a PhD anthropologist—across the 13 schools. The ethnographic procedures included regular site visits and extended note-taking on informal interviews with students and teachers at least at two points in time for each school, once in the beginning of fall semester and again towards the end of the program in the spring semester. Since our research questions focused on the experiences of students, more detailed ethnographic attention was given to six schools designated as Title I as well as women and ethnic minorities across school types. The Title I label signifies that they serve low-income students, and these schools typically have a large number of ethnic minority students. For example, in Arizona, 41.5% of students in Title I schools are White, and Latinos represent 43% of the school population [32]. The site visits included classroom observations, documenting the forms and levels of student engagement with the curriculum, and talking informally with as many students as possible as many times as possible about their curricular experiences and evolving community projects. These informal interviews are unstructured and conversational in form, and the procedure is for the interviewer to take copious notes during the conversation and/or immediately afterward, including quotes [33]. The trained ethnographic researcher conducting these informal interviews was female, non-White, and relatively young; it was hoped this would support greater disclosure by the students. The site visits also involved taking copious field notes on the following: Student motivations for joining EPICS High; motivations for enrolling in engineering; students’ experiences working and designing for a stakeholder; and student team dynamics, among other topics. In these interactions, some students offered direct or indirect information on their ethnicity, but the researcher did not directly ask. Findings from the data set should be assessed with this in mind, and this also explains why we more simply coded students generally as White or non-White solely as indicative and did not take a further step of conducting URM versus other student comparisons.

Using systematic methods of qualitative data coding and analysis [34], we analyzed the resulting detailed field notes in addition to the narrative responses (Section 2.2, N = 215) of pre/post-test surveys. This process yielded a total of 60,324 words of text. Then again, using normal procedures for code generation and assignment, we developed a codebook and used deductive coding techniques to identify theme repetition within the body of text. Literature on the benefits of K-12 STEM-based project-based curricula, and the literature on developing engineering mindsets, were used as the general guide for what to look for first in the process of code identification [34]. An example of a code is: “Failure and Learning,” described as: “Students discuss the ways in which project failures, shortcomings, and mistakes, small and large, impact their learning experiences throughout the class.” Once we coded all the text, we were able to retrieve the coded segments to verify we indeed had repeating themes and were then able to make generalizations about the student learning outcomes. Then, in order to illustrate these analytical findings below, we identified specific exemplars (i.e., coded sections of text that clearly met inclusion criteria for that specific code or code set, and thus exemplified the theme that was identified) [33].

3. Results

3.1. Survey Pre- and Post-Test Scale Results

The scale means pre- and post-test for the full sample are presented in Table 3. The results of the Mann–Whitney U test show that scores at post-test are significantly statistically higher than scores at the pre-test, for all scales (Figure 2 and Figure 3): Improvement of ideas, importance of feedback, growth mindset, social responsibility, importance of multiple perspectives, and attitudes toward engineers/engineering. Assessing the same set of pre-test and post-test scores in the matched sample showed improvement in scores; however, none of the observed differences were statistically significant.

Comparing gender differences in all completed pre-test surveys (N = 578) showed female students had significantly lower scores on attitudes toward engineering, improving ideas, importance of feedback, and the importance of multiple perspectives than their male counterparts. However, in all completed post-test surveys (N = 386), only differences in attitudes toward engineering remained statistically significant between males and females. In the matched sample analysis (N = 259), female students saw significant growth between pre-test scores and post-test scores in the attitudes towards engineering and the importance of multiple perspectives. Additionally, female students had significantly more growth in the importance of feedback compared to males, i.e., the difference between pre- and post-test scores was significantly larger for female students.

In the completed pre-test sample, Title I schools had significantly lower scores on the importance of feedback and the importance of multiple perspectives scales in the full sample. While the scores on these scales increased for both Title I and non-Title I schools, the differences remained significant when assessed in the full post-test sample. Analysis of the matched sample showed none of the differences between pre- and post-test were significant for either Title I or non-Title I schools.

In the full pre-test sample (N = 578), the nonparametric ANOVA showed significant differences in the importance of feedback and the importance of multiple perspectives across different ethnicities. Primarily, those self-identified as Asian scored the highest on the importance of feedback and multiple perspectives. Furthermore, these were significantly greater than those in the “other” category who self-identified as African-American, Native American or Pacific Islander, or were missing data on ethnic background. These other and missing categories had the lowest scores on these two scales. Differences in the importance of feedback and multiple perspectives were also observed in the full post-test sample (N = 386) and were again driven by the low scores of students in the “other” or missing ethnicity category. In addition, differences emerged at post-test for three of the other scale items: Attitudes toward engineering, improving ideas, and social responsibility. Again, the relatively low scores drove these significant findings for those students in the other or missing ethnicity category. The matched sample (N = 259) suggests that these differences were driven primarily by growth among Asian students in their attitudes towards engineering, improving ideas, and social responsibility. These were the only scales to show significant growth, and only among self-identified Asian students.

In the full pre-test sample (N = 578), results showed significant differences in the scores for attitudes toward engineering, importance of feedback, and the importance of multiple perspectives across varying levels of parental education. Attitudes toward engineering were similar among students with parents having no college degree, or with one parent having a college degree. However, the differences in the importance of feedback and multiple perspectives were driven by students with parents who did not have a college degree and students with at least one parent who had a college degree.

In the completed post-test sample (N = 386), we saw a reversal in the attitudes toward engineering. Students with no college-educated parents had the highest scores on attitudes toward engineering, compared to students with at least one college-educated parent. Additionally, the differences in the importance of feedback remained significant, with students without college-educated parents showing the lowest scores. However, the matched sample analysis (N = 259) shows none of the between or within group growth was statistically significant.

3.2. Student Open-Ended Responses, Pre- versus Post-Test

Using the two systematically coded student survey narrative questions for the matched sample for Year 2 (N = 215) (“What are your views of engineers?” and “What are your plans after high school?”) provided a different form of pre/post-test analysis.

Based on coded responses to open-ended questions, the majority of students (79.1%) who entered the EPICS High program expressed a favorable view of engineers at the start of the program (see

Table 4. Coding of responses on student views of engineers.

	Positive (%)	Negative (%)	Unknown/Neutral (%)	TOTAL (%)
Pre-test	170 (79.1)	3 (1.4)	42 (19.5)	215 (100)
Post-test	184 (86.4)	1 (0.5)	28 (13.1)	213 (100)

). Only three students (1.4%) had a negative view of engineers, and the remaining students (19.5%) had a neutral view of engineers or did not comment. After an academic year of participation in the EPICS High program, there was a decrease in students who had previously provided a negative or neutral view of engineers and engineering (0.5% and 13.1%, respectively) and an increase in the number of students who had a positive view of engineers and engineering (84.4%) (noting the actual numbers of students coded differently between pre/post-test are few in number). Here are two exemplar quotes of students’ pre- and post-test answers regarding their views on engineering.

Female, Latina, 12th Grade at non-Title I School:

Pre-test answer: “Boring.”

Post-test answer: “I think they are amazing at what they do and make it interesting.”

Female, Asian, 11th Grade at non-Title I School:

Pre-test answer: “I don’t really know much about engineers but I am interested in exploring.”

Post-test answer: “What I’ve learned is that engineers are those that work on certain projects to better the economy. There are engineers who build things and engineers who work with other things such as computers and technology. There are many types of engineers. There are civil and there are aerospace [engineers]. Personally, I’m not sure if I want to be an engineer, but I am trying to explore the field of engineering and what I am most interested in is aerospace engineering. The whole concept of engineering intrigues me and I might consider going into engineering in college.”

Based on codes applied to the student narrative responses to open-ended survey questions, most of the students who entered the EPICS High program indicated at pre-test that they planned to attend college after they graduate high school (see

Table 5. Plans after high school *.

	College	Military	Unsure
Pre-test	175	9	12
Post-test	175	14	15

Note: * Responses could be coded more than once if more than one answer was provided.

). For example, 175 students (80% of the matched responses) indicated they were planning to go to college after high school. This number remained unchanged in the post-test. This confirms that high school students who enter the EPICS High program already want to go to college prior to entering the program. However, there is evidence that their goals become more refined after finishing the academic year, as demonstrated by this exemplar quote.

Male, Latino, 12th Grade, Title I School:

Pre-test answer: “I want to pursue a career in some form of engineering while learning about entrepreneurship on my own or through a mentor. I plan to obtain a four-year degree in some form of engineering and later work as an engineer while building up my very own business.”

Post-test answer: “After high school, I want to attend the Honors College at ASU and major in Computer Systems Engineering. At the same time, I also want to be involved in my community, working alongside my peers to bring solutions to problems.”

3.3. Participant Observation/Interview Data Analysis

Based on inductive systematic coding of the qualitative data, three salient themes, with corresponding subthemes, emerged from the data analysis: (1) Increase in engineering self-efficacy and skills; (2) community embeddedness; and (3) increase in resiliency and positive relationship with failure. Below, we illustrate these findings using exemplar quotes from students.

Theme 1.

Increase in Engineering Self-Efficacy and Skills.

Subtheme 1a: Confidence and self-efficacy. Throughout site visits, students shared the ways in which participating in EPICS High helped to expand their self-efficacy in engineering. Students discussed how program participation allowed them to view themselves as engineers and gain confidence in their engineering skills. To illustrate, when asked what they gained from participating in EPICS High, this student stated:

“Getting a chance to learn engineering, seeing it’s not as hard as I thought. I can see myself possibly doing engineering. I initially thought you had to be super smart.”

(female, non-White, non-Title I)

From the notated conversations with students, we learned that program experiences served to both demystify engineering and help students to make connections to engineering, especially for students who may have come into the program indifferent or uninterested in engineering.

Subtheme 1b: Skills and real-world engineering experience. Students spoke of how the class enabled them to gain real-world experience and witness how engineering can be used beyond the classroom. Students spoke of how they liked that the class was hands-on, and that they not only are learning but are applying what they are learning as they engage in the EPICS High design process. They often cited how EPICS High serves as an opportunity for them to learn to “think like engineers” and work to implement solutions. Students expressed that working on their EPICS projects helped to increase their critical thinking and problem-solving skills, while fostering creativity.

“[EPICS] gives you a little taste of what engineering is like. It’s a lot of thinking outside the box and handling different problems based on the scenario—it’s really cool.”

(male, non-White, Title I)

“EPICS forces you to work on a project that doesn’t have a right or wrong answer, leaves more room to be creative.”

(male, White, non-Title I)

Additionally, students often spoke of how they had to learn to construct a project budget and plan, conduct research, use Gantt charts to track task assignments, and increase their technology literacy to learn new software for their projects.

Subtheme 1c: Collaboration and communication skills. While learning how to do their projects, students were able to increase their teamwork and communication skills. Through conversations with students, we learned that project experiences provided opportunities for them to leverage the synergy of different skills, personalities, and competing goals to benefit from intragroup collaboration and creativity.

“I learned in engineering that it’s hard to work on a project by yourself, you need others to help. This is good—with other people, you get new ideas, and this helps change your project for the better.“

(male, non-White, Title I)

“It didn’t start off well with my group in the beginning. There were two female strong leaders in our group. Eventually, I gained a new friend. I learned not just about her and about the group, I learned to mature and let everyone contribute.”

(female, non-White, Title I)

“It [EPICS] taught me things I will need to know in the real world. It teaches you how to work in a team and to step outside of your comfort zone. You have to work with someone else [the stakeholder] who agrees with what we said.”

(female, non-White, non-Title I)

These exemplar quotes highlight that EPICS High provides a space for students to improve their peer-to-peer collaboration skills as well as communicating with community partners, which also fosters unique student gains.

Theme 2.

Community Embeddedness and Civic Engagement.

Throughout the program, EPICS High asks its students to continually explore problems in the community that can be solved by the skills they are learning in the classroom and identify those in their community that would benefit from an EPICS project. During site visits, students expressed that they liked that their EPICS projects connected them to a community or issue and cited how the class provided opportunities to not only gain experience working with a client, but exposure to skills and values necessary to design with someone else in mind.

Subtheme 2a: Empathy. The curriculum focuses on human-centered design as a core principle. As students spoke of their experiences, many cited that their projects enabled them to foster greater empathy to better understand the viewpoints, social conditions, and needs of their stakeholders and community partners. Through experiences working on a long-term project with a client, students cited the value of empathy for both their persistence to finish and effectiveness of their projects as well as reflecting on their own learning.

“I think our stakeholder is doing more for us. They’re helping us have an understanding of their everyday life. It helps us learn more about people different from ourselves… I see how my skills can help, but also how they in turn help us as well.”

(female, White, non-Title I)

Moreover, students expressed that their projects provided value to themselves, stating that their projects were “more than just a grade” for them. Throughout the informal interviews, students expressed that project experiences provided the opportunity to create something of value for a client.

“EPICS wasn’t what I expected. There’s so much more of an adjustment period- learning to do projects that aren’t just for myself but for the betterment of the community.”

(male, White, non-Title I)

“In EPICS you have to live up to a nonprofit and not a grade. It is more fun and you hold yourself more accountable because you have to help people.”

(female, non-White, non-Title I)

For many students, working with a client was a new experience and a paradigm shift from traditional STEM projects that may be abstract and hypothetical. Through doing engineering and working with an actual client, students learned to foster empathy as an engineer and learned to position the user at the center of the design solution.

Subtheme 2b: Impact in the community. Site visits also served as an opportunity to learn the ways in which EPICS High serves as a conduit for students to learn more about engineering (and STEM) through service. Throughout the informal interviews, students often discussed the benefits of having a direct impact on the community. In fact, an overwhelming number of students expressed that they were drawn to the program for the opportunity to impact and help the community.

“As a minor [under 18 years of age], it’s hard to impact the community. In STEM, you can have an impact on the community.”

(female, White, non-Title I)

“EPICS is to serve the community—all of our projects are school-based. The idea of the class is to give and help the community—this was a big push for a lot of us I feel, doing something positive and being a role model to younger students.”

(male, non-White, Title I)

During discussions with students, we found that they were able to articulate with whom they are working, but also how their stakeholders would benefit from their projects. From the site visits, we see that students’ experiences working on projects helped them to increase their awareness that they can play a part in fixing problems in their community as the desire to impact the community taps into students’ intrinsic motivation. Ultimately, after working on their projects, students stated how they are able to see how engineering can be used to help the community.

Subtheme 2c: Personal connectedness to projects. EPICS High provided opportunities for students to not only impact their communities but make individual connections to their projects as they impact their communities. Moreover, students expressed that by working with and designing solutions for members of their local community, they were able to identify and connect the ways in which their skills and interests can be used to creatively solve problems.

“…I’ve always been passionate about the environment. In class, I was able to tie engineering and combine two of my passions. In engineering, there’s not just one job, you can do many things and tie it to your passions—it’s one of the great things about engineering.”

(female, White, non-Title I)

For many, the EPICS High service component connected students to a problem, need, or interest they were passionate about and enjoyed. This served as an additional source of motivation to adhere to and complete project deadlines when teams experienced unintended setbacks.

“Other projects you do for other classes aren’t personal—you’re just following instructions to get things done; with this class, it’s more personal.”

(female, non-White, Title I)

Additionally, students discussed that while working on their projects, tasks were assigned according to each person’s skills, experience, strengths or interests, further enabling students to make personal connections to their projects. Doing a project for an individual purpose helped students to be empowered in their learning and to take more ownership of their learning.

Theme 3.

Increase in Resiliency and Positive Relationship with Failure.

In the EPICS High curriculum, students are encouraged to approach the design process with a mindset that is open to failure, ambiguity, and feedback.

Subtheme 3a: Importance of feedback. An intended outcome of EPICS High is to impart in students a particular philosophy of engineering, which is to create human-centered solutions that provide value for real people, and this includes asking for feedback from their stakeholders. Many students highlighted that gaining experience working with and getting feedback from a client as not only a major pull to participate in the program, but a major takeaway from the program as well.

“…getting to see what an engineer actually does. Our teacher has made a point that we are engineers and we’re getting feedback and criticism like engineers do—we’re getting a lot of real- world experience. How else will I get that kind of experience?”

(female, non-White, Title I)

“[EPICS High] prepares us for real human interactions between us and our partners. It’s our responsibility [to meet their needs].”

(female, White, non-Title I)

Additionally, students shared experiences learning how to compose and execute communications with their client and incorporate stakeholder requests and feedback.

Subtheme 3b: Overcoming challenges. Throughout the conversations with students, they expressed that they learned that failure is a critical part of engineering.

“I was raised to be an honors student… I always heard that failure was okay from music students—I didn’t accept this, and it took me a while to accept the concept of failure… it’s a matter of decoding yourself because it challenges how you’re traditionally taught.”

(female, non-White, Title I)

Students often had to pivot, redesign, iterate or restart a project to ensure their solutions were both feasible and of value to their stakeholders. Many students expressed that EPICS High provided a space for them to comfortably fail in a safe environment and gain confidence in their mistakes to increase their learning potential.

“In other classes, failure is a big deal and you only get one chance and it’s more punishing to make those mistakes. You do something wrong and get a bad grade and no opportunity to do it better. It feels like you can’t make mistakes in other spaces whereas in engineering, failure is seen as progress.”

(male, non-White, non-Title I)

“The class is a life teaching class. It gives you the opportunity to have trial and error—not a lot of courses provide this.”

(male, White, non-Title I)

Furthermore, students often expressed that the opportunity to positively impact their communities kept them motivated to persist with their projects, especially when they encountered setbacks or design difficulties. These examples demonstrate how the program provides opportunities for students to learn to embrace their mistakes, evolve when necessary, and foster resilience to increase their learning potential.

4. Discussion

Prior research on problem-based and service-learning programs has also concluded that they promote valuable development of skills and knowledge at the college level [7,11,12,13,14,15]. These findings are echoed with students from the EPICS High program we tested here, and our results bolster prior smaller studies on EPICS High in particular [16,26]. We found that students who enrolled in EPICS High over a one-year period showed increasingly high positive views of engineers and were more likely to recognize the importance of feedback, multiple perspectives, and social responsibility and were more likely to see themselves as resilient to challenges and improving on existing ideas. Narrative pre- versus post-test responses also showed that they learned more about engineering and honed their career goals during the program.

Furthermore, our analysis by student and school characteristics identified who best benefitted from program participation. First, attitudes toward engineering increased for all students, but significantly for female and Asian students. Second, the importance of feedback and the importance of multiple perspectives were significantly different at baseline across gender, Title I schools, self-identified ethnicity, and across students with different levels of parental education. Female students closed the gap at post-test for these scales; however, differences remained for Title I schools and ethnicity. Finally, at post-test, we observed no differences in the importance of feedback across students with different levels of parental education, though the importance of multiple perspectives still showed significantly higher scores among students with both parents having a college degree.

Scalar data also showed that students who participate in EPICS High overall scored high on engineering-related mindsets and attitudes at the outset of programming, suggesting that these students already had a great interest and regard for engineering upon starting EPICS High programs. The ethnographic/qualitative analysis based on site visits and informal interviews provides a crucial additional set of evidence for interpreting the scalar pre/post-test findings, most especially because—while statistically significant student gains were observed—it was also hard to interpret the impact of change since starting values were surprisingly high. Qualitatively, participants increased their confidence as engineers, learned about engineering processes, used and further developed their problem-solving and creative abilities, increased their collaboration and communication skills, and learned to embrace failure as positive and how to persist when challenges arose. For instance, through gaining exposure to the culture of engineering (e.g., failing, rapid prototyping), students said they are able to decrease their aversion to failure and learn to turn it into opportunity. Furthermore, students recognized gains in understanding the applicability of engineering design to solve problems in their community as well as became civically engaged. These results suggest that EPICS High offers meaningful improvements in student outcomes that should help to meet the larger goals of increasing diversity in engineering/STEM. Thus, quantitatively, the student scores suggest EPICS High participation is not creating much change in the skills domains that could support any students’, including URM and female students’, pathways into Engineering/STEM. However, the systematic analysis of two years of ethnographic (participant observation and interview) data provides a very different set of insights in this regard and suggests the programs are positively impacting URM and female students in particular, and in ways that are meaningful and could potentially orient them toward STEM.

Based on this, we concluded that we are able to identify small but highly personally meaningful shifts in how students align engineering careers with their own diverse backgrounds, particularly the unique needs of the communities they come from and/or wish to serve. Students also expressed a sense of greater resiliency to challenges; together, these two aspects of the experience were highly meaningful to them in ways that could be predicted as important to the likelihood they will persevere in the engineering/STEM tracks even if they encounter barriers. In this way, there are grounds to conclude the EPICS High programs are serving diverse students well, most especially when project-based and experiential learning focuses them on considering and engaging the needs of local communities. This finding—of subtle but important shifts—also highlights the importance of tracking program impacts not just through scalar changes on attitudes and skills, but also using qualitative/ethnographic approaches to identify changes in what students give meaning to as they consider STEM/engineering careers.

We have several suggestions for the next steps to identify the strengths and weaknesses and track the impacts of project-based learning programs like EPICS High for meeting the goal of orienting more students toward and enhancing the diversity of STEM/engineering degree and career pathways. Given that diverse students came to these programs already positively viewing engineers/engineering, it may be better to focus on early educational (e.g., mathematics) interventions and training [17] and/or having much earlier exposure to STEM problem-based learning to show its applicability as a career [18], such as in middle schools.

There are several limitations of the study we highlight. While the liberal comparison of the total completed sample found more significant differences than the conservative matched samples tests, few of the results were replicated in the matched pre/post-test sample. This could be a power issue, given the low sample size. Power analysis indicates that in order to have 80% power to detect an effect close to those observed in the full sample tests (small to medium effect size: D = 0.1–0.3) at alpha = 0.05, we would need between 90 to 786 matched pairs. While the power analysis indicates we have a sufficient sample size to detect medium effects (~0.3), given the high scores on the scales at pre-test, we would expect smaller effect sizes (i.e., there may be insufficient room for increases to be observed). On a related point, the scales failed to capture optimum levels of variation in response, that is, to avoid further ceiling effects. Results may have been easier to interpret if Likert response scales had been 7-point rather than 5-point. Additionally, more orthogonal scale items could have helped to reduce correlations between scales, helping to identify more concretely the domains in which students are thinking about engineering and engaging with the program content. More generally, all the pre/post-test scale items are self-reported attitudinal measures. The gold standard would be to validate the scales with behavioral measures, such as pre- and post-test engineering problem-solving assessments that can measure the application of curricular elements that students learn [35]. Further, the study as designed did not include a comparison group of students in the same schools but not enrolled in EPICS. This would have helped to identify more clearly any self-selection into the EPICS High program by students that already hold more positive views of STEM. It could have also helped us to better understand differences between schools and student demographic groups in both baseline and post-test scores.