A Systematic Multiple Studies Review of Low-Income, First-Generation, and Underrepresented, STEM-Degree Support Programs: Emerging Evidence-Based Models and Recommendations

Abstract

The dramatic lack of diversity within science, technology, engineering, and mathematics (STEM) higher-education graduates is a serious issue facing the future of the STEM workforce. Colleges and universities are attempting to fix this disparity through targeted intervention programs aimed at increasing the persistence and retention of low-income, first-generation, and underrepresented (UR) students in STEM. A comprehensive synthesis of adopted strategies and analysis of their effectiveness is lacking. We conducted a systematic multiple-studies review, considering 31 empirical articles published between 2005 and 2020 that shared the outcomes of intentional intervention programs in place across the United States. We uncovered essential characteristics of successful programs and highlighted the top 10 critical program components to consider. These can be used to guide the development and execution of future programs’ support systems and interventions. This is a relatively new area of research, with most programs just in the earliest stages of implementation. It is essential to continue to follow the data on the outcomes of these programs, particularly their longitudinal impacts on the diversification of the STEM workforce, to be able to implement effective evidence-based practices.

1. Introduction

For decades, there has been a focus on increasing the number of successful higher-ed graduates in science, technology, engineering, and mathematics (STEM) fields [1]. Those efforts have been largely successful [2]. Between 2008 and 2018, there was a 55.3% increase in the number of post-secondary degrees awarded to students in STEM fields [3]. Since 2001, the number of graduates with STEM degrees has more than doubled [3]. Continuing to grow these numbers is essential to meeting the needs of the workplace of the future, as there is ample evidence that our current supply of STEM workers in the U.S. is not sufficient to meet the increasing demand [4].

It is also clear that the future of STEM will require not just numbers, but the diversity of culture, background, and expertise [5]. Historically, the United States has relied heavily on immigration as a source of qualified contributors to STEM, but that is unlikely to be an acceptable solution in the long term [4]. Therefore, the need to develop and nurture a diverse pool of STEM-focused academic talent within the U.S. is a necessity.

Critical to growing the number of skilled STEM graduates in the coming years is creating more opportunities for low-income (LI) students, those from families earning less than $25,000 annually, and first-generation (FG) students, those whose parents did not attend college. While the percentage of U.S. adults who hold bachelor’s degrees has increased by over ten percent from 1990 to 2015, the percentage of FG students has declined by four percent [6]. Furthermore, students who are considered both low-income and FG are up to four times more likely to leave higher education after their first year than peers without these risk factors [7].

Of the over 730,000 students to graduate with a degree in STEM in 2018, only 25.9% were underrepresented racial minorities (Blacks, Hispanics, Alaskan/American Indian, and two or more races) [3]. Based on the U.S. Census estimates, in 2018, racial and ethnic minorities made up 35.9% of the general population, yet those numbers are increasing rapidly. It is likely that by 2065, no single ethnic or racial group will be a majority of the population [4]. Furthermore, women, who comprise 32.4% of the STEM graduates in 2018 [3], make up 50.8% of the U.S. population. Clearly, these numbers are not aligned. As stated in a report from the Committee on Underrepresented Groups and the Expansion of the Science and Engineering Workforce Pipeline [8], “Those groups that are most underrepresented in science and engineering (S&E) are also the fastest growing in the general population” (p. 3).

1.1. Background

The lack of diversity among STEM higher-education graduates has not gone unnoticed.

Across the U.S., colleges and universities are attempting to fix this disparity by addressing many of the factors that have been shown to impact the persistence and retention of students in STEM majors through targeted STEM-intervention programs. Most of these programs have been put into place within the past 10 years or so; therefore, little has been written about their outcomes. There is only one program we found that has any longevity, the Meyerhoff Scholars Program at the University of Maryland Baltimore County [9,10,11]. Since 1989 and for the past 30 years, the Meyerhoff program has recruited and provided targeted support across thirteen areas to students of color, originally African-American students only, with an interest in a science major at UMBC.

Beyond the MSP, there is a dearth of published information about the impacts other programs of this type have made on the persistence of underrepresented (UR) students in STEM higher education. Many of the articles reviewed for this paper called for additional research to be conducted in this area. Overwhelmingly, there is a need to better understand the complexities of program design and how changing program variables such as the types of support offered [11,12], mentor race, and gender [13], the academic balance of peers in a small-group learning environment [14], and how engineering student organizations acknowledge cultural and socioeconomic differences in members [15] can impact the outcomes of students. There is also a need to dig deeper into the most efficient and appropriate way to distribute funds for these intentional interventions and how outcomes are measured [9]. Finally, it is essential that further research focuses on what organizations need to better support UR students in STEM, as opposed to what the students themselves need to do to succeed [16].

There have been few prior literature reviews that have considered topics related to the retention of UR students in STEM. Understandably, STEM as an acronym was only coined by the National Science Foundation in 2001, although the sciences have been a national education focus since the 1950s [17]. However, there have been studies that examine tangential topics.

Underrepresentation in education has been a topic of interest for early researchers going back to the 1970s, and a number of literature reviews have been written on this subject [18,19,20]. Other literature reviews examined the role identity plays in mathematics education, an important factor in retention intervention programs [21,22]. Finally, another literature review looked specifically at the career experiences of women in STEM [23].

We found a literature review conducted by the National Academy of Sciences concerning intentional program components and organizational systems designs suggested to improve persistence and retention amongst UR students in STEM in Minority Serving Institutions (MSIs) [4]. The review included “three areas of focus: (1) STEM education for students of color across higher education (MSIs and non-MSIs), (2) student success at MSIs (STEM and non-STEM), and (3) student success in STEM at MSIs” and suggested intervention strategies, which would theoretically support persistence and retention.

1.2. Purpose

Our systematic multiple studies review (MSR) builds on the NASEM [4] study in a few ways. First, we only considered empirical research findings of studies that examined the effectiveness of specific intervention programs and strategies. In addition to empirical findings of retention and persistence outcomes, we took a more holistic approach to include identity beliefs or attitudes that exist among STEM programs and students. Our study includes not only people of color but also includes low-income students and women, both underrepresented in STEM, as well as first-generation (FG) college students. Finally, we also considered research stemming from a more diverse selection of higher education institutions, including but not limited to MSIs. With this study, we strive to make sense of the many variables these programs offer in a way that clarifies which elements are most successful across all institutions of higher education. We also seek to draw connections between the frameworks of these programs and the model of institutional intervention developed by Vincent Tinto [24].

In order to examine all angles and facets of different academic intervention programs striving to increase retention and persistence among UR students in STEM, while at the same time considering the alignment of the programs within an existing respected framework, we were guided by the following two research questions:

• What intentional intervention program components are used to increase retention and persistence among LI, FG, and UR students in STEM?
• What can we learn from the empirical outcomes of interventions, intentionally designed to support LI, FG, and UR students’ retention and persistence in STEM degree programs?

2. Materials and Methods

We used a systematic MSR, focused primarily on academic literature, to investigate the use and effectiveness of intentional interventions aimed at improving the retention and persistence of LI, FG, and UR students in STEM higher-education programs [25,26,27]. An MSR is a particularly appropriate process for this, work because it can be replicated and integrates all methods of research that are returned by the search: quantitative, qualitative, and mixed-methods. As explained by Nurius and Yeaton [28] and cited in Pluye et al. [27], “The richest and most reliable summarizations of ‘what we know’ in a given area can best be achieved through an alliance between qualitative and quantitative information and methods of investigation” (p. 707).

2.1. Search Scope and Protocol

We first identified the source, strategy, and focus of our systematic search, presented below.

Intervention: Support programs, interventions

Context: a higher-education academic environment (undergraduate and graduate), including community colleges and four-year colleges and universities

Population: LI, FG, and UR, students in STEM programs, courses, or majors

Empirical study designs: qualitative, quantitative descriptive, and mixed methods

2.2. Search Strategy

Our search process began by mining our research questions for relevant search terminology. Our original intent with this study was to primarily focus on engineering programs and students, but searches within Academic Search Premier indicated that only using that term would be too limiting; therefore, we widened the scope to include STEM. We used Boolean search connectors “AND” and “OR” to then narrow our findings to students of higher education. We tried various combinations of search terms to identify those articles specifically about intervention programs, and we determined that focusing on the goals of those programs, specifically, retention and persistence, netted the most accurate results. We did not need to include any statement qualifiers using the Boolean term NOT as our results were sufficiently precise without further clarification.

2.3. Search String

The primary search string used for this study began with terms intended to identify articles relevant to our population.

• (“low income” OR “first generation” OR “underrepresented” OR “minority”) AND (“engineer*” OR “STEM”).
  We added higher-education identifiers to remove the majority of results pertaining to K-12 education.
• AND (“higher ed*” OR “masters” OR “university”).
  Finally, we added terms to indicate programs that were attempting to increase the retention of students within these programs.
• AND (Retention OR Persistence).
  We restricted our search findings to peer-reviewed academic journals published between 2005 and March of 2020.

2.4. Resources Searched

Our search was conducted using EBSCOHOST, specifically within the Academic Search Premier database.

2.5. Abstract Selection Process

This study is focused on the elements and outcomes of intentional higher-education intervention programs to support either LI, FG, and/or UR students in STEM. Articles that described these interventions while also including the results of the interventions were our primary target. As such, we developed the following inclusion criteria.

• The study was empirical.
• The study focused on an intentional intervention program at an institute of higher education.
• The study participants included members of each of LI, FG, and/or UR groups, and the results for those students were specifically measured or assessed. This includes both women and minority groups as they are underrepresented in STEM.
• The study was in STEM-specific programs that included multiple related disciplines or undergraduate curricula that could lead to engineering disciplines in the future. Curricula that could lead to engineering disciplines would be those where students would have taken as pre-requisite to engineering courses.

2.6. Inclusion and Exclusion

Articles that were excluded discussed interventions at the K-12 level; outcomes, demographics, and metrics for LI, FG, and UR STEM students who were not part of an intervention program; intervention programs within the workplace; or intervention programs for LI, FG, and UR students of other disciplines and fields of study. We also excluded articles that discussed interventions that consisted solely of fundamental changes to the curriculum for included students, as that level of the program is much more difficult to replicate. Finally, because the goal is to apply the results of this study to broader applications of LI, FG, and UR STEM-retention programs, we excluded factors that narrowed the results to a very small subset of students, for example, students with hearing impairment and learning disabilities.

2.7. Resources Identified

There was a total of 124 initial results delivered from the search strings described above. These results were exported into EndNote Web. There were no duplicates among these 124 articles. The researchers reviewed each article for inclusion based on the criteria outlined above, which resulted in 32 total articles. An additional empirical article was discovered by reviewing the references of an original article, bringing the total to 33. This particular article was not peer-reviewed; therefore, it did not come up in our initial searches. However, this work provided important context and details for another study and, therefore. it was included. These 33 articles were then put through the quality appraisal process.

2.8. Quality Appraisal

Our results included empirical research that fell into one of three different categories: qualitative, mixed-methods, or quantitative observational. There were no articles that were strictly quantitative experimental, although some of the quantitative observational articles were quasi-experimental in nature. We used a quality appraisal process outlined by Hong et al. [25] that built off the prior work of Pluye et al. [27]. The tool is particularly relevant for the review of mixed-method and quantitative-observational studies, of which we had 25. Using the mixed-methods appraisal tool (MMAT), a score was generated for each article out of one. It was determined that articles scoring at least 0.8 would be kept. After a review of all 33 articles, 31 were kept for inclusion in the study. Two articles were eliminated due to flaws or a lack of a robust description of the research design.

Table 1. Articles Identified per Search Term.

Search	Number of Article Results
Underrepresented	31
STEM	31
Engineer	30
Minority	30
First-generation	16
Low income	6

below shows the population types noted in each of the 31 articles.

2.9. Coding Structure Development

We built upon previous research by Tinto [24] for the foundation of the coding we had used to analyze the empirical research studies on intentional interventions for UR students in STEM. Tinto’s [24] work considers broadly the obligation a college or university has to help students succeed. To this end, he outlines four conditions that, “… provide institutions not only with a systematic way of thinking about what actions they can take to increase retention and graduation but also with a way of organizing and implementing those actions to enhance their sum impact on student success” [24], p. 6.

These four conditions provided the top-down structure to our coding [29]. We also employed a bottom-up process by considering the approaches and outcomes shared in the articles we reviewed. With Tinto’s model and our research questions in mind, we followed a content-analysis methodology [30] of the 31 relevant articles that we collected. Using an iterative technique, we extracted and aligned program components, elements, details, and outcomes until a clear picture of the results emerged.

2.10. Codebook Development

Once the coding structure was identified, we developed a protocol for a thorough analysis of the 31 articles as outlined in Rourke and Anderson [30], which included: identifying the purpose of the coding, identifying textual elements that demonstrated each code, revisiting and refining the definitions of each code, testing out the methodology and definitions, and developing a graphical alignment of coding categories. The codebook was developed recursively, with adjustments made throughout the process. Our analysis used the following guidelines.

• The purpose of coding was to answer the research questions with evidence from the literature. Therefore, we did not separate out any subgroups in our coding and analysis.
• Tinto’s [24] conditions provided a basis for the top-down development of the codebook by establishing overarching program categories.
• In addition, a bottom-up approach was taken, based on an initial review of the 31 articles.
• Data that did not initially align with a particular code was tracked throughout the process. We reviewed this data frequently to determine if additional codes were needed or if the information represented an example of a subcategory of an existing code.

2.11. Main Coding

We used qualitative content analysis to code the 31 relevant articles returned through our search protocol [30]. Two distinct categories of articles emerged, those that focused on program outcomes and those that focused on program design. Those that focused on program outcomes noted basic intervention program statistics: duration and descriptive statistical results. The descriptive program results were divided into four subcategories: GPA, year over year retention, graduation rates, and increases in students’ personal attributes that indicate likely persistence. All program data was self-reported.

Five of the 31 articles exemplified a second category of article types; these took a snapshot view across multiple intervention programs and described the design of individual program components. As noted by several researchers, the drawn conclusions provide important insight into possible systems design considerations [31,32,33].

2.12. Theoretical Framework for Coding

Seminal work on student retention in higher education completed by Tinto [34,35] is the underpinning theory we used to frame the results of this study. In 1975, Tinto [35] introduced one of the first models that attempted to investigate why students drop out of college. Tinto has since gone on to modernize his framework by making it less focused on why students drop out, and instead more concerned with what institutions can do to help students persist and graduate [24]. Additionally, this newer framework specifically addresses the barriers experienced by FG and UR students as noted by Tate, Fouad, Marks, Young, Guzman, and Williams [36], something that had been missing from Tinto’s 1975 [35] model.

In this more recent action-oriented work, Tinto [24] presents four conditions that increase student success in higher education: expectations, support, assessment and feedback, and involvement. We depicted involvement and expectations as encompassing support and feedback for two reasons: (1) we assumed students must be involved in the program intervention for it to have any effect; and (2) their expectations will influence how they perceive the support and feedback they receive as well as any assessment they undertake.

These four conditions guided the analysis of the 31 empirical articles. Through that review, we identified a set of individual program component codes. Ultimately, we found 13 distinct program components that met one or more of Tinto’s [24] four conditions. Because each of the conditions is multifaceted and can be incorporated into an intervention program in many ways, there was considerable overlap. From that overlap and the results of our systematic multiple-studies review, we developed a LI, FG, and UR program support model that illustrates the relationships amongst the different variables, as shown in Figure 1.

The conditions of involvement, a student’s engagement with program activities, and expectations, a student’s belief in their ability to succeed, underlie every aspect of an intentional intervention program; therefore, these are both all-encompassing aspects of the diagram. The three sub-categories that comprise the support condition, academic, social, and financial, are broken out individually as there are program elements that only align with one or two of the three. For example, financial aid, an essential aspect of most intentional intervention programs included in the literature, cannot accurately be described as a form of academic or social support. However, mentoring can be a supportive element in both an academic and social sense. Finally, five of the 13 program components also align with the condition of assessment and feedback.

3. Results

This MSR found 31 articles pertaining to intentional interventions intended to increase persistence and retention amongst UR students in STEM. While we used 31 articles, there were 25 distinct intervention programs with their outcomes discussed. Of these 25 programs, two programs were focused on graduate students, while the other 23 were put in place for undergraduates. Thirteen programs of these 23 were specifically geared towards freshmen.

3.1. Question 1: What Intentional Intervention Program Components Are Used to Increase Retention and Persistence among UR and FG Students in STEM?

In all, these 25 programs indicate that there are 13 program elements commonly used as tools within the intentional interventions. Next, we describe each of the program components we identified and provide an analysis of the design of these components so that others can learn from what has been done and how it has been done. Finally, we align each program component with the relevant Tinto [24] condition to illustrate how it fits into the larger picture of intervention.

3.1.1. Recruiting and Admission

Most of the intentional interventions we looked at started with a recruitment process to select the right participants. Not only is recruitment an essential first step, but it is also the first challenge many programs faced. Ultimately, in most cases, recruitment and marketing worked hand in hand, using common approaches to make sure the availability of the program was widely known enough among prospective students to gain the desired number of applicants.

The recruiting and admissions process for any intentional intervention program designed to serve LI, FG, and/or UR students in STEM is also the first connection of the literature to Tinto’s [24] condition of expectations. By expressing interest in and applying to an intentional intervention program, a student is making a statement about what they expect for themselves and what steps they are willing to take to attempt to live up to those expectations. The 25 intervention programs reviewed in the literature all required some form of application or admissions component. This commonality bolsters Tinto’s [24] assertion that clear expectations are an important step an organization can take to support the persistence and retention of these students pursuing STEM degrees.

Of the 25 distinct programs, 13 specifically targeted incoming freshmen, with most of those schools beginning their recruitment when students are initially accepted into the university. One program encouraged students to apply to the intervention at the same time as they applied to the university [11]. Among those that targeted already accepted students, prevalent approaches were to promote and market these programs to eligible students through emails, phone calls, and flyers mailed to incoming students’ homes or posted around campus [9,37,38,39,40]. Many programs also asked current program participants to make information-sharing visits to freshman classrooms and student orientations [14,19,41,42,43]. In one case, because the program included community-living in on-campus housing, only students who expressed interest in living on campus were contacted [44].

As shown in

Table 2. Utilization of Admission Requirements.

Admission Requirement	Number of Programs Utilizing	Percentage
Express interest in a STEM-focused major	25	100%
Incoming Freshmen	13	52%
Explicit Grade or GPA Requirement	11	44%
Flexible or Very Few Criteria	7	28%
SAT or ACT Score Limits	4	16%
Academic Underperformance with High Potential	1	4%

, the requirements for admission into intervention programs varied greatly, with the only common requirement among all studies being an expressed interest in a STEM-focused major. Programs often had required HS or first-year college GPA criteria [10,11,12,14,41,42,45]. Others considered SAT or ACT scores as a factor [37,38,39,46]. In many cases, programs were specifically looking for academically talented students [47]; however, one unique program sought out students who underachieved during their freshman year, yet who demonstrated potential beyond what their grades indicated [48]. Finally, there were other programs with very few admission requirements, for these programs students simply need to be admitted into the university and have an interest in studying STEM [9,49].

3.1.2. Support

Here, we present the most common forms of support offered through intentional intervention programs. We attempted to separate these elements into two of the support types described within Tinto’s [24] conditions: academic and social. However, it is important to note that many support components are both academic and social in nature, for example, professional development opportunities and mentoring.

Academic Support

Most intentional intervention programs utilize multiple interconnected academic interventions.

Table 3. Utilization of Academic Support Components.

Academic Support Component	Number of Programs Utilizing	Percentage
Professional Development/Networking	18	72%
Research Experiences	17	68%
Tutoring/Study Skills and Strategies (including small group learning)	14	56%
Targeted Academic Intervention (specific assignments, coursework, enrichment classes)	8	32%
Graduate School/GRE-Prep	7	28%

indicates the full breakdown of academic support mechanisms.

Professional Development/Networking. Professional development is a broad term that can apply to any activity intended to better prepare program participants for a career in STEM. Professional development is also a category that combines elements of both academic or technical and social support. Eighteen of the 25 programs reviewed included at least one element of professional development. Some of these components are also included in our discussions of research, graduate school preparation as well as later on in our discussions of mentoring as there is a vast amount of overlap among the design of these elements.

In addition to the types of professional development already mentioned, many programs offered networking opportunities with current STEM professionals [46] and STEM-focused student professional societies [42], leadership development opportunities where students learned how to lead group study sessions for other participants [9], and visits to local businesses and organizations where staff worked in STEM-related fields to see STEM in action [12].

One program we found combined elements of mentoring and professional development via a weekend retreat for participating students, in this case, all women [43]. At the retreat, women had the opportunity to spend time with current STEM professionals, understand the challenges facing women in STEM careers, and learn about an online network to help them expand their ability to network and access mentors in the future.

Research Experiences

Seventeen of the 25 programs included in the literature, 15 of which are for undergraduates, incorporate research requirements or opportunities within their design. This is noteworthy because research is often reserved for graduate-level students only. In most situations, programs aimed to give students real-world research application opportunities as a form of active learning, which was heavily supported by faculty mentors [11,40,48,50,51]. Some of the literature acknowledged the potential level of effort required on the part of faculty to support research and questioned the value to cost ratio [52]. To address this concern, some programs used peer mentors instead of faculty to support research experiences [42,44,53].

One program used participatory action research experiences, a style of research in which students become active participants of a community and use their experiences within that community to study the effects of changes they implement. Because the experience is so focused on building relationships within a community, there is less involvement from faculty and peers [54]. Finally, another program focused on a different type of guided independent investigation: having students look into STEM career opportunities, investigate ethics within research, and write mock emails to inquire about future research positions [53]. An important point raised by a number of articles was if required research placed an additional burden on students [44]. To address this concern, two of the programs suggested that students participate in research experiences, but it was not mandatory [41,55].

Tutoring/Study Skills & Strategies

Fifty-six of the intentional intervention programs discussed in the literature included an element of general academic tutoring or study sessions to support students in their coursework. In most cases, tutoring and study skills support are provided as options that students can choose to take advantage of [16,42,50,56]. In other programs, study skills were taught alongside the required freshman-level curriculum [37]. One program required a minimum level of participation in tutoring or study groups but recommended that students participate more often, which resulted in students reporting feeling more involved [41]. Another program only required tutoring if a student fell below a set level of performance in their classes; however, some students continued using the service even once they had completed their required hours, and about 50% of student participants reported that the tutoring center was important to their success [38].

Tutoring was combined with mentoring in five programs; the mentors were the ones providing the academic support and a few included research projects [14,45,48,52,53]. The design and implementation of one tutoring program encouraged the creation of a feedback loop involving the student, their teachers, and a program coordinator. This allowed the optional tutoring sessions to also serve as a type of early monitoring system [9]. Particularly when offered by mentors, tutoring covers more than just the academic support condition of Tinto’s [24] framework. One example of how this style of the program was organized and funded is the engineering-student support-center model. These comprehensive centers provided tutoring, orientation services, and mentoring, thereby serving as academic and social supports, as well as providing evidence of the importance of the conditions of involvement, assessment, and feedback [16].

Targeted Academic Interventions

Eight articles discussed programs that included short-term, targeted academic interventions. One such study examined the effects of a utility-value intervention on the success of students in a STEM program [49]. This program instituted three writing assignments for students in a first-year biology class, asking them to reflect on the value and personal relevance of their coursework.

Additional program components that we defined as targeted academic interventions included enrichment coursework, often over the summer, intended to provide students with a leg-up on their academic foundation [40,47] or research skills [44]. One university offered these enrichment classes as part of a freshman’s first semester [37]. Finally, one program provided targeted academic support through the use of small-group problem-orientated sessions. These groups met weekly and were guided by a peer mentor to apply and extend what was taught in the traditional lecture-based classes [14].

Graduate School Preparation. Of the 23 programs that worked specifically with undergraduates, seven offered an element of graduate school preparation. This involved visits to graduate and medical schools, seminars about graduate school applications and admissions, and GRE-prep courses [9,45,48]. Only one program, in particular, focused heavily on graduate school admissions, using a rigorous week-long prep program including GRE-readiness, graduate school networking, a mini-research opportunity culminating in a poster presentation session, and mentoring from faculty and senior students [12].

Social Support

The results of the literature review indicate that social systems are used in almost all of the intentional intervention programs for LI, FG, and/or UR students in STEM, as shown in

Table 4. Utilization of Social Support Components.

Social Support System Component	Number of Programs Utilizing	Percentage
Mentoring (any kind)	21	84%
Social integrations experiences (community living, student organizations, non-academic cohort activities)	20	80%
Faculty mentoring	18	72%
Peer mentoring	15	60%
Community Service	7	28%

. Twenty-four of the 25 programs we studied included at least one element that could be categorized as social support. Of the program elements that we coded as social, mentoring was used in 21 of the 25 programs. The next most common were opportunities for social integration among peers, either through living situations, organizations, or activities.

Mentoring

Mentoring is integrated as one of several components in 84% of the universities’ degree programs’ support systems for underrepresented and first-generation students (see Figure 1—LI, FG, UR program support model). This component is often designed to serve both the academic and social needs of students in support of persistence and retention. Faculty and peer mentors typically perform different functions for students, given their roles, as shown in

Table 5. Mentoring Functions.

Faculty Mentors	Peer Mentors
Academic feedback and guidance	Social integration
Study skills support	Academic support from someone who’s “been there”
Professional development and networking	Bridging students and faculty
Exposure to research	Serving as a secondary teacher to lead small-group learning
Personal and familial-style support
Bridging a cultural divide

.

Unique Mentoring Approaches

While clear definitions of mentoring were largely absent from the literature, most programs indicated that their version of mentoring consisted of an ongoing relationship developed over time between the program participant and either a faculty member or peer.

Four of the programs used mentoring primarily as a way to provide support and discipline for students’ academic needs. Three of the programs indicated that mentors met with their protegees at least weekly [14,41,53] while another simply indicated that mentoring was a critical aspect of the program, but gave no details about frequency [48].

Mentors have an opportunity to reinforce expectations set by instructors. One program, in particular, was focused on creating a feedback loop between faculty, mentors, and program coordinators to serve as a check on student performance towards their goals [9]. This model is illustrative of the importance of Tinto’s [24] condition of assessment and feedback in helping students succeed.

In three of the programs, the mentor relationship was more friendly or familiar in nature, with mentors serving in a way as parents, siblings, and church leaders. This style of mentor goes beyond traditional faculty academic support, with activities including meals, holidays, personal support, and outings [11,45], and support techniques like empathetic listening to better connect with students [13]. Five programs took great effort to intentionally match students with mentors of similar backgrounds, races, gender, and interests [13,37,43,45]. In these cases, mentors served to bridge a potential cultural divide within the fabric of the university [10].

While mentoring within these programs was designed to be an ongoing relationship, one program acknowledged the fact that, over time, students naturally develop their own network of support. Therefore, the assigned mentoring was phased out after the first year [37] in favor of students working with self-selected mentors. Another way that mentoring is brought into these intentional intervention programs is through the use of student participants as mentors themselves. Three models attempted to distribute the amount of work involved in mentoring by using multi-tiered approaches—faculty mentors who lead and support peer mentors, each with multiple students [42,44,53]. Four of these programs encouraged or even required upperclassmen in the program or program alumni to mentor freshmen and sophomores [11,39,42,48]. This style of the program provides support to program participants, while also allowing upper classmen to expand their skills and knowledge.

Mentor Preparation

Very few of the articles discussed the details of the training and support the mentors themselves received before serving in their role. However, in the few that did, this element varied greatly. One program required peer academic mentors to participate in a year-long course about pedagogy as well as meet weekly with faculty to discuss content and questions relevant to that week’s lesson [14]. One program supported its peer mentors with weekly training meetings led by faculty [53]. Another program prepared faculty mentors through intensive week-long training, held at a partner university that has experienced proven success in this field [10]. Finally, two other programs provided occasional workshops for their mentors that included topics such as mentoring best practices, expectations, and ethics [43,48].

Social Integration Experiences

A number of programs incorporated social integration experiences within the intervention program. These components provided students with support and camaraderie completely outside of the classroom, a chance to feel connected to something smaller within an often overwhelmingly large university [51]. The majority of programs strove to create a sense of community amongst the participants. They accomplished this through group trips, events, and activities [37,38,41,57], free access to meeting areas, and socialization spaces [9], and participation in STEM-focused student organizations and professional societies [42], including those that were race and gender-specific [15]. A more robust form of social integration came in the form of living-learning communities, in which program participants were either given the option or required to live together in program-specific housing to encourage more frequent casual social interaction [44,45,50,56].

Community Service

Approximately a quarter of the programs in the literature included a community service element. This included general requirements for volunteer work [11], often serving as a mentor to K-12 students within the community [42,48], as well as service-learning opportunities in which academics are tied to community betterment [12].

Transition and Summer Bridge Programs

As mentioned previously, most of the 25 programs discussed in the literature incorporated multiple elements of support. A common way to combine these elements was through a transition or summer bridge program. This intensive, short-term style of intervention was a part of 11 of the 25 programs.

Summer bridge programs are unique in that they occur outside of the traditional academic calendar of the university. From articles that discussed program length, they ranged in duration from two weeks to six weeks long, and all took place before a student’s freshman year. Despite differences in duration, all programs utilized similar components and techniques, including mentoring, enrichment coursework, study skills classes, research opportunities, and team-building exercises [11,38,39,56]. One summer bridge was particularly focused on the role that faculty played during the program and beyond, ensuring that they were well-prepared and committed to the responsibility [10]. Some of the transitional bridge programs included in the literature were focused on helping students move from a two-year community college into a four-year university program [50,57]. These programs used similar support mechanisms as those for incoming freshmen, including mentoring and academic advising.

Finally, a unique type of transition program emerged from the literature, created to help LI, FG, and/or UR students in STEM experience success in graduate school. One of these programs was a transition boot camp for Black college juniors and seniors, in the form of an intensive week of graduate school admissions prep and readiness [12]. The other program supported the transition of Masters-level students into a Ph.D. program, financially, academically, and socially. It established for them a scientific foundation that made their success in doctoral studies more likely [51].

Financial Support

One of the most straightforward means of support within these intentional intervention programs came in the form of financial aid. At least 14 of the 25 programs offered specific financial support and incentives for program participants. While the other 11 programs did not explicitly describe a financial element, some referenced the existence of traditional student loans to aid with the economics of college.

The amount of financial support varied dramatically from program to program, with two programs offering small stipends of a few hundred dollars for books [44,57], three offering larger stipends of a couple of thousand dollars that could be used for any expenses [39,42,50], and seven offering more generous scholarships, sometimes covering all student costs [10,11,37,38,41,45,47].

Influencing Persistence-Related Character Traits

Tinto’s [24] condition of involvement, also referred to as engagement, plays a factor in almost every program element we uncovered. While the literature did not discuss measures that programs took specifically in order to increase engagement beyond the monitoring and support mentioned previously, programs did attempt to assess how their interventions may have impacted character traits that are closely related to both engagement and persistence. Many programs considered the role that a student’s personal identity played in their outcomes within a STEM major. Identity in the case of UR students in STEM can refer to a sense of belonging or self-worth, but it can also simply mean a student can see themselves as a scientist [11,43,48,54,58].

Interventions also examined and attempted to influence other concepts related to identity, including a student’s belief in a growth mindset [38,48], destigmatizing the idea of failure [39], and increasing confidence and perceived relevance of their studies [49]. A primary barrier to engagement for UR students in STEM mentioned in many articles, and acknowledged within many programs, was the difficulty students have in feeling like they belonged [15,37,57]. Programs attempted to positively influence this sense of belonging through many of the elements presented prior, including mentoring, social integration, research, and professional development.

Other programs considered the impact of their interventions on similar factors. These factors include a stereotype threat, the fear of conforming to a traditional racial or gender stereotype, [13,59], performance-avoidance goals, a “focus on avoiding demonstrating incompetence relative to peers” [58]. Lastly, some researchers considered social boundary theory, or how a student may perceive themselves as similar or different from their peers, especially intellectually [51].

3.2. Question 2: What Can We Learn from Empirical Outcomes of Interventions Intentionally Designed to Support UR and FG Students’ Retention and Persistence in Stem Degree Programs?

The empirical evidence linking program components to results is shown in

Table 6. Articles Describing Empirical Evidence Linking Program Components to Results.

	Did the Program Have a Measured Positive Impact?				Did the Program Include the Following Component?
	Increased STEM Retention	Increased Graduation Rates	Increased GPA	Positive Change in Personal Characteristics	Financial Support Offered	Measure of Students’Personal Characteristics	Students Must Express STEM Goals	Research Experiences	Academic Tutoring	Professional Development	Study Skills and Strategies	Graduate School/GRE Prep	Targeted Academic Intervention	Peer Mentoring	Faculty Mentoring	Social Integration	Community Service	Summer Bridge or Transition Program
Ballen and Mason (2017) [9]	no	no	no		yes		yes	yes	yes	yes	yes	yes		yes	yes	yes
Banda and Flowers III (2017) [15]	unclear									yes						yes
Belser, Shillingford, Daire, Prescod, and Dagley (2018) [46]	yes					yes	yes	yes	yes	yes	yes			yes		yes
Cavner and Stanny (2018) [53]	partial		partial			yes		yes	yes	yes	yes			yes	yes
Chang, Kwon, Stevens, and Buonora (2016) [41]	yes	yes		yes	yes	yes	yes	yes	yes	yes	yes			yes	yes	yes	yes
Chapman, Hill, Nagel-Myers, and Ramler (2019) [37]			yes	yes	yes	yes	yes		yes	yes	yes			yes	yes	yes
Doerschuk, Bahrim, Daniel, Kruger, Mann, and Martin (2016) [42]	yes	yes		yes	yes		yes	yes	yes	yes	yes			yes	yes	yes	yes
Domingo, Sharp, Freeman, et al. (2019) [10]	yes	yes	yes		yes		yes	yes	yes	yes	yes	yes		yes	yes	yes	yes	yes
Drane, Micari, and Light (2014) [14]	yes		yes				yes		yes		yes		yes	yes
Fakayode, Yakubu, Adeyeye, Pollard, and Mohammed (2014) [50]	yes				yes		yes	yes		yes					yes			yes
Gibson, Siopsis, and Beale (2019) [38]	yes	yes	no	partial	un-clear		yes	yes	yes	yes	yes			yes		yes		yes
Hecht, Harackiewicz, Prinski, Canning, Tibbets, and Hyde (2019) [49]				yes									yes
Hernandez, Bloodhart, Barnes, Adams, Clinton, et al. (2017) [43]				yes	un-clear	yes	yes			yes					yes
Kendricks, Nedunuri, and Arment (2013) [45]	yes				yes		yes	yes	yes			yes			yes	yes
Lisberg and Woods (2018) [39]	yes		yes		yes	yes	yes		yes		yes			yes	yes	yes		yes
McGonagle, Freake, Zinn, Bauerle et al. (2014) [57]	yes	yes	partial	yes	yes	yes	yes		yes	yes	yes			yes	yes	yes		yes
Ononye and Bong (2018) [47]	partial	partial	partial		yes		yes		yes	yes			yes	yes	yes	yes
Oseguera, Park, De Los Rios, Aparicio, Johnson (2019) [11]	yes		yes	yes	yes	yes	yes	yes	yes					yes	yes	yes	yes	yes
Peteet and Lige (2016) [12]	yes				un-clear		yes			yes		yes	yes			yes		yes
Posselt, J., Reyes, K.A., Slay, K.E., Kamimura, A., and Porter, K.B. (2017) [51]	yes	yes			yes	yes	yes	yes					yes		yes	yes		yes
Schneider et al., 2015, Bickel, and Morrison-Shetlar (2015) [44]	yes				yes		yes	yes					yes		yes	yes
Schwartz (2011) [52]	yes	yes								yes		yes			yes
Tomasko, Ridgway, Waller, and Olesik (2016) [40]	yes		partial					yes		yes			yes			yes		yes
Weinberg, Trott, and McMeeking (2018) [54]				yes		yes		yes					yes			yes	yes
Wilson, Holmes, deGravelles, Sylvain, Batiste, et al. (2011) [48]	yes	yes				yes				yes	yes	yes		yes	yes	yes	yes

Note. The blank cells above depict outcomes and/or interventions that were not reported in the empirical results of the article. The light-gray cells labeled “yes” depict improved outcomes and/or interventions that were reported in the empirical results of the article. The dark-gray cells labeled “partial” depict outcomes for URM students that are at least equal to non-URM students in the empirical results of the article for some participants. The dark-gray cells labeled “unclear” depict outcomes that may or may not be directly related to the program design and/or interventions that were not experienced by all participants. The black cells labeled “no” depict articles describing outcomes and empirical results for which the authors found no positive change related to the interventions identified.

. The table includes the author and year for each article, and the results and component details for each of the 25 distinct intervention programs aimed at increasing the retention and persistence of UR students in STEM majors.

We had hoped to discover clear patterns of relationships between specific program components and measures of program success. However, the data did not support overwhelmingly clear conclusions about which program components were the most effective in terms of their ability to increase persistence and retention amongst underrepresented students. Instead, we found that a program’s success did not seem to specifically correlate with which program elements it employed. For example, the outcomes of one program that included 10 different program elements (e.g., tutoring, mentoring, and professional development) demonstrated no correlation between program participation and student retention, GPA, or graduation rates [9]. Whereas another program that utilized a single targeted academic intervention in the form of three writing assignments over the course of one semester found gains in student grades for that semester and students’ perceived levels of confidence, both factors that may increase persistence over time [49].

In the most general terms, all but one intervention program found at least partial success: 96% of programs experienced positive outcomes. However, those successes must be taken with a grain of salt. Simply by the nature of the articles having been written, there was a bias towards success. While it would be difficult to know how many similar programs are in existence that lack measurable positive results, such programs are less likely to appear in peer-reviewed academic journals.

Despite the shortage of apparent correlation between program design and student outcomes, the literature led us to seven guidelines regarding documentation of success outcomes (i.e., individual students and environmental support systems) and program design decisions.

3.2.1. Success Definitions Are Important, and Indicators Vary Widely

All 25 intentional interventions discussed in the literature were put in place to increase retention and persistence. However, how persistence and retention were measured varied from program to program. Table 6 outlines the four most common measurements used to evaluate the success of an intervention program:

3.2.2. Year over Year Retention

The majority (21) of programs evaluated their success based on the percentage or number of students who remained in a STEM major from one year to the next. However, the studies shared did not always include a control group or provide any way to evaluate the percentage of students retained against the school averages [15,47]. That said, 17 programs demonstrated success on this measure for participants as compared to non-participants at that same university based on their self-reported data.

3.2.3. Graduation Rates

Ten of the 25 programs specifically measured the graduation rates of program participants. Some programs measured the six-year graduation rate [41], while other programs with shorter durations were only able to measure graduation rates after two [52] or four years [10,38]. Eight programs indicated that participants were graduating at a higher rate than non-participant STEM majors at the university. Two of the studies compared their graduation rates with the national averages [48,51], and both demonstrated success on that measure.

3.2.4. Grade Point Averages

Eleven programs measured student grades and GPA as indicators of program success. The common hypothesis was that participation in the program would lead to higher grades and overall GPA. The challenge in using grades as a measurement is their variability from professor to professor and school to school. It was also demonstrated that a program could increase retention and graduation rates without the participants earning higher grades than non-participants [38,57]. That calls into question the efficacy of grades as a program measure. Of the programs that evaluated grades, five saw a clear improvement for participants, and four experienced partial success, with grades improving in some cases but not in others.

3.2.5. Student Self-Reported Beliefs

Twelve programs evaluated participants using measures of personal characteristics that are traditionally linked with increased persistence. All of these programs used surveys in which students self-reported their perception of certain characteristics over time. These characteristics included a sense of belonging [37], scientific identity [37,43], engagement [41], perceptions of growth [42], and confidence [38,49], all of which have been shown to have a positive correlation to student persistence. All 12 programs that measured these characteristics demonstrated increases in at least one of the studied factors. While an increase in factors relating to persistence is a good thing, it is important to consider the effect of attentional bias on student characteristics that are being overtly discussed and measured.

3.3. Program Design and Implementation Details Are Sparse and Varied

Based on the results from this literature review, it would be difficult to say with confidence what precise details would offer the best return on investment. Many of the articles we reviewed described very little, if any, details regarding the program design decisions made by those who created the programs. Sharing these details has been shown in the research to be a key factor in advancing professionals’ problem-solving [33], research [33], and integral to building design knowledge grounded in successes and failures [32,60]. Further, the average duration of the intervention programs analyzed is four years. While some programs were able to track participants as they entered the STEM workforce after graduation, the real impact of participating in an intentional intervention may not be fully realized for years to come.

Another variable that was usually unaccounted for in the literature is the quality of program implementation. This variance is hinted at when articles discuss how some mentors were evaluated as more helpful and prepared than others [44]. A program may be designed with all of the best intentions, but if the execution is lacking the outcomes will likely be as well.

This emphasizes the importance of proper training, preparation, and buy-in for all program staff and faculty, another program element that was discussed only briefly in most articles. Domingo et al. [10] went into detail on the benefits of the extensive weeks of training the study-subject faculty received from the long-running Meyerhoff Scholars Program at the University of Maryland Baltimore County. The author attributes the training and the ongoing support of the partner program as the primary reason for the success they experienced.

3.4. All Program Components Should Not Be Required for All Participants

While the multi-pronged intervention programs in the literature usually included a wide variety of academic and social elements, an important distinction was raised by some programs: are these activities required or suggested? It is now understood that what works for one student might not work for others [61]. In acknowledgment of that principle, some programs used suggested, rather than required program activities.

The research was one of the components mentioned as an element that had the potential to be overwhelming to students if it were a program requirement. Students in one program cited the requirements to conduct research and support courses as a teacher’s assistant as the primary negatives of participation [55]. One program in particular very intentionally did not make research a requirement, rather it was a recommendation along with a number of other activities, yet the program still demonstrated improvements in the students’ sense of involvement, STEM retention, and graduation rates [41].

In a similar vein, some programs discussed the implications of required participation on the faculty as well. One program, in acknowledgment of the amount of time and effort required of faculty to effectively support students, reduced an instructor’s required course load [42]. Another article went even further and warned of the potential professional, emotional, and financial toll that serving as a mentor within an intervention program could take on a faculty member [52].

3.5. Acknowledging and Addressing the Considerable Challenges of Underrepresentation on Students Leads to Better Outcomes

A common theme among programs was creating a sense of belonging for participants. This referred to belonging within the university, but also within the fields of STEM. By the nature of being underrepresented, students within these programs often had a difficult time feeling like they fit in. However, cultivating a sense of belonging can be challenging.

A number of programs indicated that they attempted to provide students with mentors of the same race and/or gender as the participating students [13,37,43,45]. Their intention with this matching was to create a bridge for students across the potential cultural divide that existed at the universities [10]. Despite the fact that research indicates that matching the race and/or gender of a mentor and a student does not produce any academic benefits, students still felt more supported by mentors who were of the same race or gender [62].

One program, in particular, attributed its limited success with retaining LI, FG, and/or UR students to a lack of diversity amongst program mentors. Because the mentors were not of the same racial or ethnic backgrounds as the UR students, the article suggested the program may have been less effective for those students, despite seeing success with non-UR students in the program [53].

Another program discussed how students intentionally sought out membership in organizations that were race or ethnicity-specific, yet often they still felt alienated because not all STEM students of the same ethnicity have the same backgrounds, wants, or needs [15]. Therefore, while a race, gender, or ethnic-specific group might help some students feel more connected, it is not always that simple.

3.6. Research Supported by Mentors Seems to Be an Essential Program Component

There were many program elements common to most of the interventions, yet none were discussed at more length as an essential component of mentor-supported research. For four of the programs, research was the primary element of the intervention. Three of those programs had increased student retention from year to year [44,49,52], and the fourth demonstrated a positive impact of research on students’ scientific identity and confidence [54]. One of those programs had incredible retention rates among participants, seeing over 98% of students retained in STEM majors from year to year, while the average STEM retention for the school prior to the program was approximately 40% [49].

For STEM fields, research is a bridge between academia and professional experience. It is a chance for a student to experience what it means to be a scientist. And as many articles discussed, being able to see themselves as a scientist is a vital part of success for UR students in STEM [43]. From an academic perspective, research is an example of active learning that encourages the transfer of knowledge to action [24]. In addition, most of the articles referenced described research that was supported by either a faculty or peer mentor, which also gives it a social support component.

In one program, participation in research reduced a student’s fear of demonstrating incompetence to their peers called performance avoidance. This program found that replicating an authentic scientific process, including the existence of failures and setbacks, was an essential component of successful student interventions [58]. In another program, all of the students who participated in research remained in STEM majors through graduation [38].

When evaluating program components against Tinto’s [24] conditions, mentor-supported research ticks many of the boxes. Research has clear and high expectations of a student, it is a socially supported academic activity in which students are engaged and involved, and when done well, it provides them with feedback on their performance. Therefore, it is not a surprise that so many programs found it to be an essential component.

3.7. A Commitment to the Program and Its Students Makes a Positive Difference

While only mentioned in one article, a noteworthy sentiment is the idea that in order to be successful, an institution must be fully committed to the intervention model. Proper execution is a variable that is hard to evaluate, which is why it is not surprising that it was rarely mentioned. However, we can take a lesson from the one program that did complete a thorough review of the program’s implementation [10]. This program had remarkable increases in retention and GPA for UR participants, and their successes led to similar programs being adopted at two additional schools.

The stated keys to the success of this intervention were a commitment to the program model, sufficient ongoing support from administration, recruitment of full-time staff, and extensive training and participation from faculty [10]. It should go without saying that even the most well-designed programs are less likely to succeed if they are not given the appropriate amount of support and resources from the faculty and administration of the institution.

3.8. There Is One Program with Clear and Overwhelming Success

There is one model of ongoing long-term success in place at the University of Maryland Baltimore County since 1988, the Meyerhoff Scholars Program (MSP). This multi-pronged intervention includes over a dozen program components including financial support, mentoring, community service, and research. Participants of the Meyerhoff Scholars Program graduate with STEM degrees at twice the rate of non-participants and they are five times as likely to go on to a Ph.D. program in STEM [63].

The Meyerhoff Scholars Program (MSP) has been so successful that it was cited as the implementation inspiration for programs in three of the articles returned in our search. Two of those programs themselves saw success [10,11]. Interestingly, however, the third model inspired by MSP was the one program from our research that did not see student gains in retention, GPA, or graduation rates [9]. This reinforces the idea that with so many variables at play, it is difficult to know why some programs succeed when others do not.

While we are hesitant to place too much emphasis on one intervention model, because of the success it has demonstrated the Meyerhoff Scholars Program should be considered an excellent source of insight into what can be effective. Stolle-McAllister et al. [63], published a study to share students’ perceptions indicating the most important elements of the program to student success based on student feedback as shown in

Table 7. Important Meyerhoff Scholars Program Components.

MSP Components	Details
Financial Support	Free tuition Funding for undergraduate research and conferences Additional funding for graduate research work
Summer Bridge	Introductory college coursework Social community building Academic planning support Mentoring Study and professional skills classes Research Opportunities
Identity Building	Fosters a strong sense of successful student and scientific identity through recruitment, selection, and participation because of the program’s reputation, high expectations, and ongoing support.
Sense of Belonging	Program is designed to create a family-like feeling by focusing on group success and eliminating competition, and through student’s shared goals, interests, and skills. Program maintains connections with students beyond graduation.
Network Building	From the start of the program, participants are connected with program alumni, graduate schools, internship opportunities, and professional scientists.

below.

The results of this systematic MSR indicate that intentional select interventions, put in place to increase the retention and persistence of LI, FG, and/or UR students in STEM, show considerable promise in their ability to create positive outcomes. These articles show that interventions are incorporating all four of Tinto’s [24] conditions within their program structure. However, overwhelmingly, the results indicate that there is limited information available about the specifics of program design. In addition, there is a need for a more universal measurement of retention and persistence. While studies indicate that most of these interventions are experiencing success, it is difficult to compare outcomes from program to program, and it is difficult to know what kind of long-term impacts may exist from their use.

4. Discussion

It is clear from the available research that intentional intervention programs put in place to increase the retention and persistence of LI, FG, and/or UR students can work. All but one program in the literature demonstrated successful outcomes. However, what is still somewhat unclear is which program components were most impactful and why.

While studying intervention programs for LI, FG, and/or UR, in other academic fields, Hallet, Reason, Toccoli, Kitchen, and Perez concluded that “how program elements are implemented is more important than what elements are included within a support program” [64], p. 253. Unfortunately, with limited information on the “how” available, we are restricted in our ability to suggest a precise course of action for an institution seeking to establish a STEM-focused intervention program. It is also difficult to extrapolate these findings into an understanding of graduate higher-education programs. While successful completion of a STEM undergraduate degree program is the first step towards success in a STEM career, most STEM fields require more advanced education. The dearth of research on graduate-level programs indicates that more work must be done in this area.

What we can do is use the general framework put forth by Tinto [24] and add details from the successful Meyerhoff Scholars Program as well as the 25 other programs reviewed from the literature, to establish a starting point for our determination of what program components should be included in any new intentional intervention program. At the same time, it is essential to keep in mind that the ultimate outcomes of any program are reliant upon the commitment and support of staff, faculty, and administration.

As the suggestions in

Table 8. A Synthesis of Tinto’s Conditions and MSP Components to Yield Direction for Emerging Future Program Designs.

Tinto’s Conditions (2012)	Related Important * Meyerhoff Scholars Program Components	Suggested Components to Include in Emerging Future Program Designs
Expectations	Identity Building	Robust marketing, recruitment, application, and selection process Mentor supported research opportunities Committed staff and faculty to monitor and create accountability
Support	Full Financial Aid Summer Bridge Networking	Sufficient financial aid to reduce economic barriers Peer and faculty mentoring Summer bridge transition program with academic and social components
Assessment and Feedback	Summer Bridge Networking	An ongoing academic feedback loop with students, mentors, and teachers Summer bridge transition program with academic and social components Access to program graduates and professional scientists for career and academic guidance
Involvement	Summer Bridge Sense of Belonging Identity Building	A balance of required and suggested program components Mentor supported research opportunities Community learning and living started via a summer bridge transition program Training and information on growth mindset, study skills, conflict resolution, professional development

* As determined by student feedback.

indicate, an intentional intervention program to increase the persistence and retention of LI, FG, and/or UR in STEM should include a number of different components. These components should be balanced across Tinto’s [24] four conditions to ensure a robust system that will attract the right students, set appropriately high expectations, provide adequate financial, academic, and social support, create a feedback loop amongst students, staff, and faculty, and encourage and monitor engagement throughout the duration of the program.

4.1. Limitations

Conducting a study on LI, FG, and/or UR students is a sensitive undertaking, even more so when the researchers are of Caucasian descent. Women are undoubtedly included in the phrase UR in terms of participation and success in STEM education. The research author team is diverse and while some of the researchers cannot fully know the barriers and challenges that come with being an underrepresented racial minority in any context others of us can identify more closely with minoritized populations in the U.S. Furthermore, as women, several of the researchers of this project do have a unique understanding of what it means to be a woman in both science and academia.

Moreover, while all articles and studies used in this paper were empirical in nature, not all sources reported their study details with the same stringency. Because our methods did not give more priority to studies with more robust research protocols, it is possible that too much attention was given to articles that may have been less valid in their approaches. Related to this, we equally considered qualitative, mixed-methods studies, and quantitative research, which can make the data more difficult to compare and evaluate.

Lastly, we acknowledge our work is grounded in an economist view. We looked at empirical studies and their impacts on retention, graduation, and GPA. Future humanist research approaches could add valuable perspectives to consider.

4.2. Threats to Validity

To obtain more relevant and timely findings, we restricted the date range of our search to the years from 2005 through March of 2020. While research may exist prior to our date range and not considered, we focused on the last 20 years to ensure our results are relevant and timely. Moreover, we only considered sources written in English and most represented North American contexts, therefore, there may be available information in other languages from other geographic areas that were not consulted.

Possibly, we omitted relevant empirical research articles that were unavailable in our library database. For example, our search only yielded English-language studies. Moreover, the majority described studies conducted in the U.S.

Finally, as with all empirical research published in academic journals, there is the threat of publication bias. In most cases, the articles we referenced were written by someone with a connection to the university where the intentional intervention was implemented. It is possible that programs with success may have been more likely to seek publication, just as it is possible that journals may have been more likely to choose to publish articles describing a noteworthy or successful outcome.

4.3. Implications

Our goal is to provide a high-level roadmap for universities that are considering implementing an intentional intervention program in the hopes of retaining and graduating more UR students in STEM. While our research was not able to answer all of the necessary questions an institution might have during that process, we believe that the evidence indicates that certain program components are more likely to result in successful outcomes than others. The success of the MSP, the subsequent success of other programs modeled after MSP, and the work done by Tinto [24], all combine to provide useful empirical evidence of practices that work.

Intentional intervention programs should:

• Set clear and high expectations for incoming students and put in place systems to support students when those expectations are not being met.
• Provide adequate financial support.
• Offer combined academic and social programs, including research, mentoring, and professional development, potentially through a summer bridge transition program.
• Focus on increasing students’ sense of belonging, engagement, and scientific identity.
• Include faculty and administrators who are committed and well-trained in supporting underrepresented students.

It is important to note, however, that due to the limited amount of research conducted on intervention programs geared towards LI, FG, and/or UR graduate students, we cannot be certain if the program design suggestions made for undergraduates would be equally as applicable for students in masters and doctoral programs.

4.4. Future Research

The opportunities for future research are vast. The need for well-prepared diverse graduates of STEM higher education programs will only continue to increase, as such the existence of intentional interventions to support diversity in these fields is likely to increase as well. Given the relative newness of the literature available on these programs, it is also likely that there are many other studies occurring at this moment that will be written about in the coming years.

That said, the true dearth of data on this topic lies in the isolation of program variables to be able to determine what really works. While there is student-supplied information on their perceptions of which program components were most beneficial, it is been well documented that students often have false perceptions of what works best for their own learning and development.

To effectively and definitively determine which program elements are most effective at increasing the persistence and retention of UR students in STEM, many more descriptive and controlled studies in a variety of contexts could be conducted where program elements are treated as isolated variables. Ideally, there would also be financial measurements of the true cost of each component. That is a challenging proposition, and also potentially a waste of precious time. Rather, it may be possible to roll out an intervention program in stages, such that new elements are added to the program periodically, with impacts measured longitudinally over time to identify the added value of each new component.

Furthermore, the MSP notwithstanding, there is also a lack of longitudinal data that follows program participants into their post-academic careers to determine the long-term impacts. While graduating with a STEM degree is an obvious accomplishment, data that indicate the types of careers and futures that these students are now experiencing would also provide more insight into the value of an intervention program.

Finally, there is also very limited information available on the success of these types of intervention programs within graduate school settings. Only three of the 31 articles we considered for this study focused on LI, FG, and/or UR graduate students. Therefore, it it is difficult to know if what seems to work for LI, FG, and/or UR undergraduates will also work for graduate students. Because many STEM careers require advanced higher-education degrees, we see an opportunity to extend these intervention programs into graduate school to gain a useful understanding of how an institution can further support LI, FG, and/or UR students as they prepare to enter the STEM workforce.

5. Conclusions

This novel systematic-multiple-studies review of the empirical literature adds a synthesis of published empirical research study patterns, which are just developing in this area. A systematic multiple-studies review demonstrates stronger evidence than one study alone and is highly significant for decision-makers who need evidence-based practice guidance for their programmatic designs [65]. This systematic MSR of the literature [27] was based on a collection of 31 articles published between 2000 and March 2020, narrowed from an original field of 124. A search of Academic Search Premiere using the same search terms indicated earlier, with dates restricted from 1979 through 2005 resulted in two potentially relevant records. That dramatic increase in available data alone is an indication that intentional intervention programs for UR students in STEM are growing in prevalence and demand.

Our findings demonstrate that there is no single list of elements one can include to guarantee improved retention, GPA, or graduation results, which is significant yet not universally well-understood. The 31 articles considered herein described 25 unique intervention programs used at colleges and universities across the U.S. Of these programs, 24 experienced some form of success. Our goal was to understand why these programs were successful and to extract key features from each that might help us create a model for the intervention programs of the future: a program that included only the most beneficial elements to make the greatest impact for the smallest cost. Although variability exists among effective strategies across different programs, we did uncover a number of essential details about what successful programs are doing. By aligning these details to other models of persistence and retention, specifically those of Tinto [24] and the UMBC MSP, we were able to craft an outline to guide the development and implementation of future interventions.

The next ten years are critical to the evaluation of intentional intervention programs to support UR in STEM. As these programs increase in frequency, more data will become available that may help better determine exactly which program elements are most impactful. In addition, as the STEM students of yesterday and today complete their schooling and begin their careers, we will be able to see the real impact of program participation on the rapidly growing demand for a diverse and well-prepared STEM workforce.