Integrated STEM Approaches and Associated Outcomes of K-12 Student Learning: A Systematic Review

Abstract

Educators and researchers are increasingly recognizing the potential benefits of integrated science, technology, engineering, and mathematics (STEM) education to improve students’ learning outcomes, including the learning achievements, interest in STEM, learning motivation, and higher-order thinking skills of K-12 students. While there is a considerable body of research on this topic, it lacks a comprehensive synthesis of the available evidence to provide a more rigorous and systematic understanding of the relationship between integrated STEM approaches and associated outcomes of K-12 student learning. Therefore, the purpose of this study was to examine the integrated STEM approaches and associated outcomes of K-12 student learning through a systematic literature review. The studies were accessed using the Scopus, ERIC, and Google Scholar databases in February 2022. A total of 47 studies were retained for inclusion in the review. We used the ecological triangulation method for data extraction and synthesis. A total of 23 ecological sentences developed from existing studies revealed that the associated outcomes of K-12 student learning occur differently when using different integrated STEM approaches. For example, STEM project-based learning activities in the science curriculum focused on improving students’ learning achievement and higher-order thinking skills, while out-of-school STEM project-based learning activities focused solely on students’ STEM career interests. Finally, we note several directions for future research related to student learning outcomes using integrated STEM approaches.

1. Introduction

Science, technology, engineering, and mathematics (STEM) education is increasingly being globally recognized as the foundation for national development and productivity, economic competitiveness and social prosperity [1,2]. STEM knowledge and skills are key to enhancing the quality of the STEM workforce [2]. Therefore, developing competencies in STEM disciplines is a key goal of education systems, as countries recognize the importance of STEM competencies for students’ future careers in the 21st century [2,3]. In K-12 schools, efforts to improve STEM teaching and learning have focused on interdisciplinary or integrated instruction, commonly referred to as “integrated STEM education”, rather than a separate subject approach [4]. Integrating STEM subjects into a new interdisciplinary subject provides K-12 students with the opportunity to make sense of the integrated world, rather than learning and practicing fragmentary pieces of knowledge [5]. Although an integrated STEM education has been well established through national and international policy documents, disagreements regarding implementation models for integrated STEM and the associated outcomes of K-12 student learning continue to be problematic [1,4]. In the research, an overarching report showing a convincing relationship between integrated STEM approaches and outcomes for K-12 student learning has yet to be completed [3,6,7]. In practice, teachers face difficulties implementing integrated STEM, because they lack guidance on effectively integrated STEM approaches and the learning outcomes intended for their students [3,4,5,8,9,10]. Teachers can be overwhelmed by the wide variety of integrated STEM activities that can be applied in practice, including pedagogical models, such as STEM project-based learning, STEM camps, STEM clubs, STEM activities based on the 5E model, STEM activities based on the engineering design model, STEM competitions, and university–school partnership programs [11], and their impact on learning outcomes. For example, integrating engineering into middle school science classrooms helps students to better understand science and engineering, but has no significant impact on students’ interest in science and engineering [12]. Therefore, the purpose of this study was to examine integrated STEM approaches and associated outcomes regarding K-12 student learning through a systematic literature review.

Such a systematic review will synthesize existing research findings to determine: “what integrated STEM approaches work for what types of learning outcomes for what types of students under what educational settings”. This can be beneficial for educators/teachers, policymakers, school leaders and researchers developing and implementing effective STEM education programs to maximize student learning outcomes.

2. Conceptual Framework

2.1. Integrated STEM Approaches

There are various definitions of integrated STEM in the literature and policy documents [4,8]. Definitions commonly include the use of real-world context to link some or all of the four STEM fields [4]. In a basic sense, integrated STEM education can be defined as an approach to teaching the content of two or more of the four STEM subjects, using real-world/authentic contexts to integrate the content of these subjects and enhance student learning [3,4,13]. Instead of teaching knowledge and skills pertaining to separate subjects and expecting students to see their connections with real-world problems, integrated STEM education seeks to clearly define the connections between STEM subjects and provide a relevant context for learning STEM content [3].

Furthermore, we understand the term “integrated STEM approach” as referring to implementation models for an integrated STEM education [14]. The integrated STEM approach aims to find connections between STEM subjects and build a relevant context for learning the content [3]. An integrated STEM approach requires teachers and students to be aware of when and how to apply the knowledge and practice obtained from STEM subjects [7]. In K-12 schools, integrated STEM is typically implemented with problem- or project-based learning activities, engineering design-based learning activities, 5E instructional models, STEM-oriented robotics, where the necessary knowledge can be distributed differently across STEM subjects [7]. A variety of STEM activities are generally described as integrated by their creators.

The researchers are based on the degree of related discipline overlap to classify STEM activities into the appropriate category of integrated STEM approaches [13,15]. Some scholars classify integrated STEM activities into the categories of cross-disciplinary, multidisciplinary, interdisciplinary and transdisciplinary [13]. Others refer to integrated STEM approaches with labels such as isolated, connected, nested, multidisciplinary, interdisciplinary, and transdisciplinary [15]. These categories differ in whether the boundaries between STEM subjects are clear, blurred, or entirely dissolved [13,15]. Although the idea of classification is inspiring, it may not be easy for STEM teachers to directly apply in their teaching and learning practice.

One of the primary implications of this study pertained to STEM teachers: we required a framework for the classification of integrated STEM approaches that could be directly applied by STEM teachers in K-12 schools. For that reason, we used the classification framework of integrated STEM approaches proposed by Rennie et al. [13]. By interviewing STEM teachers, Rennie et al. identified six types of integrated STEM approaches, including synchronized, thematic, project-based, cross-curricular, school-specialized, and community-focused programs [13].

−

Synchronization-based integrated STEM approach: Teachers identify common knowledge and skills in two or more subjects, and teach those subjects separately but create knowledge connections to reinforce these concepts [13].

−

Thematic-based integrated STEM approach: Teachers work collaboratively to teach their subject around a local or global theme [13]. They teach their subjects separately and make connections with the theme.

−

Project-based integrated STEM approach: The focus of the lesson is on the implementation of project tasks that require knowledge and skills from a variety of subjects [13]. Projects often require a final product.

−

Cross-curricular-based integrated STEM approach: STEM integration occurs when many inter-connected lessons are conducted to develop student’s knowledge and skills through the study of interconnected topics. Its purpose is to develop a student’s overall skills or competencies [13].

−

Specialized school-based integrated STEM approach: When a school has a long-term focus on a specific STEM area, such as a coastal high school with a specialization in marine studies, teachers can customize their courses so they all have a clear association with this specialization [13].

−

Community-focused integrated STEM approach: When a community issue becomes the focus of a STEM curriculum, such as technological solutions for the prevention of the COVID-19 pandemic, teachers can orient their teaching of subjects to help students understand problems from different perspectives and seek potential solutions [13].

In this study, we used six types of integrated STEM approaches presented by Rennie et al. [13] to explore the implementation models within each type of integrated STEM approach and the associated K-12 student learning outcomes.

2.2. K-12 Student Outcomes in Integrated STEM Approaches

There is a growing interest in integrated STEM approaches and their potential to improve student learning outcomes [16], including learning achievements and effects (motivation, interest, higher-order thinking skills) [4,5,17,18,19]. Proponents of the integrated STEM approach believe that using real-world problems as a learning context provides positive motivation for learning STEM content [3]. Engineering and technology provide a hands-on context in which students can test their own scientific knowledge and apply it to the practices of engineering design, which will enhance their higher-order thinking skills for practical problem-solving, improve their understanding/learning achievements in STEM subjects, and foster their interest in STEM as they recognize the interplay between science, engineering, and technology [3,4,20]. Therefore, this study focused on K-12 student outcomes when using integrated STEM approaches, including learning achievements, learning motivation, interest in STEM, and higher-order thinking skills.

Students’ learning achievements represent cognitive performance outcomes, including the knowledge and understanding of students in a STEM program, which can be measured by standardized tests or degrees/certificates [21]. With its interdisciplinary nature, an integrated STEM education offers the opportunity to solve real-world problems more easily by providing a visual and hands-on learning for students [16]. As a result, an integrated STEM education can increase learning achievements, ensure active participation, enable solid and in-depth learning, and ensure meaningful learning [16,22]. In addition, integrated STEM approaches often involve collaboration and teamwork, which can improve students’ learning achievements by allowing for them to learn from their peers and build on each other’s strengths [16].

Students’ learning motivation is defined as the process by which learners’ attention is focused on meeting their educational goals [23]. Students’ learning motivation is expressed through enjoyment, perceptual ability, effort, pressure and perceived usefulness [24]. It is believed that an integrated STEM education links different disciplines and skills and integrates them into a real-world problem; accordingly, it makes the lesson more interesting and different, creates a positive learning environment and makes the learning process fun and active [16,25]. Integrated STEM approaches also often involve hands-on learning experiences through experiments, projects, and other interactive activities. This type of learning can be more motivating than simply reading about a concept in a textbook. Additionally, collaboration and teamwork in STEM activities can be motivating for students who enjoy working with others and obtain a sense of accomplishment by contributing to a group effort [9].

Students’ interest in STEM represents their desire to learn STEM-related content and skills [26,27,28]. One of the most important purposes of a STEM education is to encourage and foster students’ interest in STEM learning and careers [29,30]. More broadly, students’ engagement and retention in STEM is essential to ensuring that 21st-century STEM jobs are filled with skilled workers [31]. For that reason, students’ interest in STEM is an important outcome of the integrated STEM education [32]. When students are exposed earlier to STEM learning experiences, they become more interested in STEM content [26]. It is believed that interdisciplinary connections in integrated STEM education create a positive attitude toward STEM learning among students [16]. Integrated STEM approaches often provide students with interesting and challenging problems; therefore, they may become more interested in learning about STEM fields. Integrated STEM approaches also allow for students to see career opportunities, which may foster students’ interest in pursuing a career in the STEM field [16].

Students’ higher-order thinking skills are a main goal of education in the 21st century, including STEM education [33]. They represent the student’s ability to apply their knowledge, skills and values in reasoning and reflecting to problem-solving, decision-making, innovation and creativity [34]. In Bloom’s Revised Taxonomy, the analysis, evaluation and creation levels of learning are higher-order thinking skills [33]. Higher-order thinking skills can be developed but cannot be automated, and require practice [33]. Integrated STEM education is considered an effective approach to fostering students’ higher-order thinking skills by engaging them in solving engineering challenges and teaching htem to use technology flexibly and creatively like an engineer [34]. These engineering and technology experiences can enhance elements of students’ higher-order thinking, such as problem-solving, critical thinking, creative thinking and scientific thinking [16,34,35]. It is also believed that students’ scientific inquiry/process skills can be fostered by asking questions, making and testing hypotheses, and conducting research like a scientist. [16,34].

Overall, it is believed that integrated STEM approaches are effective in improving many types of student learning outcomes, such as the learning achievements, interest in STEM, learning motivation, and higher-order thinking skills of K-12 students. While there is a considerable body of research on this topic, it is fragmented and lacks a comprehensive synthesis of the available evidence. Therefore, a systematic review is needed to provide a more rigorous and systematic understanding of the relationship between integrated STEM approaches and associated outcomes of K-12 student learning.

2.3. Research Questions

The research question in this study was: What does the existing literature reveal about integrated STEM approaches and associated outcomes of K-12 student learning?

There were six sub-questions, as follows:

(1)

What does the existing literature discuss about the synchronization-based integrated STEM approach and associated outcomes of K-12 student learning?

(2)

What does the existing literature discuss about the thematic-based integrated STEM approach and associated outcomes of K-12 student learning?

(3)

What does the existing literature discuss about the project-based integrated STEM approach and associated outcomes of K-12 student learning?

(4)

What does the existing literature discuss about the cross-curricular-based integrated STEM approach and associated outcomes of K-12 student learning?

(5)

What does the existing literature discuss about the specialized school-based integrated STEM approach and associated outcomes of K-12 student learning?

(6)

What does the existing literature discuss about the community-focused integrated STEM approach and associated outcomes of K-12 student learning?

3. Methodology

3.1. Method

This study was a systematic review that uses the ecological triangulation proposed by Banning [36]. Ecological triangulation is a method for extracting and synthesizing data from the existing knowledge to synthesize the mutually interdependent relationships between behaviors, persons and environments [36,37]. This is used to create an evidence base that requires the synthesis of cumulative and multi-faceted evidence to find ’what approach works for what kind of outcomes for what kind of persons under what kind of conditions’; this is known as the synthesis of ’ecological sentences’ [36,37]. In this study, we aimed to determine ’what integrated STEM approaches work for what types of learning outcomes for what types of students under what educational settings’. The review was conducted by three authors: LHC, NVH and NTL.

We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 Flow Diagram for study selection [38] (Figure 1).

3.2. Information Sources and Search Strategy

We selected Scopus and ERIC as the two main scientific databases in the field of educational research for online searches (Figure 1). Among them, Scopus is considered to be one of the most prestigious academic databases, consisting of only quality journals that fulfill the criteria of being international, peer-reviewed and recognized in the scientific community [39]. In addition, Google Scholar database was used to search for additional studies, focusing on the first 200–300 displayed results [40]. The Web of Science database was not used, due to the lack of access by authors. These searchable databases provided a high likelihood of identifying relevant publications for this systematic review.

Based on the purpose of the study and research questions, we identified keywords used for online searches, including “STEM”, “Outcome”, “Performance”, “Achievement”, “Interest”, “Motivation”, and “Thinking”. Two Boolean operators (AND, OR) were used to combine keywords in the following way: “STEM” AND (“Outcome*” OR “Performance*” OR “Achievement*” OR “Interest*” OR “Motivation*” OR “Thinking*”). In the Scopus database, we entered the keywords in the search field “title, abstract, keywords”, then limited to the subject area: “Social sciences”, document type “Article” or “Conference paper”, keyword “STEM”, Source type “Journal” or “Conference proceeding” and language “English”, excluding the keyword “Higher education”. In the ERIC database, we limited the search to the fields of information “Peer reviewed only”, title “STEM”, abstract “STEM”, descriptor “STEM education”, education level “Elementary education” or “Secondary education”.

3.3. Phases of Study Selection

As shown in Figure 1, the study selection for systematic review was conducted in four phases: identification, screening, eligibility, and included studies.

The first phase was the search in Scopus and ERIC in February 2022. It resulted in the finding of 1433 results in total, with 1166 results in Scopus and 238 results in ERIC. In addition, the first 300 Google Scholar search results were also screened, resulting in 29 additional articles. In the second phase, all identified studies were exported to Mendeley software to check for duplicates, and then 32 duplicate studies were removed. In the third phase, the remaining 1401 articles were screened for eligibility by considering their title, abstract, and keywords. The third phase resulted in 1346 studies that were excluded because the following criteria were not met:

−

Studies were peer-reviewed and published in an academic journal.

−

Studies were published in the English language.

−

Participants were elementary and secondary school students.

−

The type of study was quantitative design. Qualitative studies were excluded because they did not provide clear evidence of the relationship between integrated STEM approaches and associated student learning outcomes.

−

The extracted data were potentially relevant to the research questions.

To avoid bias in the third phase, the two authors NVH and LHC worked together to screen the studies based on the inclusion criteria. In the event of disagreement among authors, a third author, NTL, was invited to the meeting for consultation.

In the fourth phase, a total of 55 potentially eligible studies were retained for full-text screening. A study quality evaluation rubric designed by Margot and Kettler was used to examine the full text of the study in terms of the following aspects: (1) objectives and purposes, (2) literature review, (3) theoretical framework, (4) participants, (5) methods, (6) results and conclusions, and (7) implications [41]. Each of the seven criteria was scored on a four-point scale, where 1 = Does Not Meet Standard, 2 = Nearly Meets Standard, 3 = Meets Standard, and 4 = Exceeds Standard [41]. Articles that had a total score equal to or less than 14 points were excluded [41]. To avoid bias in the fourth phase, both LHC and NVH authors worked independently to score 55 potentially eligible studies. We agreed on 43 out of 55 studies with scores greater than 14 points. Traditionally, we calculated “interrater reliability” using a percent agreement [42]. The calculated agreement rate was 0.78. We then held a meeting to score the remaining 12 studies until we reached an agreement for this work. Finally, we added 2 eligible studies. After assessing study quality, 8 studies were excluded, and 47 studies were retained for inclusion in the review.

3.4. Data Extraction and Synthesis

(i) Data extraction for ecological sentences.

The ecological triangulation approach focuses on interventions, persons, settings/environments and outcomes, and the transactional relationship among these variables [36,37]. Therefore, the data for the above variables were extracted from the included studies.

−

General information: Author(s) and year, study location, and type of design.

−

Interventions: Integrated STEM intervention was used in the study. Based on the nature of integration of the STEM interventions in the extracted data, the articles were classified into the appropriate category of integrated STEM approaches suggested by Rennie et al., including synchronized, thematic, project-based, cross-curricular, school-specialized, or community-focused programs [13].

−

Educational settings: The context in which the intervention was placed.

−

Learning outcomes: Learning achievement, motivation, STEM interest, and higher-order thinking skills.

−

Persons: Participant/student attributes (education/grade level, gender, ethnicity,…)

(ii) Ecological sentence synthesis.

Based on the extracted data, the relationships between integrated STEM approaches and the associated outcomes of student learning were drawn by comparing variables of the same STEM intervention. We observed the learning outcomes that were achieved by the same STEM intervention, and then synthesized a related ecological sentence. Ecological sentences were synthesized based on cumulative and multi-faceted evidence. Ecological sentences can be synthesized and constructed with the pattern: “With Intervention A in setting B, outcomes D occur with persons C (education/grade level, ages, genders, ethnicities…)” [36,37].

To avoid bias in the data extraction, both LHC and NVH authors worked independently, with 47 articles included for data extraction, and then each study was classified into one of six categories of integrated STEM approaches. We agreed on 38 of the 47 studies, which were similarly classified into the integrated STEM approach. The calculated agreement rate was 0.81. Analysis of the remaining 9 disagreement codes helped us to better understand the classification of studies into integrated STEM approaches, and we reached an agreement for this work. Finally, we worked together to refine the extracted text segments for ecological sentences with the variables of interventions, persons, settings/environments, and outcomes.

4. Results

4.1. Data Extraction for Ecological Sentences

We extracted text segments related to author(s), study year and location, type of design, STEM intervention, educational setting, learning outcomes and participant/student attributes. Based on the nature of the integration in STEM interventions, we classified studies into appropriate categories of integrated STEM approach.

Table 1. The matrix of data extraction for ecological sentences.

Authors (Year)/Location/Design.	with This Intervention	in These Settings	These Outcomes Occur	with These Students
Synchronization-based integrated STEM approach (n = 14)
Yoon et al. (2014)/ U.S/Quasi-experimental design [43]	Integrated STE education	Science curriculum	Engineering career interest	Grades 2–4
Chonkaew et al. (2016)/Thailand/Mixed design [44]	Integrated STEM education using problem-based learning	Science curriculum	Analytical thinking and science attitudes	Grade 11
Gülen (2019)/Turkey/Quasi-experimental design [45]	Integrated STEM education using argumentation-based inquiry	Science curriculum	Learning achievement and reflective thinking	Grade 6
Hasançebi et al. (2021)/Turkey/Explanatory sequential design [46]	Integrated STEM education using argumentation-based inquiry	Science curriculum	Learning achievement and reflective thinking	Grade 7
Huri (2019)/Malaysia/Mixed methods [47]	Integrated STEM-lab activities	Science curriculum	Knowledge construction	Grade 9
Hasanah (2020)/Indonesia/Quasi-experimental design [48]	STEM instruction using inquiry-based learning	Physics education	Reasoning skills	Grade 10
Pahrudin et al. (2021)/Indonesia/Quasi-experimental design [49]	STEM instruction using inquiry-based learning	Mathematics and natural sciences curriculum	Critical thinking skills	Grade 10
Khozali (2020)/Malaysia/Mixed method research design [50]	Interdisciplinary Facebook Incorporated STEM Education	Science curriculum	Learning achievement	Grade 9
Seage (2020)/U.S/MANOVA [51]	Blended-learning STEM curriculum using Canvas	Science curriculum	Learning achievement	Grades 3–5
Ültay et al. (2020)/Turkey/Quasi-experimental design [52]	STEM-focused activities using 5E instructional model	Science curriculum	Learning achievements, learning interest and motivation	Grade 3
Tsai et al. (2021)/Taiwan/Experimental design [32]	STEM-focused activities using 5E instructional model	Science curriculum	Learning motivation and interest	Grade 9
Wahyu et al. (2020)/Indonesia/Quasi-experimental design [53]	Mobile augmented reality assisted STEM-based learning	Science curriculum	Scientific achievement	Grade 4
Chang et al. (2021)/Taiwan/Quasi-experimental design [54]	Peer assessment-facilitated STEM	Mathematics curriculum	Learning achievement, higher-order thinking skills	Middle school
Kırkıç (2021)/Turkey/Survey [55]	STEM-based teaching	Technology and Design Curriculum	Learning achievement and STEM attitudes	Grades 7–8
Thematic-based integrated STEM approach (n = 7)
Crotty et al. (2017)/U.S/Mixed design [56]	Integrating engineering in science units	Science curriculum	Learning achievement in engineering	Grades 4–9
Guzey et al. (2019)/U.S/Mixed-methods design [12]	Integrating engineering in science units	Science curriculum	Learning achievement	Middle school
Acar et al. (2018)/Turkey/Quasi-experimental design [25]	Engineering design-based STEM activities	Science and mathematics curriculum	Learning achievement, STEM career interest	Grade 4
Sarican (2018)/Turkey/Quasi-experimental design [57]	Engineering design-based STEM activities	Science curriculum	Learning achievement	Middle school
Kurt (2020)/Turkey/Quasi-experimental design [58]	Engineering design-based STEM activities	Science curriculum	Learning achievement, STEM career interest, and problem-solving skills	Grade 6
Hacioglu (2021)/Turkey/Mixed design [59]	Engineering design-based STEM activities	Science curriculum	Critical thinking skills, STEM perceptions, career awareness	Grade 7
Sarı et al. (2018)/Turkey/Single-group experimental design [60]	Problem-based STEM activities	Science curriculum	Learning motivation, STEM career interest	Grade 5
Project-based integrated STEM approach (n = 10)
Nugent et al. (2010)/U.S/Quasi-experimental design [61]	STEM-oriented robotics course	STEM summer camp	Learning achievement and motivation	Middle school
Barak (2018)/Israel/Experimental design [62]	STEM-oriented robotics course	School classrooms	Learning motivation	Middle school
Han et al. (2015)/U.S/Linear model [63]	STEM project-based learning activities	Mathematics curriculum	Mathematic achievement	High and middle school
Siew (2018)/Malaysia/Quasi-experimental design [64]	STEM project-based learning activities	Science Curriculum	Scientific creativity	Grade 5
English (2019)/Australia/Quantitative design [65]	STEM project-based learning activities	Science curriculum	STEM knowledge	Grades 4
Kartini et al. (2021)/Indonesia/One-group experimental design [66]	STEM project-based learning activities	Science curriculum	Problem-solving skills	Grade 7
Mohr-Schroeder et al. (2014)/U.S/Embedded mixed design [67]	Out-of-school STEM through hands-on project-based learning experiences	STEM summer camp on the college campus	Motivation and interest in STEM fields	Middle school
Shahali et al. (2016)/Malaysia/Quasi-experimental design [68]	Out-of-school STEM through hands-on project-based learning experiences	Bitara-STEM: Science of Smart Communities Program	STEM career interest	Middle school
Mohd Shahali et al. (2019)/Malaysia/Survey and interviews [69]	Out-of-school STEM through hands-on project-based learning experiences	Bitara-STEM: Science of Smart Communities Program	STEM career interest	Middle school
Chittum et al. (2017)/U.S/Survey and Interviews [70]	Out-of-school STEM through hands-on project-based learning experiences	Studio STEM: Engineering design-based science learning environment	STEM career interest	Grades 5-7
Cross-curricular-based integrated STEM approach (n = 12)
Miller et al. (2018)/U.S/Survey [71]	Robotics, science fair, information technology	STEM-related after-school program: STEM competitions	STEM career interest	High school
Allen et al. (2019)/U.S/Survey and observations [72]	State after-school networks across the US	STEM-related after-school program	STEM identity, career interest, critical thinking, and perseverance	Grades 4-12
Stringer et al. (2020)/U.S/Survey [73]	Girls in STEM, Science Olympiad, and Math Counts	STEM-related after-school program: STEM extracurricular programs	STEM career identity and science motivation	Middle school (Girls)
Asigigan (2021)/Turkey/Mixed design [74]	Science Club: Gamified STEM activities	STEM-related after-school program: Science Club	Critical thinking	Grades 3–4
Hite (2021)/U.S/Experimental single case study [75]	Robotics, Science Olympiad, Girls Who Code, ...	STEM-related after-school program	STEM interest and motivation	Middle school
Gilliam et al. (2017)/U.S/Interviews and survey [76]	Alternate Reality Games: The Source	STEM summer camp	STEM interest	High School
Kitchen et al. (2018)/U.S/Survey [77]	College-and university-run STEM activities	STEM summer camp	STEM career interest	High school
Baran et al. (2019)/Turkey/Survey and Interviews [78]	Hands-on STEM activities	University	STEM interest	Grade 6
Saw et al. (2019)/U.S/Multiple regression [79]	Hands-on STEM activities	University	Interest in math and math-related careers	Grade 8
Parker et al. (2020)/U.S/Survey [80]	Hands-on STEM activities	University	Interest in science and engineering	Grades 3–5
Ng (2021)/Hong Kong/Survey [81]	Hands-on STEM activities	University	Learning motivation	Middle school
Wang et al. (2021)/China/Survey [82]	Informal STEM learning experiences	Informal STEM-related programs	STEM interest	Grade 10
Specialized school-based integrated STEM approach (n = 2)
Alemdar et al. (2018)/U.S/Mixed-methods design [83]	Engineering courses	Applied STEM courses (career and technical education programs)	Science and mathematic achievement, STEM interest	Grades 6-8
Plasman (2018)/U.S/Survey [84]	Information Technology, and Scientific Research and Engineering courses	Applied STEM courses (career and technical education programs)	Mathematic achievement and STEM interest	Grade 10
Community-focused integrated STEM approach (n = 2)
Collins et al. (2020)/U.S/Observations and survey [85]	STEM service-learning experiences	STEM summer program	Learning motivation and STEM career interest	High school
Benek (2021)/Turkey/Nested mixed design [86]	Socio-scientific STEM activities	Science curriculum	21st century skills	Middle school

presented a matrix of the extracted data for ecological sentences and study classification.

In Table 1, the publication years of the studies ranged from 2010 to 2022, of which 38 studies (76%) were published from 2018 to 2021. There were 31 studies using quantitative methods (experimental or survey) and 16 using mixed methods. Studies were conducted in Europe and North America: 19 studies (US: 18 studies; UK: 1 study); the Asia Pacific: 14 studies (Thailand: 1 study; China: 1 study; Malaysia: 5 studies; Indonesia: 4 studies; Taiwan: 2 studies; Hong Kong: 1 study); Turkey: 12 studies; Israel: 1 study; and Australia: 1 study. The United States and Turkey had the highest number of studies included in the review.

All 47 included studies were classified into six categories of integrated STEM approach: synchronized (n = 14), thematic (n = 7), project-based (n = 10), cross-curricular (n = 12), school-specialized (n = 2), or community-focused (n = 2). For each included study, we extracted a single ecological sentence. For example, from the study by Chonkaew et al. [44], we extracted the following ecological sentence: “With integrated STEM education using problem-based learning in the science curriculum, analytical thinking and science attitudes occur with 11th-grade students”. Within each category of the integrated STEM approach, studies with the same STEM intervention were placed side by side to facilitate the subsequent ecological sentence synthesis.

4.2. Ecological Sentence Synthesis

Ecological sentence synthesis involves examining the relationship between extracted data to observe whether the same STEM intervention produces the same learning outcomes. The extracted data regarding STEM interventions and student learning outcomes can be analyzed for convergence, complementarity, or divergence. Convergence refers to a strong degree of overlap in the same STEM intervention, producing the same outcomes. Complementarity builds a richer picture of learning outcomes by allowing for outcomes from different studies to inform each other under the same STEM intervention. Divergence reveals whether STEM interventions or outcomes are flawed, whichh could be see” as a knowledge gap that needs further investigation. Based on the extracted data in Table 1, we observed the learning outcomes that occurred following the same STEM intervention, and then synthesized a related ecological sentence. If we observed a STEM intervention standing alone, the information in the ‘Type of synthesis’ column was coded as ‘Not applicable’, and a single ecological sentence was developed. The results of the ecological sentence synthesis were presented in

Table 2. The ecological sentence synthesis.

Studies	Type of Synthesis	Related Ecological Sentence
Synchronization-based integrated STEM approach
Yoon et al. (2014) [43]	Not applicable	With integrated STE education in the science curriculum, engineering career interest occurs with elementary school students [43].
Chonkaew et al. (2016) [44]	Not applicable	With integrated STEM education using problem-based learning in the science curriculum, analytical thinking and science attitudes occur with high school students [44].
Gülen (2019) [45]; Hasançebi et al. (2021) [46]	Convergence	With integrated STEM using argumentation-based inquiry in the science curriculum, learning achievement and reflective thinking occur with middle school students [45,46]
Huri (2019) [47]	Not applicable	With integrated STEM-lab activities in the science curriculum, knowledge construction occurs with middle school students [47].
Hasanah (2020) [48]; Pahrudin et al. (2021) [49]	Convergence	With STEM instruction using inquiry-based learning in the mathematics and natural sciences curriculum, higher-order thinking skills (reasoning skills and critical thinking skills) occur with high school students [48,49].
Khozali (2020) [50]	Not applicable	With interdisciplinary facebook incorporated STEM education in the science curriculum, learning achievement occurs with middle school students [50].
Seage (2020) [51]	Not applicable	With blended-learning STEM curriculum using Canvas in the science curriculum, learning achievement occurs with elementary school students from low socioeconomic areas [51].
Ültay et al. (2020) [52]; Tsai et al. (2021) [32]	Complemen-tarity	With STEM-focused activities using 5E instructional model in the science curriculum, learning achievements [52], learning interest and motivation [32,52] occur with elementary and middle school students.
Wahyu et al. (2020) [53]	Not applicable	With mobile augmented reality assisted STEM-based learning in the science curriculum, scientific achievement occurs with elementary school students [53].
Chang et al. (2021) [54]	Not applicable	With peer assessment-facilitated STEM in the mathematics curriculum, learning achievement and higher-order thinking skills occur with middle school students [54].
Kırkıç (2021) [55]	Not applicable	With STEM-based teaching in the technology and design curriculum, learning achievement and STEM attitudes occur with middle school students [55].
Thematic-based integrated STEM approach
Crotty et al. (2017) [56]; Guzey et al. (2019) [12]	Convergence	With integrating engineering design challenge in science units to provide learning context in the science curriculum, learning achievements in science and engineering occur with elementary and middle school students [12,56].
Acar et al. (2018) [25]; Sarican (2018) [57]; Kurt (2020) [58]; Hacioglu (2021) [59]	Convergence and complemen-tarity	With engineering design-based STEM activities in the science and mathematics curriculum, learning achievement, STEM career interest and higher-order thinking skills (problem solving skills and critical thinking skills) occur with elementary and middle school students [25,57,58,59].
Sarı et al. (2018) [60]	Not applicable	With problem-based STEM activities in the science curriculum, learning motivation and STEM career interest occur with elementary school students [60].
Project-based integrated STEM approach
Nugent et al. (2010) [61]; Barak (2018) [62]	Convergence	With STEM-oriented robotics course in the school classroom and STEM summer camp, learning achievement and motivation occur with middle school students [61,62].
Han et al. (2015) [63]; Siew (2018) [64]; English (2019) [65]; Kartini et al. (2021) [66]	Convergence and complemen-tarity	With STEM project-based learning activities in the mathematics and science curriculum, learning achievement (mathematic achievement and STEM knowledge) [63,65] and higher-order thinking skills (scientific creativity, problem-solving skills) [64,66] occur with K-12 students.
Mohr-Schroeder et al. (2014) [67]; Shahali et al. (2016) [68]; Mohd Shahali et al. (2019) [69]; Chittum et al. (2017) [70]	Convergence	With Out-of-school STEM through hands-on project-based learning experiences in the STEM summer camp on college campus, Bitara-STEM and Studio STEM, STEM career interest occurs with middle school students [67,68,69,70].
Cross-curricular-based integrated STEM approach
Miller et al. (2018) [71]; Allen et al. (2019) [72]; Stringer et al. (2020) [73]; Asigigan (2021) [74]; Hite (2021) [75].	Convergence and complemen-tarity	With STEM-related Robotics, Mathematics Contest, Science Olympiad, Information Technology, Girls in STEM, Gamified STEM activities,… in the STEM related after-school program (STEM competitions, STEM extracurricular and science club), STEM interest and motivation [73,75], STEM career interest [71,72,73], critical thinking [72,74] occur with K-12 students.
Gilliam et al. (2017) [76]; Kitchen et al. (2018) [77]	Convergence	With STEM-related Robotics, Alternate Reality Games (The Source) and College-and university-run STEM activities in the STEM summer camp, STEM interest and related career occur with high school students [76,77].
Baran et al. (2019) [78]; Saw et al. (2019) [79]; Parker et al. (2020) [80]; Ng (2021) [81]	Complemen-tarity	With hands-on STEM activities at university, STEM interest and related careers [78,79,80], and learning motivation [81] occur with elementary and middle school students.
Specialized school-based integrated STEM approach
Alemdar et al. (2018) [83]; Plasman (2018) [84]	Convergence	With Engineering courses, Information Technology, Scientific Research and Engineering courses in the career and technical education program, science and mathematic achievement, and STEM interest occur with middle and high school students [83,84].
Community-focused integrated STEM approach
Collins et al. (2020) [85]	Not applicable	With STEM service-learning experiences in the STEM summer program, learning motivation and STEM career interest occur with high school students [85].
Benek (2021) [86]	Not applicable	With Socio-scientific STEM activities in the science curriculum, 21st century skills occur with middle school students [86].

.

In Table 2, a total of 23 ecological sentences are shown. These were developed from existing studies, consisting of 11 single ecological sentences and 12 ecological sentences with cumulative and multifaceted evidence. No divergence was observed in ecological sentences, indicating that student learning outcomes occurred consistently within the same STEM intervention. The eleven ecological sentences of the synchronization-based integrated STEM approach indicated a wide variety of integrated STEM activities applied in the science curriculum. The two single ecological sentences of the community-focused integrated STEM approach also revealed that the learning outcomes of middle and high school students differed between STEM service-learning and socio-scientific STEM activities. In the thematic-based integrated STEM approach, engineering-design-based STEM activities in the science curriculum and associated learning achievements of elementary and middle school students showed the most prominent relationship. In the project-based integrated STEM approach, three ecological sentences were synthesized from ten studies. This revealed that STEM project-based learning activities in the science curriculum focused on improving students’ learning achievement and higher-order thinking skills, while out-of-school STEM project-based learning activities focused solely on students’ STEM career interests. In the cross-curricular-based integrated STEM approach, three ecological sentences were synthesized from twelve studies. The results showed that students’ learning achievements were absent from the integrated STEM activities. Finally, in the specialized school-based integrated STEM approach, a synthesized ecological sentence revealed that career and technical education programs were effective in improving the learning achievements and STEM career interest of high school students.

5. Discussions

5.1. Synchronization-Based Integrated STEM Approach and Associated Outcomes of K-12 Student Learning

In the synchronization-based integrated STEM approach (see Table 2), 13 studies discussed the integrated STEM approaches in the context of science education, and one study focused on a survey in the Technology and Design course. The findings from existing studies indicated that integrated STEM activities were effective in improving student learning achievements in science education, including integrated STEM education using argumentation-based inquiry [45,46], integrated STEM lab [47], interdisciplinary Facebook-incorporated STEM education [50], blended-learning STEM curriculum [51], STEM activities based on the 5E model [32,52], mobile augmented-reality-assisted STEM-based learning [53], and peer-assessment-facilitated STEM [54]. STEM activities based on the 5E model were effective in improving students’ motivation in science education [32,52]. In addition, STEM activities fostered students’ interest in STEM and related careers in science education, including integrated STE education [43], integrated STEM activities using problem-based learning [44], and STEM activities using the 5E instructional model [32,52]. Finally, STEM activities were found to improve students’ higher-order thinking skills in science education, including STEM activities using problem-based learning for improved analytical thinking skills [44], integrated STEM education using argumentation-based inquiry for improved reflective thinking skills [45,46], STEM instruction using inquiry-based learning for improved reasoning and critical thinking skills [48,49], and peer-assessment-facilitated STEM for improved problem solving and critical thinking skill [54]. In short, a wide variety of integrated STEM activities were applied in the science curriculum, and their associated outcomes regarding student learning differed. Therefore, STEM teachers are recommended to use the synchronization-based integrated STEM approach to reform science teaching and improve student learning achievements in science education. However, there is still a lack of studies examining the relationship between other STEM activities and students’ motivation, STEM interest, and higher-order thinking skills in science education, including: integrated STEM lab, Facebook-incorporated STEM, blended-learning STEM curriculum, mobile augmented-reality-assisted STEM-based learning, and peer-assessment-facilitated STEM. The synchronization-based integrated STEM approach and associated outcomes of K-12 student learning have also not been investigated in the context of other subjects, such as math and technology courses. These are seen as a knowledge gap that should be investigated further.

5.2. Thematic-Based Integrated STEM Approach and Associated Outcomes of K-12 Student Learning

In the thematic-based integrated STEM approach (see Table 2), all seven studies discussed STEM activities in the context of science curriculum for K-12 students. The findings from existing studies indicate that integrating engineering design challenges in science units to provide a learning context in the science curriculum is effective in improving the science and engineering learning achievements of elementary and middle school students [12,56]. Engineering-design-based STEM activities in the science and mathematics curriculum were found to be effective in improving the learning achievements, STEM career interest and higher-order thinking skills (problem-solving skills and critical thinking skills) of elementary and middle school students [25,57,58,59]. Finally, problem-based STEM activities in the science curriculum were effective in improving the learning motivation and STEM career interest of elementary school students [60]. In short, ecological sentences of the thematic-based integrated STEM approach showed the convergence and complementarity of the assertion that engineering design-based STEM activities were most effective at improving student learning outcomes in science education. However, examining the thematic-based integrated STEM approach and associated outcomes of K-12 student learning also revealed knowledge gaps that should be further investigated. In the first knowledge gap, the thematic-based integrated STEM approach and associated outcomes of student learning have not been investigated in the context of other subjects, such as math and technology courses. Future studies should also explore the relationship between engineering-design-based STEM activities and the learning motivation of K-12 students.

5.3. Project-Based Integrated STEM Approach and Associated Outcomes of K-12 Student Learning

In the project-based integrated STEM approach (see Table 2), five studies involved project-based STEM activities during school time, and the remaining five studies involved after-school STEM activities. We identified three integrated STEM activities that emerged during the synthesis of the extracted data. Firstly, STEM-oriented robotics courses in the school classroom and STEM summer camp were effective in improving the learning achievements and motivation of middle school students [61,62]. Secondly, STEM project-based learning activities in the mathematics and science curriculum were effective in increasing the learning achievements (mathematic achievement and STEM knowledge) [63,65] and higher-order thinking skills (scientific creativity, problem-solving skills) [64,66] of elementary and secondary school students. Finally, out-of-school STEM through hands-on project-based learning experiences in the STEM summer camps on college campuses, Bitara-STEM and Studio STEM were effective in improving the STEM career interest of middle school students [67,68,69,70]. Additionally, we identified several topics that should be further investigated. The first is the lack of studies examining the relationship between out-of-school STEM project-based learning activities and associated outcomes of student learning, including learning achievements, motivation, and higher-order thinking skills. Secondly, students’ STEM interest and higher-order thinking skills have also not been investigated in STEM-oriented robotics courses.

5.4. Cross-Curricular-Based Integrated STEM Approach and Associated Outcomes of K-12 Student Learning

In the cross-curricular-based integrated STEM approach (see Table 2), all 12 studies discussed STEM interventions in the context of after-school and out-of-school settings, but none discussed STEM interventions in subject education. We identified three integrated STEM activities that emerged during the synthesis of the extracted data. Firstly, STEM-related robotics, mathematics contests, science olympiad, information technology, girls in STEM projects, gamified STEM activities, etc., in STEM-related after-school programs (STEM competitions, STEM extracurricular activities and science clubs) were effective in increasing interest in and motivation regarding STEM [73,75], increasing STEM career interest [71,72,73], and improving the critical thinking skills [72,74] of elementary and secondary school students. Secondly, STEM-related robotics, alternate reality games (Game: The Source) and college-and university-run STEM activities in STEM summer camps were effective in inceasing interest in STEM and STEM-related careers in high school students [76,77]. Finally, hands-on STEM activities at universities increased interest in STEM and STEM-related careers [78,79,80] and the learning motivation [81] of elementary and middle school students. In summary, cross-curricular-based integrated STEM activities were effective for the development of higher-order thinking skills and in preventing the decline in students’ motivation and STEM career interest. However, whether students’ make learning achievements using cross-curricular-based integrated STEM programs should also be investigated.

5.5. Specialized School-Based Integrated STEM Approach and Associated Outcomes of K-12 Student Learning

The specialized school-based integrated STEM approach focused on technical and career education programs in high school, such as applied STEM courses, information technology, science and engineering studies (Table 2). These courses are beneficial for improving learning achievements in science and math, as well as middle and high school students’ interest in STEM [83,84]. A reason for this may be that engineering courses require students to actively use foundational math and science practices, which can lead to increased engagement, self-efficacy, persistence, and achievements in STEM [83]. Engineering design challenges are seen as vehicles through which students can strengthen and deepen their general STEM foundation and develop habits of thought and action in using math and science in engineering. In short, the practice of knowledge and skills related to science and math in the context of middle school engineering classes has significant benefits in terms of both interest in STEM and the learning achievement of students. However, not many studies have focused on on the specialized school-based integrated STEM approach.

5.6. Community-Focused Integrated STEM Approach and Associated Outcomes of K-12 Student Learning

In the community-focused integrated STEM approach (see Table 2), STEM service-learning experiences in the STEM summer program were effective in improving the motivation to learn and STEM career interest of high school students [85]. Service-learning provides students with opportunities to see the value of their work in everyday life, thereby increasing underrepresented students’ engagement with STEM and potentially motivating them to pursue STEM careers [85]. Additionally, socio-scientific STEM activities in the science curriculum were effective in improving the 21st-century skills of middle school students [86]. The reason for this may be that the technical designs of STEM activities are applied to solve socio-scientific issue topics, such as wind energy, global warming, and space pollution, helping students to apply scientific and ethical reasoning to controversial social issues related to science, thereby promoting students’ higher-order thinking skills, such as critical thinking, problem-solving, and creativity. Similar to the situation of specialized school-based integrated STEM approach, there were not many studies on the community-focused integrated STEM approach.

6. Conclusions

This study was the first systematic review using ecological triangulation to examine the integrated STEM approaches and associated outcomes of K-12 student learning. Its purpose was to determine ‘what integrated STEM approaches work for what types of learning outcomes for what types of students under what educational settings’. A total of 23 ecological sentences were developed from existing studies. The findings of the study revealed that no divergence was observed in ecological sentences. This means that student learning outcomes occurred consistently within the same STEM activity. The findings also revealed that the associated outcomes of K-12 student learning differed among the integrated STEM approaches. The synchronization-based integrated STEM approach encompassed a wide range of integrated STEM activities applied to the science curriculum, resulting in different learning outcomes for students. The thematic-based integrated STEM approach showed the most prominent relationship between engineering-design-based STEM activities in the science curriculum and the associated learning achievements of elementary and middle school students. In the project-based integrated STEM approach, STEM project-based learning activities in the science curriculum focused on improving students’ learning achievements and higher-order thinking skills, while out-of-school STEM project-based learning activities focused solely on students’ STEM career interests. In the cross-curricular-based integrated STEM approach, students’ learning achievements were absent from the integrated STEM activities. In addition, there was a dearth of studies on integrated STEM activities and the associated student learning outcomes when using the specialized school-based and community-focused integrated STEM approaches.

6.1. Recommendations for Practice

This study reviewed existing studies to observe the integrated STEM approaches and associated outcomes of K-12 student learning. Based on the research findings, two recommendations for educational practice were proposed, one for STEM teachers and one for school leaders. Firstly, the main goal of STEM teachers in integrated STEM-based instruction is to improve the learning outcomes of their students. It is important to note that different STEM approaches can produce different student learning outcomes. Therefore, STEM teachers should carefully consider the goals of their STEM program and select an approach that aligns with those goals and meets the needs of their students. In addition, school leaders should understand the relationship between integrated STEM approaches and associated outcomes of K-12 student learning to make decisions about STEM implementation in their schools.

6.2. Recommendations for Further Research

In this systematic review, we observed several directions for future research. Firstly, we observed that the synchronization-based integrated STEM approach and thematic-based integrated STEM approach were not implemented in mathematics, engineering, and technology. Secondly, a future study should focus on examining whether students’ STEM interests and higher-order skills are developed in hands-on project-based learning activities with STEM-related robotics. Another study should examine whether student learning achievements are reached in cross-curricular-based integrated STEM activities. Finally, future studies should also examine the K-12 student learning outcomes that occur in the specialized school-based integrated STEM approach and community-focused integrated STEM approach. Such studies will validate and further expand the relationship between integrated STEM approaches and associated outcomes of K-12 student learning.