A Systematic Review on Design Thinking Integrated Learning in K-12 Education

: Design thinking is regarded as an essential way to cultivate 21st century competency and there has been a concomitant rise of needs and interest in introducing K-12 students to design thinking. This study aimed to review high-qualiﬁed empirical studies on design thinking integrated learning (DTIL) in K-12 education and explore its future research perspectives. After a systematic search in online database via a keyword search and snowballing approach, 43 SSCI journal papers with 44 studies were included in this review. The results indicate that: (1) There has been a growing popularity of integrating design thinking into K-12 education over the past decade, and most empirical studies target middle school students with small group size and a short period; (2) Studies tend to pay more attention to STEM related curriculum domains by incorporating non-uniﬁed design thinking models or processes, and the core concepts of design thinking in K-12 education have been frequently valued and pursued including prototype, ideate, deﬁne, test, explore, empathize, evaluate, and optimize; (3) The mostly evaluated learning performances are design thinking, followed by emotional/social aspect, subject learning performance and skill. For evaluation, qualitative assessments are used more frequently with instruments like survey/questionnaire, portfolio, interview, observation, protocol analysis, etc. (4) interventions with non-experimental study, formal classroom setting, collaborative learning, and traditional tools or materials have been mainly applied to the open-ended and challenging activities in real situated DTIL. Overall, the 43 papers suggest that design thinking shows great educational potential in K-12 education, however, the empirical evidence that supports the effectiveness of DTIL is still rather limited. Research gaps and future directions derived from reviewed papers are also discussed.


Introduction
Originating from design field, design thinking has attracted considerable interest from practitioners and academics alike, as it offers a novel approach to innovation and problemsolving [1,2]. In the artificial world, design plays an indispensable role in the progress of human society by realizing "the transformation of existing conditions into preferred ones" [3]. In the field of design, design thinking is associated with the understanding of expertise in design, such as what constitutes expertise in design, and how to assist novice students to gain that expertise so that they can become expert and outstanding designers [4]. According to Jonassen's typology [5], design problems are usually among the most complex and ill-structured kinds of problems that are encountered in practice. Therefore, with their creative ways of solving problems, expert designers are seen as one group of innovative problem solvers, then they and their thinking have something important to offer for wider areas [1].
As an innovative problem-solving method, the concept of design thinking has gradually expanded from a professional concept to a more general one. Nowadays, "Design Thinking" is identified as an exciting new paradigm for dealing with problems in many

Method
In order to conduct a reliable systematic review on this topic, we followed the threestage guideline provided by Tranfield et al. [24]: stage I-planning the review, stage IIconducting a review, stage III-reporting and dissemination. Additionally, in stage II, we mainly used the snowball method [25] to select relevant studies.
Under stage I, as discussed in the previous section, we identified the need and goal of this review based on the scoping study surrounding the field.
Under stage II, to select useful and high-quality papers, we performed a keyword search in an online bibliographic database ISI Web of Science. The main reasons for selecting this database were: (1) to retrieve authoritative research articles that were of sufficiently high quality for analysis, (2) to draw more representative and reliable conclusions of this systematic literature review.
After the initial search, we utilized a snowball method [25] using the references in the selected articles before 14 March 2022 to search more literature. A snowballing approach refers to using the reference list of a paper or the citations to the paper to identify additional papers for a literature review [25]. The following inclusion criteria were used to select papers: (a) Full peer-reviewed English paper published in SSCI journals. (b) Empirical study conducted for K-12 students, the term "empirical study", which means either quantitative or qualitative, and not a literature review, framework, or proposal. (c) Papers involved in DTIL (using "design thinking" in any part of the paper, such as title, abstract, keywords, or main text). (d) The DTIL featured in the study targets students rather than pre-service/in-service teachers.

Keyword Search
Considering searching for relevant and mainstream papers that lie within the scope of this study, we used representative search terms to capture articles in the field. The keywords consisted of two clusters: design thinking and K-12. We adopted the keywords such as K-12, middle school, high school, and children, etc., to ensure that the target age levels of this study could be obtained. Initially, we used the search string "design thinking" AND (K-12 OR K12 OR middle school OR high school OR secondary school OR primary school OR elementary school OR kindergarten OR preschool OR children OR child) to identify SSCI journal papers in the ISI Web of Science. The results returned 99 papers. We quickly analysed the titles, abstracts, and methods sections of these papers with reference to the above four inclusion criteria. As a result, 27 papers were left.

Snowball Approach
After finishing the keyword search, we employed the first-round snowballing approach with the 27 papers as seeds to search papers in the ISI Web of Science. A total of 1485 references from backward snowballing and 236 citations from forward snowballing (a total of 1721 papers) were found. Using the above four inclusion criteria, 12 new papers were selected.
In the second round, 12 papers were identified as new seeds for the snowballing approach from the results of the aforementioned searches. A total of 703 references from backward snowballing and 123 citations from forward snowballing (a total of 826 papers) were examined. As a result, three new papers were singled out.
We then launched the third-round snowballing approach. This time three papers were used as new seeds and we retrieved 186 papers (173 references from backward snowballing and 13 citations from forward snowballing). Only one new paper was selected for review.
At the end, the fourth-round of snowballing was conducted, and no more papers were found, thus the iterative process of snowballing approach could be ended.
After four rounds of the snowballing approach, as depicted in Table 1, 43 papers were selected as samples for the subsequent literature review. Under stage III, to compare the features of the sample papers, a coding form was created (see Tables A1 and A2 in Appendix A). To address the above four research questions, information on the author, publishing year, educational level, sample size, duration, subject, and the implemented design thinking model is presented in Table A1. Table A2 summarizes each study's research design, course type, group size, design task/challenge, learning tools/materials, dependent variable, and evaluation instrument. The coding of the papers was initially done by one author, another author was responsible for checking, and the two authors studied together in ambiguous cases. Finally, the coding was analysed to draw conclusions.

Distribution of Articles on Design Thinking over Time
The yearly publication distribution of the 43 articles on empirical studies on design thinking illustrated in Figure 1 reveals an upward trend from 2010 till now, consistent with the growing popularity and importance of design thinking [2]. Although we did not limit the publication period of papers in the literature retrieval process, in the K-12 education context, design thinking has attracted increasing attention since about ten years ago. Specifically, only 2 (4.7%) of the 43 papers were published before 2015, and related research has increased year by year. Especially, after 2017, it shows a steady upward trend, and subsequent years' publications peaked in 2021 (12 or 27.9%). Appl. Sci. 2022, 12, x FOR PEER REVIEW 5 of 34

Sample Group Level, Size, and Duration
Considering one paper included two experiments, we identified a total of 44 studies from the 43 papers. Figure 2 shows the results for the different categories of educational levels. Researchers conducted studies across various educational levels. In total, design thinking in the middle school setting accounted for the largest proportion (23 or 52.3%), or more than half of the reviewed studies. There were 10 (22.7%) studies for elementary school students, seven (15.9%) studies for high school students, two (4.5%) for kindergarten children, and two (4.5%) studies for mixed level students; one is mixed with kindergarten and primary school students, another is middle and high school across categories. Among the 44 empirical studies, 43 mentioned the number of participants. Nineteen (44.2%) of the papers were for studies that recruited less than 40 participants (see Figure  3). Only 11 (25.6%) of the papers involved more than 100 participants. This indicates that the sample sizes were not large in the K-12 educational research of design thinking. Promisingly, three papers distanced themselves from a small sample size: [26] surveyed 613 high school students who experienced a 16-week engineering design training course; [27] collected data from 576 children that participated in STEM Learning via 3D technologyenhanced makerspaces; [28] analysed data from 350 eighth and ninth graders.

Sample Group Level, Size, and Duration
Considering one paper included two experiments, we identified a total of 44 studies from the 43 papers. Figure 2 shows the results for the different categories of educational levels. Researchers conducted studies across various educational levels. In total, design thinking in the middle school setting accounted for the largest proportion (23 or 52.3%), or more than half of the reviewed studies. There were 10 (22.7%) studies for elementary school students, seven (15.9%) studies for high school students, two (4.5%) for kindergarten children, and two (4.5%) studies for mixed level students; one is mixed with kindergarten and primary school students, another is middle and high school across categories.

Sample Group Level, Size, and Duration
Considering one paper included two experiments, we identified a total of 44 studies from the 43 papers. Figure 2 shows the results for the different categories of educational levels. Researchers conducted studies across various educational levels. In total, design thinking in the middle school setting accounted for the largest proportion (23 or 52.3%), or more than half of the reviewed studies. There were 10 (22.7%) studies for elementary school students, seven (15.9%) studies for high school students, two (4.5%) for kindergarten children, and two (4.5%) studies for mixed level students; one is mixed with kindergarten and primary school students, another is middle and high school across categories. Among the 44 empirical studies, 43 mentioned the number of participants. Nineteen (44.2%) of the papers were for studies that recruited less than 40 participants (see Figure  3). Only 11 (25.6%) of the papers involved more than 100 participants. This indicates that the sample sizes were not large in the K-12 educational research of design thinking. Promisingly, three papers distanced themselves from a small sample size: [26] surveyed 613 high school students who experienced a 16-week engineering design training course; [27] collected data from 576 children that participated in STEM Learning via 3D technologyenhanced makerspaces; [28] analysed data from 350 eighth and ninth graders.  Among the 44 empirical studies, 43 mentioned the number of participants. Nineteen (44.2%) of the papers were for studies that recruited less than 40 participants (see Figure 3). Only 11 (25.6%) of the papers involved more than 100 participants. This indicates that the sample sizes were not large in the K-12 educational research of design thinking. Promisingly, three papers distanced themselves from a small sample size: [26] surveyed 613 high school students who experienced a 16-week engineering design training course; [27] collected data from 576 children that participated in STEM Learning via 3D technology-enhanced makerspaces; [28] analysed data from 350 eighth and ninth graders.  Among the 44 studies, 38 reported duration of the research on design thinking. As can be seen in Figure 4, the largest proportion of studies (9 or 23.7%) were conducted over a duration less than one month. While the second largest proportion of studies (8 or 21.1%) lasted for less than 2 months. Besides, five (13.2%) of the studies were one-off tasks, which lasted for less than 1 day. Only four studies were conducted for more than 6 months. Overall, the duration of most studies was comparatively short (i.e., typically for one semester).

Distribution of Studies on Curriculums
We examined the curriculum domains of the reviewed studies. Overall, there were four main categories of curriculum domains involved in the research, including STEMrelated, design-related, non-STEM multidisciplinary and other curriculums. As shown in Figure 5, among the 44 studies, 41 reported the subjects or curriculums, and science (11 or 26.8%) was the subject/curriculum matter most often researched, followed by Among the 44 studies, 38 reported duration of the research on design thinking. As can be seen in Figure 4, the largest proportion of studies (9 or 23.7%) were conducted over a duration less than one month. While the second largest proportion of studies (8 or 21.1%) lasted for less than 2 months. Besides, five (13.2%) of the studies were one-off tasks, which lasted for less than 1 day. Only four studies were conducted for more than 6 months. Overall, the duration of most studies was comparatively short (i.e., typically for one semester).  Among the 44 studies, 38 reported duration of the research on design thinking. As can be seen in Figure 4, the largest proportion of studies (9 or 23.7%) were conducted over a duration less than one month. While the second largest proportion of studies (8 or 21.1%) lasted for less than 2 months. Besides, five (13.2%) of the studies were one-off tasks, which lasted for less than 1 day. Only four studies were conducted for more than 6 months. Overall, the duration of most studies was comparatively short (i.e., typically for one semester).

Distribution of Studies on Curriculums
We examined the curriculum domains of the reviewed studies. Overall, there were four main categories of curriculum domains involved in the research, including STEMrelated, design-related, non-STEM multidisciplinary and other curriculums. As shown in Figure 5, among the 44 studies, 41 reported the subjects or curriculums, and science (11 or 25.6%) was the subject/curriculum matter most often researched, followed by  We examined the curriculum domains of the reviewed studies. Overall, there were four main categories of curriculum domains involved in the research, including STEM-related, design-related, non-STEM multidisciplinary and other curriculums. As shown in Figure 5, among the 44 studies, 41 reported the subjects or curriculums, and science (11 or 26.8%) was the subject/curriculum matter most often researched, followed by STEM/SETAM in eight (19.5%) studies, and engineering in seven (17.1%) studies. Five (12.2%) studies covered multiple disciplines (non-STEM), three (7.3%) studies mentioned design related subjects/curriculums, such as design, design and research, design, and technology. The technology curriculum also had three (7.3%) studies. And the number for the remain- ing four curriculums was one (2.4%), including science and engineering, technology and engineering, robotics, and geography. Overall, subjects or curriculums related to STEM, such as Technology and engineering (1), technology (3), STEM/STEAM (8), science and engineering (1), science (11), robotics (1), engineering (7), accounted for the largest proportion (32 or 78.0%). In other words, in the K-12 educational context, design thinking was mostly integrated with STEM/SETAM subjects or curriculums. To compare the application of subjects by different educational levels, the number of empirical papers is shown in Figure 6. This shows that engineering was most popular in the high school level, while science was most popular in the middle school and elementary school levels. Engineering and STEM were applied in kindergarten. STEM/STEAM was applied in the two mixed level papers.  (7), accounted for the largest proportion (32 or 78.0%). In other words, in the K-12 educational context, design thinking was mostly integrated with STEM/SETAM subjects or curriculums. To compare the application of subjects by different educational levels, the number of empirical papers is shown in Figure 6. This shows that engineering was most popular in the high school level, while science was most popular in the middle school and elementary school levels. Engineering and STEM were applied in kindergarten. STEM/STEAM was applied in the two mixed level papers.

Design Thinking Model Implemented in the Empirical Studies
Among the 44 studies, 37 mentioned the design or design thinking phases or elements. The authors described design thinking in different ways and associated a variety of attributes with the concept, and different terms and sequences of action were also employed [2]. Thus, models varied in the sample of empirical studies. For example, the typical five-stage model proposed by Stanford was adapted to integrate within the K-12 educational context (7 or 18.9%), see for example [29][30][31][32][33]. Some studies drew on other design thinking models, such as the model proposed by IDEO, see for example [27].
There also existed varied vocabulary in expression of the models [11]. For example, prototype was expressed in words as "modelling", "making", "create", "producing", "building". Ideate was also described by "brainstorming", "develop possible solutions", "generate design ideas", "product ideation", etc. Define also meant "clarify problems and constraints", "identify problem", "understanding and defining the problem", "Understand" etc. It was not easy to identify all the models that applied. Despite this, some elements among these were employed more regularly, suggesting a level of concurrence [2]. Therefore, we summarized the mostly mentioned design thinking competences in the selected studies, including prototype, ideate, define, test, explore, empathize, evaluate, and optimize. Explanation and codes are also listed (see Table 2).  Table 3. The measured competences mainly covered four aspects of students' learning in the DTIL context: subject learning performance, design thinking, emotional/social aspect and skill.
Among these dependent variables assessed (see Figure 7), 31 (or 70.5%) studies tested the design thinking, 17 (or 38.6%) discussed the emotional/social aspect, nine evaluated the subject learning performance, and eight tested skills (or 18.2%).

Evaluation Instruments
Despite the variability in students' learning performance assessed for DTIL, multiple measurement instruments emerged from the literature, including the qualitative and quantitative assessment types. The results of each instrument were revealed in Figure 8. Among the 44 studies, 11 (25.0%) just adapted one kind of measurement instrument. Some employed more than one assessment instrument to collect multidimensional evidence of students' learning performances. Thirteen (29.5%) used two measurement instruments, and twenty (45.5%) combined three or more evaluation methods.
For the measurement instruments, 23 (52.3%) of the studies adopted a survey/questionnaire for measuring the students' learning performances in the design thinking integrated context, which is the one with the largest proportion (see Figure 8). The second-

Evaluation Instruments
Despite the variability in students' learning performance assessed for DTIL, multiple measurement instruments emerged from the literature, including the qualitative and quantitative assessment types. The results of each instrument were revealed in Figure 8. Among the 44 studies, 11 (25.0%) just adapted one kind of measurement instrument. Some employed more than one assessment instrument to collect multidimensional evidence of students' learning performances. Thirteen (29.5%) used two measurement instruments, and twenty (45.5%) combined three or more evaluation methods.
For the measurement instruments, 23 (52.3%) of the studies adopted a survey/questionnaire for measuring the students' learning performances in the design thinking integrated context, which is the one with the largest proportion (see Figure 8). The second-largest method adopted was portfolios (14, or 31.8%), followed by interview, observation, protocol analysis, and test categories, which all accounted for a frequency of 11 (25.0%). Next was design work evaluation and video recording, and the frequency was 10 (22.7%) for both. Audio recording (6 or 13.6%) and journal (5 or 11.4%) were also included in the measurement instruments. Explanation and examples are also presented in Table 4. Because some studies used a combination of evaluation methods, the sum of the numbers was greater than 44 (or 100%). Overall, qualitative measurement instruments were used more than quantitative ones.  Table 4. Measurement instruments adopted in the empirical studies.

Protocol analysis
Protocol analysis is often adopted to understand the thought process of individual or groups in a natural way, the object of its analysis includes the coded verbal communication, or the thought process being asked to speak out (e.g., [41,46,59]).
Design work evaluation Design work is seen as a direct way to reflect students' learning outcome, and it is widely implemented in the evaluation of DTIL (e.g., [31,35,48]).
Video recording/Audio recording In DTIL, researcher recorded the design activities by video or audio so that to understand the participants' learning process more fully (e.g., [38,44,61,62]).

Journal
Journals or diaries from participants is analysed to help understand the process by which learning occurs (e.g., [27,29,38]).  Table 4. Measurement instruments adopted in the empirical studies.

Protocol analysis
Protocol analysis is often adopted to understand the thought process of individual or groups in a natural way, the object of its analysis includes the coded verbal communication, or the thought process being asked to speak out (e.g., [41,46,59]).
Design work evaluation Design work is seen as a direct way to reflect students' learning outcome, and it is widely implemented in the evaluation of DTIL (e.g., [31,35,48]).
Video recording/Audio recording In DTIL, researcher recorded the design activities by video or audio so that to understand the participants' learning process more fully (e.g., [38,44,61,62]).

Journal
Journals or diaries from participants is analysed to help understand the process by which learning occurs (e.g., [27,29,38]).

Study Design
According to the existing taxonomy framework [63], we divided the research designs of these empirical studies into three types: experimental (using random assignment), quasiexperimental (no random assignment but with a control group or multiple measures), and non-experimental (no random assignment, no control group or multiple measures). Among the 44 studies, most used a non-experimental design (31 or 70.5%) (see Figure 9). These studies usually presented one or more cases where an intervention was applied and measured student performance through various instruments. Seven (or 15.9%) studies used quasi-experimental designs, and six (or 13.6%) used experimental designs. Appl 15.9%) studies used quasi-experimental designs, and six (or 13.7%) used experimental designs.

Course Type
Apart from some unclear research contexts, we summarize the course types from 40 studies. Results (see Figure 10) show that 34 (or 85.0%) studies conducted empirical research in the formal K-12 school courses, like the regular or elective courses offered during the conventional teaching schedule. Six (or 15.0%) were conducted in the informal course environment, for example, conducting the experiment during a period of school holidays [23], or the study was conducted as an afterschool program which was not applicable within the conventional school hours [37,53].

Grouping
Twenty-nine of the forty-four studies mentioned grouping in the empirical research. Four studies mentioned that students used group learning but did not specify the size: one study used both individual and collective design activities [64], and students in another three studies were organized in groups [48,55,57]. In the remaining 25 studies (see Figure 11), the group size varied in the research. Six studies employed a group size of 3-4 students per group [ Course type Formal Informal

Course Type
Apart from some unclear research contexts, we summarize the course types from 40 studies. Results (see Figure 10) show that 34 (or 85.0%) studies conducted empirical research in the formal K-12 school courses, like the regular or elective courses offered during the conventional teaching schedule. Six (or 15.0%) were conducted in the informal course environment, for example, conducting the experiment during a period of school holidays [23], or the study was conducted as an afterschool program which was not applicable within the conventional school hours [37,53]. Appl 15.9%) studies used quasi-experimental designs, and six (or 13.7%) used experimental designs.

Course Type
Apart from some unclear research contexts, we summarize the course types from 40 studies. Results (see Figure 10) show that 34 (or 85.0%) studies conducted empirical research in the formal K-12 school courses, like the regular or elective courses offered during the conventional teaching schedule. Six (or 15.0%) were conducted in the informal course environment, for example, conducting the experiment during a period of school holidays [23], or the study was conducted as an afterschool program which was not applicable within the conventional school hours [37,53].

Grouping
Twenty-nine of the forty-four studies mentioned grouping in the empirical research. Four studies mentioned that students used group learning but did not specify the size: one study used both individual and collective design activities [64], and students in another three studies were organized in groups [48,55,57]. In the remaining 25 studies (see Figure 11), the group size varied in the research. Six studies employed a group size of 3-4 students per group [ Course type Formal Informal Figure 10. The frequency of studies by course type.

Design Thinking Task/Challenge
The design tasks or challenges used in these empirical studies are summarized in Appendix A. It can be found that these tasks or challenges are usually real situated, openended, challenging, and cooperative.
On one hand, most of these design activities were based on authentic situations, which are close to the real-life experience of students. Challenges were carried out around topics related to social life, such as to identify and redesign systems that existed at school [38]; the design of an escape room for the local fire department to allow participants to playfully and interactively improve awareness of fire safety in and around the house [30]; working revolved around interviewing senior citizens and creating prototypes that met their needs on the background of the aging society [32]; to create a secure environment for the elderly without taking away their freedom [40]; or designing a heat retaining food container for street food vendors at a taxi depot [46]; designing a water filter system for the city's wastewater management plant to help prevent the pollution of a local river [34]; to develop sustainable food products for peer group [55]; designing and constructing shoes [21]; the design theme on preventing bullying in the social context of the class [58]; or targeting on more local or social problems that could be solved with new products, services, or other solutions [33,50,66], etc.
On the other hand, for all design activities, there were no unified answers. Students were encouraged to design as many creative solutions as possible and choose the relatively optimal solution to solve the problem. Many tasks were engineering, such as the marshmallow tower activity and the trebuchet design activity [45], the design of toys [39,43], the engineering design activities of musical instrument, simple machines, and bioinspired flower [47], or the design of water filters, bridges, circuits in maglev vehicles and windmills, and pollinators and knee braces [62], etc. In another study, students were asked to design a 3D model and print a 3D artefact [57].
In addition, the design activities were usually challenging and cooperative. The problems addressed by designers are wicked problems, such as the design of complex systems or environments for living, working, playing, and learning, and the wicked problems theory of design was explained in [67]. In school education, an important goal is to develop cooperative citizens, which helps to equip students with the ability to collaboratively solve

Design Thinking Task/Challenge
The design tasks or challenges used in these empirical studies are summarized in Appendix A. It can be found that these tasks or challenges are usually real situated, openended, challenging, and cooperative.
On one hand, most of these design activities were based on authentic situations, which are close to the real-life experience of students. Challenges were carried out around topics related to social life, such as to identify and redesign systems that existed at school [38]; the design of an escape room for the local fire department to allow participants to playfully and interactively improve awareness of fire safety in and around the house [30]; working revolved around interviewing senior citizens and creating prototypes that met their needs on the background of the aging society [32]; to create a secure environment for the elderly without taking away their freedom [40]; or designing a heat retaining food container for street food vendors at a taxi depot [46]; designing a water filter system for the city's wastewater management plant to help prevent the pollution of a local river [34]; to develop sustainable food products for peer group [55]; designing and constructing shoes [21]; the design theme on preventing bullying in the social context of the class [58]; or targeting on more local or social problems that could be solved with new products, services, or other solutions [33,50,66], etc.
On the other hand, for all design activities, there were no unified answers. Students were encouraged to design as many creative solutions as possible and choose the relatively optimal solution to solve the problem. Many tasks were engineering, such as the marshmallow tower activity and the trebuchet design activity [45], the design of toys [39,43], the engineering design activities of musical instrument, simple machines, and bio-inspired flower [47], or the design of water filters, bridges, circuits in maglev vehicles and windmills, and pollinators and knee braces [62], etc. In another study, students were asked to design a 3D model and print a 3D artefact [57].
In addition, the design activities were usually challenging and cooperative. The problems addressed by designers are wicked problems, such as the design of complex systems or environments for living, working, playing, and learning, and the wicked problems theory of design was explained in [67]. In school education, an important goal is to develop cooperative citizens, which helps to equip students with the ability to collaboratively solve wicked problems in social life. Therefore, to help students solve these complex and wicked problems, some activities were game-based and aimed to create interesting design situations for learners. Especially for the younger children, the fun of activity design is especially important. Thus, some activities were role-playing, such as the design of castle according to an imaginary engineering situation [64]; or storytelling, such as reading the narrative content of the books and solving the engineering problems presented in the books, and finally rewriting the story [52]. Others also employed computer games to study students' learning based on the design process [60,68].

Design Tools and Materials
Among the 44 studies, 20 mentioned the design tools or materials used. Six provided students with conventional design tools or materials, such as paper, pencil, glue, tapes, scissors, etc., [36,42,44,46,50,51]. Specially, one of these studies employed a reverse engineering teaching context, in which students learned from already designed products rather than started from scratch [51]. By dissecting the product's components and structures, it helps students understanding how those components function and work together, and then students can redesign or build their own product to properly solving a design challenge with micro-innovation or applying the scientific concept they had learned during the learning process.
Fourteen studies employed digital platforms as teaching aids of designing and making in the class. For example, some studies organized design activities with open-source hardware and software. Kim, Seo, and Kim [52] provided students with Arduino Leonardobased device to solve the engineering problems, by which artifacts can be made through assembly of various blocks with sensors and actuators, and also be driven by the programming language based on Scratch and app inventor. In [49], Raspberry Pi with programmable microelectronics, adapted versions of SNaP and Tiles toolkits, and Google's Design Sprint Kit techniques were combined to help IoT design across generations. Another platform, such as the mobile telepresence robot called KT, which is controlled over the internet through a web interface, was used to design human-centred robots that served a need in the local environment and allowed remote peers to explore the local spaces [61]. Christensen et al. [40] also explored middle school students' design literacy who had experience with digital technology in maker settings (e.g., FabLabs).
Lin, Chang, and Li [31] studied virtual reality (VR) teaching application on engineering design creativity. In their study, the junior high school students experienced design teaching with VR devices used as teaching tools. The VR box (for a 360 • view), zSpace built-in software Franklin's Lab (for detailing motor operation), Cyber Sciences-3D (for explaining circuits), and Leopoly-3D (for modelling simulation) (created by zSpace, Inc., San Jose, CA, USA) were used to design an electric model vehicle capable of automatically avoiding obstacles.
There were other tools, meant for prototyping or concept understanding, for example, the 3D design and printing tools, that were employed in some reviewed papers. In [26], the design stage used computer-aided design to create a three-dimensional model of the toy. Forbes et al. [27] studied children' STEM learning performance in 3D technology-enhanced makerspaces by using iPad and 3D design software and print device (Makers Empire, created by Makers Empire Pty Ltd., Adelaide, Australia). Leinonen et al. [57] discussed the digital fabrication in elementary school education. The 3D model design by Tinkercad software (created by Autodesk, Inc., San Francisco, CA, USA) and 3D artefact printing by Ultimaker printer were adopted to aid the design activities, such as designing and printing a name tag, a floor plan, and a game piece. In [43], the teacher explained the mechanical functions and STEM knowledge via physical and 3D virtual simulation models, and then in the inquiry experiment stage, students were asked to assemble the LEGO to design a movable toy with various mechanical structure types.
Besides 3D modelling tools, other design platforms were also employed. Four studies involved students in the poster design activities, and three of them used poster design games on computer [60,65,68], while another one used WPS Writer ® to complete the new year poster design [48].
A tool dedicated to assisting design communication was also mentioned. Won et al. [56] reported on the ways that middle school students appropriated a social networking forum (Edmodo) as a part of the iterative design process in an informal learning environment. In their study, the forum was utilized for the purpose of collaborative design by interaction in progression of the design process. Among the selected 43 empirical papers, most were published after 2017 and most (12) in 2021 (see Figure 1). It was concluded that the growing trend of high-quality empirical studies on DTIL in K-12 education was significant in the past five years. "The proper study of mankind is the science of design, not only as the professional component of a technical education but as a core discipline for every liberally educated man" [3]. According to the rapid development of empirical research, we can predict that more evidence-based research will focus on this topic in the future, and the potential value of design thinking in K-12 education will be further explored.

Educational Level
Considering the uneven distribution of the age groups in K-12 education, more research is needed to engage the younger (like kindergarten and primary school students) and older learners (high school students) into DTIL. As shown in Section 3, the research covered students of all levels in K-12 education, aged between 3 and 18 years. Students from kindergarten to high school were all involved, and among them, the papers studied middle school students' learning accounted for the largest proportion (23 or 52.3%). It shows that the distribution of existing studies was uneven across educational levels. As a methodology for innovative problem solving, the design thinking intelligence is not necessarily something that comes naturally, it needs to be nurtured and developed in pedagogical approach. According to [69], design must be nurtured from early beginnings, as building design capital is a visionary and long-term job. Wells [70] also pointed out that, similar to language learning, design thinking is something that should be carefully nurtured from an early age and be included in all areas of education. However, in comparison with the studies conducted in university or other adult learning scenarios, there is still a lot of room for research in K-12 education.

Samples and Duration
Among the 44 studies extracted from the 43 papers, many selected small samples or short periods to carry out the empirical research. The reason may be that the activities integrating design thinking had higher requirements for teachers and students. First, teachers need to spend more time preparing various sources for class teaching, such as the design tools and materials. Second, during the lesson, support and guidance are required for student activities. Especially in the hands-on activities, teachers are also concerned about the safety of their students. Third, the diversification of evaluation, such as the evaluation of the design process and design works, also requires teachers to spend more time on observation, interviews, surveys, etc., to obtain and analyse multi-dimensional learning data. The limited energy of teachers makes it difficult to carry out large-scale class teaching for a long time. Fourth, for students, the design thinking tasks are usually more challenging, which require a longer cycle to constantly iterate and optimize.
However, for DTIL in K-12 education, empirical studies with longer periods and larger samples are still expected to provide more sufficient and convincing evidence to demonstrate the application effect. Despite the difficulties, researchers and teachers could try to solve these problems with relevant technologies, thereby expanding the sample size and extending the experimental period. For example, teachers could try to use intelligent teaching assistants to assist in monitoring students' learning, real-time feedback and forewarning of learning, etc., so that they can better communicate with students, guide them according to students' needs, and realize personalized teaching. Students could use intelligent technologies such as intelligent learning partners to carry out learning, reduce their learning burden and increase their interest, so that they can actively participate in longer-term learning activities. In addition, multiple subjects are encouraged to participate in teaching and research related to DTIL in K-12 education. In addition to research institutions (e.g., university), K-12 schools, and government departments, more social forces are also expected to join, such as communities, science and technology museums, enterprises, etc., to jointly provide support for DTIL, thus ensuring larger scale and longer cycles teaching practice.

Curriculum
More research is needed to explore the integration of design thinking with single or multiple disciplines from a wider subject range. As the social movement of design thinking has called for the involvement of design thinking into K-12 classrooms, we put together the disciplines mentioned in the reviewed empirical studies. The results revealed that the implementation of design thinking in non-design classrooms is increasingly concerned. Although design thinking by its nature originated from design field in an effort to encourage people to think and practice like an expert and prepare future designers [70], it is now become a simplified version of "designerly thinking" or a way of describing a designer's methods that is integrated beyond the design context [7,9,15]. In addition to design-related courses, most researchers employed the design thinking integrated teaching and learning in the STEM related subjects, such as science, STEM/STEAM, engineering, technology, etc., which is consistent with previous findings [11]. While other researchers have extended design thinking to non-STEM subjects, such as geography [38]. This has a lot to do with the rise of STEM education in recent years. There is societal recognition of the role played by STEM education in preparing students for college and career readiness, and design challenge exactly provides a pathway for the STEM disciplines to work together [14]. Additionally, it has adopted the stance that design and technology should be embedded in various school subjects [71].
How is design thinking integrated into these curriculums? Despite the differences in the curriculums themselves, there are some common threads of DTIL. First, for the purpose of DTIL, it mainly responds to the need for interdisciplinary and innovative talent cultivation in the age of intelligence. Therefore, design thinking is integrated by enabling students to learn and connect multidisciplinary knowledge and skills via involving the design process/skills. Second, for the learning content, it is usually embedded inside the open-ended and authentic problem context, in which students' learning content stems from real life, not those hypothetical questions. That is, the integration of design thinking provides ideas for the reframing of teaching content or topics. Third, for the organization of learning, the curriculums adapted DTIL are characterized by student-centred and co-creative. Teachers act as coach to enable active learning, while students can communicate and collaborate with peers and even with stakeholders so that be able to solve problems creatively.

Design Thinking Model
The design thinking models should be aligned with the complex teaching situations. It was found that the core concepts of design thinking are valued and pursued in the field of K-12 education. However, in terms of the design thinking model, there is no panacea for all situations, that is, there is no unified framework or process in academic. For example, when talking about the design process and characteristics of a certain design activity (e.g., the technology and engineering design), it does not mean that there is just one process [14].
Researchers have also pointed out that if teachers rely too much on the confined procedural, pre-determined process structure, it may strangle the qualitative forms of intelligence inherent in design thinking [70]. In fact, in the human-made world, different fields or disciplines have their applicable design processes, the process of design thinking is surely not deterministic [44]. Therefore, the key ideas of the existing studies for design thinking provide a foundation for understanding the capabilities related to design. On this basis, teachers can adjust the design thinking model reasonably according to the actual situation.
In this study, based on the evidence of the reviewed empirical papers, it reveals that the highly concerned aspects of design thinking are the following: prototype, ideate, define, test, explore, empathize, evaluate, and optimize. These key elements provide a holistic outline for students to solve problems with design thinking. Overall, the results of this study are consistent with previous studies. For example, ITEEA [14] elaborated eight key ideas of design in the PreK-12 technology and engineering education, including a fundamental human activity, open-ended and can always be improved and refined, iterative, a range of skills, universal principles and elements, making, optimized by criteria and constraint, and diversity of approaches. In [11], design thinking in the K-12 educational context was noticed, and they organized the multiple codes of design thinking competences into six areas, including reasoning, problem setting, empathy, ideation, modelling, and process management.

Dependent Variables
Knowledge, skills, and dispositions are all covered by the dependent variables reported from the reviewed papers. For example, the existing empirical studies measured students' knowledge of relevant subject concepts, e.g., physics concepts of acceleration, Newton laws of motion, velocity, motion, energy, etc., [37], or the understanding of the mechanical concepts [43]. Some studies also checked students' understanding of design thinking (e.g., [39]). Various skills, like creativity, productive thinking, problem solving, and critical thinking, are evaluated as crucial learning effects. Except for the concepts and skills, students' performance on design thinking practice (including the design process and design work) has been paid close attention, and it is the most frequently evaluated one. This exactly confirms the practical orientation of design, or the non-verbal competence [15] that is required. Besides, many studies surveyed students' dispositions, as there exists a bi-directional relationship between emotion and design-based learning [12]. To sum up, in the DTIL context, design thinking is considered in many studies as both a dependent variable and partly as an independent one. Results of the learning effects indicate optimistic learning outcomes of students; additionally, the competences cultivated by the design thinking activities were often associated with the 21st century skills, like communication, collaboration, creativity, and critical thinking [11,27,29,40].

Evaluation Instruments
Considering the benefits of applying design thinking into other subjects, more objective and apt evaluation instruments need to be developed to emphasize the alignment between design thinking competences and domain knowledge. The evaluation instruments used in the existing empirical studies include survey/questionnaire, portfolio, interview, observation, protocol analysis, design work evaluation, test, video recording, audio recording, and journal. Qualitative and quantitative assessments are both included, and the qualitative assessments are used more frequently. However, it was found that most of them were self-designed measures, such as portfolio, survey, interview, observation, design work evaluation. These assessments were highly subjective. To better serve the trending integration of design thinking into STEM and non-STEM subjects in K-12 education, it calls for more objective and apt design thinking-embedded evaluation instruments that evaluate comprehensive performance for researchers and educators to adapt in their interventions or classrooms. Researchers have pointed out the shortcomings of current evaluation methods, for example, Blom and Bogaers [46] indicated that current verbal protocol analysis methods and theoretical frameworks did not explain how internal and external information sources contribute to novice designers' moment-to-moment thought processes, and they employed a protocol analysis method-Linkography-to investigate the nature of novice designers' thought processes. Furthermore, given that DTIL is usually suggested to be involved in the complex problem-solving situations, more timely evaluation and feedback are needed to help students gain a better learning experience. In the era of intelligence, there is also an urgent need to explore assessment supported by various intelligent technologies.

Study Design
Most of the studies tended to report the DTIL via non-experimental case study in K-12 context [16], the number of which far surpassed that of the quasi-experimental and experimental studies. In the educational context, samples were usually restricted by the original class settings, therefore, it was relatively more difficult to carry out rigorous experimental research that emphasizes the randomization of samples. In this instance, the application of case studies does have value and facilitates the introduction of concrete practical experiences. However, the problem that ensues is the difficulty in identifying the certain competences can be developed by participating in design thinking integrated activities [11], so that a compromise approach is to conduct quasi-experimental study design. According to [63], a study with no random assignment but a control group, multiple measures can be classified into the type of quasi-experimental design to examine whether such a design thinking integrated intervention, as the independent variable, influences students' learning, as the dependent variable. In the future, more rigorous research with quasi-experimental or experimental study design is needed to strengthen the learning effect verification.

Course Type
Most studies were conducted in a formal educational context (i.e., classrooms). The high proportion of studies conducted in the formal educational setting appeared to align with the move of involving design thinking into educational context. Besides, it is also consistent with the finding of large number of research conducted in standard courses, such as science, engineering, technology, etc., (see Section 3). In the future, more studies should be conducted for design thinking integrated interventions or activities applied in informal educational scenarios, such as the design workshops or makerspaces, so that it can be a complementary way to highlight the educational value of design thinking.

Grouping
Most research defaults to a collaborative learning approach to design thinking embedded activities. Collaborative learning has the potential to foster students' understanding of various subject domains, help complex problem solving, and master social skills [58,72]. The group sizes typically employed were between two and four people and were usually randomly grouped or arbitrarily appointed by the teachers. It lacked detailed research to explain the specific reasons or rules. A few studies have looked at the impact of gender differences on design thinking embedded learning [31,34,37]. More in-depth studies are needed to explore applicative collaborative strategies, such as group size, grouping methods like role assignments, dynamic grouping or fixed grouping, gender combination, etc.

Design Thinking Task/Challenge
The design thinking embedded activities were usually comprehensive, transdisciplinary, and integrate a variety of learning methods, such as project-based learning [55], problem-based learning [21], inquiry-based learning [36], game-based learning [60], storytelling [64], etc. It was in line with the characteristics of transdisciplinary learning such as STEM. Additionally, process-oriented design (design-no-make) and product-oriented design (design-and-make) were both adopted to produce various creative solutions, object models, smart products, engineering devices, etc. For example, while involved in the activities of poster design [48,65], playground design [41,42], or the design of smart-things ideas for an outdoor park environment [49], students were encouraged to propose creative ideas to meet certain needs. When participated in the product-oriented design tasks, students were also required to construct artifacts, such as designing and constructing a device that was engineered to provide electricity to a third world community scenario [36]. Results of this study should inspire researchers and teachers to develop design thinking embedded activities that are reality situated, open-ended, challenging, and cooperative, to engage younger students to better develop their domain knowledge and thinking skills, activities that are fun and hand-and-brain on will be in greater need. In future, the integration of design thinking in project-based learning and STEM/STEAM education can be more diverse, and it would be more expected if it can be combined with local cultural characteristics. For example, in one study [73], the "Chinese Wooden Arch Bridges' Intelligent Monitoring system " project was developed to explore the integration of cultural education, scientific inquiry and design thinking. In this STEAM project, the integration of design thinking is believed to not only promote transdisciplinary learning, but also help cultural understanding and inheritance.

Learning Tools and Materials
The tools and materials employed are expected to help students formulate and express creative design ideas. For example, the design cards are convenient for students to sort out the design process and carry out reasoning [49]. The 3D design and printing platforms can help students design and visualize prototypes [27,57]. Conventional design and construct tools, as well as the digital fabrication kits, provide students with hands-on opportunities to materialize ideas. According to the reviewed articles, there were more applications of conventional tools and materials than intelligent ones. Researcher argued that the functions of design are making better what is inadequate and delivering a value that has not yet been delivered, thus technology is a key driver of the project as a whole and will play an important role in that [74][75][76], likewise, design can promote technological literacy. Future study could be conducted to further explore the technology-enabled design thinking activities in K-12 context.
Especially, in the emerging digital lean manufacturing paradigm [77], the advent of Industry 4.0 (I4.0) technologies such as Artificial Intelligence (AI), Internet of Things (IoT), Virtual Reality (VR), and Mixed Reality (MR), etc., have provided significant enlightenment to education. On one hand, new technologies can be used to enable better performance. For example, during the COVID-19 pandemic, technologies make such large-scale online teaching activities possible. In particular, for experimental or hands-on courses, the technology application like the online simulation platform (i.e., IRobotQ3D) [78], the integration of VR and motion capture system [79], etc., can solve the problems of interactivity and presence to a certain extent. On the other hand, the essence of I4.0 is also to better meet the needs of users, which coincides with the human-centred concept emphasized by design thinking. As we enter the era of intelligence, customers require new and customised products more and more often, and companies have hence to be able to easily follow these requirements in order to survive in today's competitive market [79]. The implication for education is that to integrate technology into DTIL activities may help students collect and analyse data more scientifically, open up broader design space, and enrich the ways and forms of design work presenting, so as to promote the cultivation of innovative intelligent manufacturing talents that meet the needs of the era.
In conclusion, the implications and suggestions for DTIL in K-12 context are: (a) The potential value of design thinking in K-12 education needs to be further explored with larger samples and longer durations, in particular, more attention needs to be paid to students at the preK-elementary and high school levels. (b) Design thinking models be aligned with the domain learning situations are needed to expand the application of DTIL in a wider range of disciplines, especially non-STEM disciplines. (c) DTIL is expected to facilitate the acquisition of higher-order learning achievements, correspondingly, more objective and apt evaluation instruments are needed. (d) The educational value of design thinking does not arise spontaneously, more targeted interventions are expected to reach its full potential, such as rigorous study design in the pre-post mode or with control groups, more focus on learning in informal settings, intensive grouping strategies that consider gender, group size, role, etc., pertinent design thinking tasks/challenges what are both fun and challenging, and finally, support design activities with appropriate technical tools or materials.

Conclusions and Implication
In the wake of the popularity in design thinking, this literature review analyses 43 high-quality empirical papers on DTIL in K-12 education. The current status and future directions are identified to narrow the gaps. The major findings are summarized as follow.
First, as deduced from the publication trend (see Figure 1), a positive picture of design thinking emerges. However, the empirical evidence to support the effectiveness of DTIL is still rather limited. The results indicate that although samples from preschool to high school are all involved, it shows an uneven distribution of the age groups in K-12 education, which calls for more research to engage the younger learners (like kindergarten and primary school students) into DTIL. If possible, studies with longer periods and larger samples are expected.
Second, DTIL has the potential to deepen students' learning with diversified models in various subject domains, including that of STEM and non-STEM, while the most mentioned applications are the fields of STEM, which is consistent with the result of previous study [12]. In addition, the design thinking models applied varied among studies, which suggests inconsistency in current perceptions of design thinking. On one hand, a prevailing view was to apply design thinking from a thinking model of designer or designer's mindsets (e.g., [40,[49][50][51][52]), in other words, think of it as a unique problem-solving methodology (e.g., [33,35,37,42,44,48,53]). Our study also supports this view. The theoretical basis of design thinking has been propositioned to be interconnected to Dewey's notion of pragmatist inquiry and aesthetic experience [23,36]. The corresponding training goals were the abilities required to solve the complex and open-ended problems, such as communication skills, cooperation skills, creativity, and critical thinking, etc. On the other hand, however, there were differences in the focus of studies on the application of design thinking, leading to the emergence of different models or competence frameworks. For example, some studies focused on inquiry and experimentation in the early stages of problem solving (such as discover, define, empathy), others focused on modelling or prototyping during the problem-solving process, and still others discussed the impact of emotions, etc.
How do teachers choose or construct a design thinking model or framework that meets the needs of the teaching situation? In fact, it is difficult to construct a universal design thinking model to deal with a variety of complex educational scenarios. Therefore, to meet the teaching needs of more subject domains, future study needs more exploration of the design thinking models that could be compatible with teaching practice in K-12 education. Referring to the existing empirical research, it may be necessary to comprehensively consider related factors such as students, teachers, and the school environments. In addition, although existing research claims that design thinking has great educational value, it is difficult to cover all aspects at once. Therefore, it is a feasible way to develop various competencies by stages or disciplines. Future study could bring design thinking into both STEM and non-STEM curriculums to further verify how design thinking will help students build knowledge and skills in a wider range of domains. Furthermore, teachers can make targeted combinations or adjustments of elements in a design thinking model or framework to better meet teaching needs.
Third, in response to the many existing design thinking models and subject domains, various dependent variables were measured by different instruments. Mostly, the empirical studies evaluated students' performance on design thinking, such as the design processes and design works. Another two types of learning performances that were paid close attention were emotional/social aspects and subject concepts. Besides, the design thinking concept and other skills like creativity, problem solving, critical thinking, etc., were also involved in the measured variables. For assessment, qualitative and quantitative assessments were both included, and the qualitative assessments were used more frequently, which highlights the practice-orientation and process-orientation of design thinking.
Implications for instruction include: (1) for the dependent variables, both process performance and learning outcomes should be emphasized. In particular, it is helpful to explore the development process of students' thinking by analysing the process of students' design activities. (2) Besides, the culture competency also needs to be included in the assessment. Design was seen as an important force driving the development of the artificial world, so one of the core characteristics of design thinking is that it is human-centred [1,7,32,80,81]. Every citizen should have a certain quality of cultural understanding and inheritance, which emphasizes the practical implementation of these values contained in the excellent culture at the behavioural level [82]. On this basis, we can better solve the problems faced by human beings through design thinking activities, to promote the benign development of human society. (3) To verify the design thinking embedded learning more scientifically and efficiently, more objective and apt evaluation instruments are needed. In addition to conventional subject knowledge and skills, for the evaluation of design thinking, existing research has provided some experience for reference. For the understanding of design thinking concepts, it can be assessed by test [35], survey [39], questionnaire [40], or other qualitative method, such as interview, video/audio recording, etc. [38]. For design thinking, evaluation instruments that can be used are portfolio (e.g., [21,33,42,55,61]), observation (e.g., [45,64]), survey (e.g., [31,33,55,56]), protocol analysis (e.g., [21,42,44,46,59,62]), design work evaluation (e.g., [31,35,45,47,58,60]), interview (e.g., [55]), etc. (4) Furthermore, there is more room for development of relevant evaluation methods supported by technology. In the existing studies, evaluation instruments supported by technology are rarely used. In the future, the application of artificial intelligence technology, multimodal data collection and analysis, real-time evaluation and feedback technology in the classroom still needs to be explored to promote the improvement of teaching quality.
Finally, interventions including study design, course type, grouping, design task/challenge and teaching tools and materials are summarized from the empirical studies. Results show that most research reported DTIL by conducting non-experimental studies in formal classrooms. In the reviewed studies, collaborative learning was frequently used, through which students were usually arranged in groups of 2-4 people. Except for cooperative, the design tasks or challenges employed in the empirical studies were also characterized as real situated, open-ended, and challenging. Moreover, conventional tools or materials were used more often, technical tools and materials were relatively lacking. Overall, at present, the development of design thinking intervention in K-12 education is not sufficient. We are inspired to pay more attention to the application of rigorous study design, balance between formal and informal classrooms, intensive grouping strategies, applicative tasks or challenges, and technical materials in the future. For instruction, because the situations vary among schools, the DTIL intervention should vary accordingly. Moreover, in the selection of design materials or tools, it is also necessary to pay attention to the application of technical materials or tools, to help students understand the technological world in which they live and learn to use technology to innovate and shape the world.

Limitations
As this study mainly focuses on students' learning in a design thinking integrated context of K-12 education, there are some limitations. First, we selected the reviewed papers only from SSCI journals, which ensured the representativeness and quality of sample but lacked comprehensiveness. Second, the school levels were limited to preschool to high school, while colleges and other adult learners were not included, so that the conclusions are only applicable to K-12 education. Third, teachers were considered crucial for learning; separate study is needed to amply elaborate the role of teachers in DTIL. In future, it is expected that more research on teaching, learning and evaluation of DTIL will be carried.