Pedagogical Resources for Conducting STEM Engineering Projects in Chemistry Teacher Education: A Design-Based Research Approach

Abstract

Project-based learning provides a common context for STEM education at all educational levels. However, before future chemistry teachers can implement it in their teaching, they need to have experience in completing complex projects by themselves. According to previous research, an engineering perspective in STEM projects has been difficult to implement. Therefore, this design-based research project focuses on producing pedagogical resources for conducting STEM projects based on authentic engineering practices. Through three-cycle design research, we crafted Excel templates that support a step-by-step framework for completing complex engineering projects and an evaluation matrix that includes formative and summative tools. The design solutions were validated through empirical problem analysis, which yielded qualitative insights into the possibilities and challenges of the produced tools. From this data, we formulated five best practices for teachers to focus on achieving successful project outcomes, with priority being to support the progress of the engineering approach and support it via guidance and peer collaboration. For future chemistry teachers, artificial intelligence tools offer support, especially for hardware assembly and software coding. The research produced educational artifacts that support conducting STEM projects in higher education and insights into their best practices. Since design solutions are based on research and real-life engineering practices, they are useful for all fields in higher education that conduct STEM projects and aim to teach authentic engineering skills.

1. Introduction

The most recent advances in teaching STEM subjects tend to promote engineering practices within a science education. Engineering enables an integrative approach to STEM, which is more comprehensive than teaching individual subjects separately (Cheng et al., 2024; Dolgopolovas & Dagienė, 2021; García-Carmona & Toma, 2024; Hallström et al., 2023; Ozkizilcik & Cebesoy, 2024). That said, the holistic pedagogical approach can pose significant challenges for future teachers without an engineering background (García-Carmona & Toma, 2024; Mumba et al., 2024). One solution for meeting the challenge is to familiarize future STEM subject teachers with authentic engineering practices (Aslam et al., 2023; Maiorca & Mohr-Schroeder, 2020).

Hands-on experimental study is a widely recognized type of exercise for ensuring learning in STEM subjects (Christian et al., 2021; Güder & Gürbüz, 2022; Ozkizilcik & Cebesoy, 2024; Pernaa & Wiedmer, 2019; Zaher & Damaj, 2018). However, there is a great need to develop authentic engineering contexts in which to learn essential experimental work skills. Through our research, we propose that one possible mode for introducing hands-on experiments into STEM education is the design of experimental setups based on simple mechatronic systems using sensors and actuators connected to microcontrollers or single-board computers.

Our previous research (Ambrož et al., 2023) indicates that this pedagogical approach offers numerous opportunities to support learning in chemistry teacher education. However, there are also several challenges that need to be addressed in course planning. Our research has revealed that to achieve a successful project outcome, future chemistry teachers need assistance in designing a proper chemistry context, in planning the measurement setting, step-by-step support for project management, and workshops for learning basic engineering skills, such as soldering, coding, and 3D printing. In general, future chemistry teachers tend to find an open engineering project demanding because it combines so many different knowledge domains, such as chemistry, electronics, and computer science. The challenge is that they lack such skills since, by default, chemistry teacher training programs do not offer classes in computer science or electronics. Students would need to select them as extra courses (Ambrož et al., 2023).

It is well documented in the literature that future teachers find the engineering component in STEM education challenging, with most not possessing an engineering background (Baze et al., 2025; García-Carmona & Toma, 2024; Mumba et al., 2024). In our previous research (Ambrož et al., 2023), we addressed this challenge by teaching future chemistry teachers an authentic engineering work process, which we called the “engineering approach” (Karakasic et al., 2018; Zadnik et al., 2009). It offers many possibilities for strengthening the teaching of engineering in STEM education. However, at first students found it difficult to understand (Ambrož et al., 2023), most likely because it is a professional-level work process that has not been designed for educational purposes. Therefore, we argue that an engineering approach needs pedagogical crafting to be better suited to the needs of teaching and learning. In this regard, the aim of the present research project was to develop pedagogical resources for supporting the use of an engineering approach in chemistry teacher education.

To ensure a research-based approach, this study was carried out as a design-based research (DBR) project (Anderson & Shattuck, 2012; Edelson, 2002). Therefore, the article has a typical DBR structure. First, we introduce DBR as a research methodology and present the DBR-driven research questions (Section 2). Then we present the design narrative by describing the cyclic design process, starting from theoretical problem analysis (Section 3) and continuing to empirical validation (Section 4).

2. Design-Based Research Approach

DBR is a methodology that enables research-based, systematic development of educational artifacts, such as learning environments, materials, assessment rubrics, and pedagogical models. It was originally developed to reduce the gap between educational research and educational praxis (The Design-Based Research Collective, 2003). Indeed, there was seemingly a great need for this kind of methodology because during the last few decades, DBR has been extensively adopted in the field of education (Anderson & Shattuck, 2012).

In this research, we make use of one of the most cited DBR approaches, published by Daniel Edelson in 2002. Edelson’s model organizes DBR by focusing on three design perspectives: problem analysis, design procedure, and design solution. The design solution is the actual research-based pedagogical artifact produced. Often, DBR includes both theoretical problem analysis, such as a review of research literature (Section 3), and empirical problem analysis, such as case study interventions, where authentic end users interact with the developed design solutions (Section 4). Design procedure is a framework that describes the project’s timeline and the resources used in the project (Edelson, 2002). DBR suits our development purposes quite well because it is at the same time practical, producing real educational solutions, but also research-driven via the conducted problem analyses.

2.1. Design Procedure

This DBR project started in 2022 as an educational collaboration between the University of Helsinki (UH, Finland) and the University of Ljubljana (UL, Slovenia). The article represents the culmination of the larger research project and reports on the third design cycle. During the three cycles, the design objectives and foci changed, which is common in iterative DBR projects (Sandoval, 2004).

To describe the changes in each cycle, we use a model called TPACK as a framework (see Figure 1). The analysis of design foci per cycle was performed retrospectively, and its purpose is to produce an accurate description of what the development focused on in each cycle. TPACK is an acronym referring to Technological (TK), Pedagogical (PK), and (A) Content Knowledge (CK). TPACK is widely used in educational technology research. It was first devised two decades ago in 2005 (Koehler & Mishra, 2005), and since then, it has been used in thousands of educational research articles. For example, TPACK can be used for understanding what kind of knowledge domains are needed for the meaningful pedagogical use of a specific technology (Koehler & Mishra, 2009; Mishra & Koehler, 2006).

Cycle 1/2022: The first design cycle began with a need analysis assessment, which usually is how DBR projects begin (Juuti & Lavonen, 2006). The need analysis involved theoretical problem analysis, where we reviewed earlier research on STEM in chemistry education. Then, we analyzed the possibilities offered by single-board computers (SBC) for chemistry measurements and used them as a context for designing engineering projects for future chemistry teachers. We combined all the knowledge and crafted a five ECTS master’s level course, taught at UH during spring 2022. The course represented design solution 1 of the DBR project. The final phase of the first cycle involved empirical problem analysis, where we gathered data from students who attended the course. The results of cycle 1 were reported in an article by Ambrož et al. (2023). In this first cycle, we started from scratch, and thus we needed to work on every TPACK domain (see Figure 1).

Cycle 2/2023: DBR is an iterative process (Edelson, 2002). Therefore, the second cycle was based on the results of the first cycle. First, we changed the hardware platform from Raspberry Pi to Arduino because the unit responsible for the course at UH (Chemistry Teacher Education) received 10 Arduino education kits as a resource donation from the faculty. To support the introduction of Arduinos, we began to collaborate with [omitted for review] University ([omitted for review]), where we located an Arduino expert. The expert helped us to re-design course version 2 around Arduino microcontrollers (design solution 2). During the second version of the course, we encountered a series of unexpected events as generative artificial intelligence (AI) chatbots were published (OpenAI, 2022), and they spread rapidly across all educational levels. Therefore, we instantly needed to re-design coding assignments during the course. Fortunately, the AI chatbots helped our course a great deal because teachers found them quite useful for producing basic-level Arduino code in the context of conducting chemistry measurements (Pernaa et al., 2024). In the second cycle, the change from Raspberry Pi to Arduino and the launch of AI chatbots forced us to re-design the course, especially from a technological perspective (see Figure 1).

Cycle 3/2024–2025: This article focuses on reporting the findings from the third design cycle, conducted in spring semesters 2024 and 2025. It began as theoretical problem analysis because we needed to explore the latest published research addressing the topic. As mentioned in the introduction, the objective of design cycle 3 was to strengthen the way in which an engineering approach is introduced to students because we had experienced challenges with it in the previous two attempts at teaching the course. In addition, we wanted to develop the pedagogical toolset by crafting teaching tools that would support the evaluation of engineering projects. Especially, there was a need to support formative evaluation (Ambrož et al., 2023). Improving the means for assessing a project is important, as noted by other STEM scholars (Hellum et al., 2024). In this regard, the design focus in cycle 3 was mainly pedagogical, focusing on how to introduce an engineering approach to future chemistry teachers and how to assess their projects (see Figure 1).

2.2. Research Questions

To fulfill the aims set for cycle 3, we formulated the following research questions (RQ):

• RQ1: What are the best practices for implementing an engineering approach in STEM projects as part of chemistry teacher education?
• RQ2: What kind of assessment tool can be used to evaluate STEM projects conducted by future chemistry teachers?

3. Theoretical Problem Analysis

Cycle 3 started via theoretical problem analysis. It was conducted as a literature-based need analysis that provided a theoretical grounding for further design steps (Edelson, 2002). As reported in the first cycle, we selected project-based learning (PBL) as the pedagogical model for best implementing engineering projects during the course (Ambrož et al., 2023). It was a logical step since the engineering approach is a systematic process that can be broken down into project milestones that offer a coherent structure for PBL.

However, we felt that project evaluation would be overly challenging during the first design cycle; thus, we focused on pedagogical development (Ambrož et al., 2023). Therefore, the focus of the theoretical problem analysis is on reviewing the literature that addresses the evaluation of PBL. Before introducing the relevant PBL literature, we first define and describe the background to the engineering approach.

3.1. Engineering Approach

The engineering approach refers to a systematic iterative process for designing engineering systems using unified concepts and tools. The study model includes the following steps:

• Concept synthesis.
• Concept evaluation.
• Concept selection and production.

The first two steps are normally performed iteratively to ensure that all the project parameters are considered. This iteration process is referred to as the “design loop” (see Figure 2) (Karakasic et al., 2018; Zadnik et al., 2009).

In our pedagogical case, the process of creating a hands-on experimental setup began with the idea definition and then proceeded to an analysis of its functions and main principles. The approach is used to synthesize multiple concepts by applying the morphological matrix tool. At the end of the design process, the generated concepts are evaluated using a multicriteria value analysis tool. Based on the evaluation results, the most suitable concept is selected for production (Ambrož et al., 2023).

3.2. Project-Based Learning and Assessment in STEM Education

PBL is based on socio-constructivist theories of learning, and as such, it is a context-based, collaborative, and student-centered pedagogical approach that organizes learning around clearly defined projects linked to understanding real-world phenomena or solving concrete problems (Haatainen & Aksela, 2021; Kokotsaki et al., 2016; Navy & Kaya, 2020). It is one of the main pedagogical approaches currently used in STEM education (Halawa et al., 2024). However, previous research has also reported various challenges faced by teachers when implementing PBL in STEM education. For example, supporting and facilitating students’ PBL is one of the main challenges (Haatainen & Aksela, 2021; Mentzer et al., 2017). To ensure successful outcomes, PBL implementations must meet research-based design principles that describe the essential components of a PBL approach. According to such principles, PBL can be described as a process of learning that starts with a clearly stated guiding question or problem; it should include student-engaging activities, such as inquiry; and it should result in artifacts or final products that address the set question (Haatainen & Aksela, 2021). In addition, PBL should include the learning of curriculum concepts and transversal skills, such as STEM-related knowledge practices, and these curriculum-related contents and skills should be clearly stated as learning goals and included in the assessment of the project (Haatainen & Aksela, 2021; Navy & Kaya, 2020).

All the above perspectives have been considered in our research case. For the project, future chemistry teachers are tasked with designing and building an SBC-based chemical measurement device and developing a PBL activity around it (Ambrož et al., 2023).

Previous literature on PBL notes the importance of providing teacher guidance throughout the project process (Haatainen & Aksela, 2021; Kokotsaki et al., 2016). Teachers should focus particularly on formative assessment and using scaffolding instructions, both of which support the learning process. Scaffolding instructions refer to any method or resource (e.g., teachers, peers, learning materials, and technologies) used by teachers to help students accomplish more difficult tasks than they otherwise might be capable of completing on their own, identified as one of the key design principles of PBL (Haatainen & Aksela, 2021). According to PBL research (Condliffe et al., 2017; Kokotsaki et al., 2016; Savery, 2019), two key elements of scaffolding are as follows:

• Scaffolds need to be tailored to a student’s current level of understanding.
• Scaffolds should be phased out over time as students learn to apply the new knowledge or skills on their own.

Formative assessment includes this same idea of understanding students’ current level of thinking and supporting them in moving forward through positive feedback (Black & Wiliam, 2009; Carless, 2019; Offerdahl et al., 2018). Formative assessment and associated feedback are based on a robust dialogue between teacher and student, which creates an iterative feedback loop (Carless, 2019) that in turn supports teachers through the use of appropriate scaffolding instructions. Black and Wiliam (2009) have conceptualized formative assessment based on five key strategies for teachers:

• Clarify and share learning goals and criteria for success with students.
• Design and implement classroom activities that elicit evidence of student understanding.
• Give constructive feedback that helps students move forward.
• Activate students as instructional resources for one another, for example, through peer evaluation.
• Support students’ authority over their own learning, for example, through self-evaluation.

For instance, classroom discussions, concept maps, Kahoot quizzes, or questioning can be used to achieve the second key strategy, while teachers can use oral feedback or comment-only marking to achieve the third. Evidence of student thinking can be a single word used during a class, in project diaries, or on worksheets, or it could be found in other artifacts, such as group discussions and diagrams drawn on whiteboards (Haatainen & Aksela, 2021; Offerdahl et al., 2018). Especially in PBL, any formative assessment should include students themselves in the assessment process since it is a student-centered pedagogical approach, and the assessment should likewise include a specific end-of-project phase that ensures their reflection on the project artifact, how it answers the driving questions, and what the students have learned during the project (Haatainen & Aksela, 2021).

Even though research has clearly indicated the positive effect that formative assessments and associated feedback can have on learning, variations in instructional practices mediate their effectiveness (Carless, 2019; Offerdahl et al., 2018). Furthermore, when using PBL in STEM education, teachers need to acknowledge the differing epistemological characteristics of various STEM disciplines. For example, the integration of engineering into a science classroom requires recognizing the fact that engineering has epistemological characteristics that differ markedly from chemistry or physics; these characteristics must be accommodated when planning, implementing, and assessing the project to preserve the intellectual integrity of each integrated field (El Nagdi et al., 2018). In summary, there is a crucial need for best practices guidelines and adequate assessment tools when implementing STEM engineering projects in higher education.

4. Empirical Problem Analysis

The alternation of empirical and theoretical phases is the strength of DBR projects and enables us to answer the set RQs (Edelson, 2002). In practice, empirical problem analysis allowed us to validate artifacts developed based on the findings of the theoretical problem analysis. In this regard, we first introduce the developed course that served as the research setting for this study. Then, we describe the data gathering and analysis methods.

4.1. Research Setting

The data were collected in spring 2024 from a course entitled “Research-Oriented Integrative Chemistry Education” (5 ECTS), organized by the University of [omitted for review]. The curriculum was adapted from a course designed previously and reported in an earlier article (Ambrož et al., 2023). The main idea of the course is that students learn to design and implement projects in which they can use PBL in their future careers as chemistry teachers. In the course, students had to design an Arduino-based, hands-on classroom experiment that could be used in upper-secondary chemistry education. The course included the following principal phases.

4.1.1. Phase 1: Theory and Arduino

The course began by familiarizing students with the theoretical background of PBL and experiment automation and introducing them to the basics of Arduino hardware and coding. It included such topics as the basics of sensor and actuator design, data acquisition, microcontroller and single-board computer applications, and control software development. The students were asked to take notes during the presentations, followed by a discussion and question-and-answer session after the theoretical component of the course.

4.1.2. Phase 2: Project Ideas

After the theoretical introduction, students selected the project ideas for implementation. They were presented with a list of idea suggestions that could be used as inspiration for their own ideas. They were encouraged to consult the available literature for other similar project ideas and to combine the gained knowledge to devise a unique project idea of their own. Course teachers did not place any constraints on the main parameters of those ideas, and the students were allowed and even encouraged to discuss their ideas amongst themselves. The students then reflected on the selected ideas in a group meeting.

4.1.3. Phase 3: Engineering Approach

With the project ideas finalized, students began the engineering approach-based design process. The first step was defining the required and desired functions of the product and their principles of realization. The step had proved challenging in previous instances (Ambrož et al., 2023); hence, special attention was given to providing the students with adequate guidance. The students were given general instructions on how to describe the required and desired functions of the product and guidelines on how to research the possible principles used during its implementation. During the process, they were allowed total freedom, but the teachers regularly evaluated, discussed, and, if required, corrected the results so that the students could proceed further in the design process. Thus, the teachers were only involved to the extent of ensuring that the ideas were sound and well defined before students progressed to the next steps of the design process.

Once the product’s functions and the implementation principles were well defined and adequately formulated, each student prepared their own morphological matrix as a tool for concept synthesis (see Figure 3). They then used the matrix to synthesize 4–5 concepts suitable for evaluation. Once the concepts for evaluation were ready, another group discussion, with reflection, took place.

In the next step of the design process, the students crafted synthesized concepts using concept evaluation. For this step, weighted usability function analysis (Karakasic et al., 2018) was used, which accounted for the project’s technical and economic value (see Figure 4). As with the previous design process step, and much like when the course had been taught before, the students found the project evaluation procedure challenging. Therefore, we reflected on and discussed the results once the students had completed the step. Design loop iterations were performed where necessary to ensure that the proposed concepts were synthesized according to the requirements. As a result of the final design process, each of the students had a well-defined, readily realizable project concept ready for production after the design process step had been completed.

The final step was to write control and measurement software and make it work with the assembled microcontroller-based chemistry measurement instrument. The project resulted in working prototypes prepared by each participant (see Figure 5 for an example of a prototype).

4.1.4. Phase 4: Seminar and Peer-Reviewed Article

Students presented the working prototypes at a seminar held at the end of the course. The presentations simulated standard academic oral presentations, such as at conferences. The seminar was organized as a contact session where all the course participants and both teachers were present. Each participant presented their finished and working experimental setup. After each presentation, the participants reflected on the project presented. Together with teachers, they posed questions to the presenter and participated in an open discussion on various aspects of the presented project and concepts. The discussion included, but was not limited to, the following:

• The sources and modes of gathering the initial product ideas.
• The challenges encountered during the design and build process.
• The educational setups planned for the use of the developed prototype.
• The modes and means of inter-peer and other cooperation on the path from the idea to the finished product.
• The use of AI tools to help with coding and writing.

The question-and-answer sessions and the discussions at the seminar were the most important sources of teacher notes, used as one of the datasets for qualitative analysis. As the final assignment of the course, the participants were required to write a peer-reviewed scientific article in which they described their project idea and its realization and reflected on the process that led them to the finished product.

4.2. Data Gathering

The empirical data used for validating the design solutions were collected using a qualitative research approach. A qualitative approach was selected for practical reasons because the number of students in the 2024 course instance was six, and in 2025, it was three (N_total = 9). To support the validity and reliability of the data via triangulation, we collected data from four different sources (Cohen et al., 2018; Tuomi & Sarajärvi, 2018).

Teachers’ notes (1) were taken during and after the seminar presentations. They include free-form descriptions of each teacher’s observations during the reflection meetings held after the milestone tasks, the questions posed to the participants by the teachers and their peers after the presentations, and the participants’ responses to those questions.

Feedback forms (2) were collected from all participants with their informed consent (Appendix A).

Semi-structured interviews (3) were conducted face-to-face or online with four participants after they had completed the course, before the final grading of the students’ projects. The purpose of the interviews was to complement the answers given by the course participants in the survey and yield more in-depth insight into the participants’ opinions of the course and its outcomes. The two course teachers conducted the interviews, and they included the following questions:

• “Now that you have nearly finished the course, can you tell us what the most difficult challenge was that you faced?”
• “What are the most important new skills that you acquired during the course?”
• “If you look back at the device you made, how important do you think the systematic engineering approach was in bringing it to fruition?”
• “Can you describe a typical educational setup in which you plan to use your device?”

The participants could answer freely and add their own remarks and opinions in addition to the direct responses given to the interview questions. The average duration of the interviews was 20 min 33 s (minimum duration: 16:17 and maximum duration: 22:42).

As the final assignment, each participant had to write a short article (4) describing their project from the initial idea to its realization. One aim of the article writing task was to provide the course participants with the skills needed for academic writing. To achieve this end, the draft articles were submitted to a peer-review process, just like with any academic research article. The peer-review process was performed by the course participants themselves (each reviewed one article by another participant) and by the course teachers.

Reviewing the articles confirmed the responses given by the participants in the web survey and in the interviews, with students noting in the web survey that writing the article was difficult. The answers given in the interviews revealed that the participants had trouble following the precise writing instructions and the requirements for the article. In the future emphasis should be put on studying current, verifiable, and relevant references and how to properly cite them. The reviewer’s suggestions were considered in the final versions of the articles. The teachers then graded the articles using the grading matrix, and they were submitted for publication in the LUMAT-B journal. Six articles were accepted and published (Kemppainen, 2025; Ojala, 2025; Öörni, 2024; Roiha, 2024; Taponen, 2024; Turpeinen, 2025). Two manuscripts were rejected, but they are uploaded anonymously to Zenodo with the consent of the authors (Pernaa, 2024).

4.3. Data Analysis

All the data were analyzed using qualitative content analysis (Tuomi & Sarajärvi, 2018). The analysis was carried out by first reading the original material and making observations based on research questions. For example, to answer RQ1, we looked for any challenges that should be considered when developing the course. Next, the observations were classified as themes. See

Table 1. Data analysis example.

Original Expression	Challenge	Solution or Support	Best Practices (RQ1)
“Probably the most difficult was just to pick the subject you’re going to work on.” R3	Selecting the chemistry context is difficult	A list of reference projects collected from the research literature	Supporting the selection of the design context

below for an example of how the data were analyzed. Analysis was first performed by the second author and reviewed by the first author. The analysis process was cyclical and iteratively elaborated until consensus was reached.

5. Results and Discussion

5.1. Design Solution 1: Best Practices for Teaching an Engineering Approach (RQ1)

During the qualitative analysis, we identified five best practice recommendations that should especially be addressed when an engineering approach is implemented in chemistry teacher education.

• Recommendation 1: Support the progress of an engineering approach

Students found the engineering approach challenging in our previous research (Ambrož et al., 2023), and so this time we paid more attention to ensuring that all students could finish the required steps properly. To support students in their project management, we designed project tasks for each engineering step and grouped them into an Excel template as separate tabs (see Appendix B).

Appendix B Tab 1: Ref_Proj is a list of projects and evaluation criteria that helps students evaluate their individual strengths and weaknesses from several relevant perspectives (Figure 6).

Appendix B Tab 2: Own_Proj is a tool for mapping the initial target specifications of the device. It guides students to think about what functions are required at minimum and what the desired full feature set is (Figure 7).

Appendix B Tab 3: Morp_Matrix is a tool that makes it possible to generate different concepts from selected functions and principles (see Figure 3 for an example).

Appendix B Tab 4: Conc_Eval is the final step in the engineering approach before building a prototype. A concept evaluation matrix enables an objective selection of the most suitable device concept (see Figure 4 for an example).

Despite the Excel template, the data indicates that the participants found the “design loop” (consisting of concept synthesis, evaluation, and selection) quite difficult to understand.

According to interview data, participants found the engineering approach a useful work strategy for developing SBC-based chemical instruments. All participants agreed that the introduction of a systematic approach to the experimental setup design was beneficial, but most agreed that it requires mastering a learning curve before it can be practically applied. This perspective is reflected in the following answers:

“I didn’t really understand it at first, but now that I’ve done that once, I think it really helped to, like, to come up with the best kind of solution or … how I could take it further after… this one prototype. So, I think it was hard at first, but it helped me to think at the end.”

R3

“I found it really helpful, and especially you had a chance to engage with different kinds of concepts and compare them, which was super good because easily you would just go down one route and not really compare, like, any alternatives.”

R1

“It felt a bit confusing at first, but it was really helpful to start to, like, I don’t know, to, like, divide this bigger project into smaller parts. So, in that way it helped to create the abstract idea for the concrete thing. … And with this process, I feel like I … can also justify for myself why I made certain decisions.”

R2

One participant found the engineering approach less useful, claiming that his initial idea was clear enough to not need any concept optimization. The participant expressed this feeling in the following answer:

“I think it was useful, but since … I got a very clear vision already in the first phase, when I started planning the project … I didn’t feel all these steps were so necessary, particularly for… me. But yeah, they were … useful, but not necessary in my case.”

R5

However, the respondent also saw value in an authentic process and could use it in another context:

“…as a concept, all of these phases together, I found them very interesting. And actually, I thought maybe I might use it in the future in some cases for myself.”

R5

Incidentally, this participant received the lowest grades for the project idea and conceptualization components of the project, which were assigned but not communicated before the interview.

Finally, not everything always went according to plan. The data indicates that most participants did not experience that the finished product closely resembled what they had envisioned in the initial steps of the project. This observation shows that the design loop may not have been properly carried out by those participants, indicating the need for even more rigorous continuous monitoring of the project’s progress and regular meetings with course teachers.

• Recommendation 2: Support the selection of design context

As reported in our previous research study, the future chemistry teachers found the task of devising the proper chemistry context difficult (Ambrož et al., 2023). This time, we assisted their decision-making process by introducing a list of prior projects discussed in the literature. Our data indicates that two students took their idea directly from the list and four students combined information from several sources. No one generated their own project idea, and they did not read any studies other than the recommended literature. Therefore, the students seemingly felt that generating an original idea outside the presented framework was difficult and thus avoided it. The observation was confirmed in the interviews.

Five participants also wrote a free-text description of how they selected their project idea. The answers reflect the fact that most of the participants had a basic idea of what they wanted to make already before starting the research project and, for the most part, did not want to explore other possibilities during the initial phase of the project. This finding indicates that in the future, students will need to be provided with a stronger incentive to take the initiative and generate original project ideas, as noted in the course evaluations. Several students offered the following comments:

“Probably the most difficult part was just to pick the subject you’re going to work on.”

R3

“I would say the most difficult part was to get the idea …”

R2

Compared to our previous experiences with teaching the course (Ambrož et al., 2023), we found that the reference project list helped. According to the feedback, five students said that it made “selecting the idea for implementation” easy, which can be attributed to the fact that some participants may have perceived the process of selecting the idea for implementation as detached from the project itself, as the following answer suggests:

“I had a few different ideas, and I started narrowing them down with what I found to be the most, either interesting … and also … what was something I was definitely able to do…”

R5

During the discussion on the most difficult aspects of the project, the teachers also asked the participants about the source of their ideas. The answers to this question indicate that two students consulted people outside the course when selecting an idea for implementation:

“I searched for a couple of reference projects, and I tried to look at the equipment from them, but they were pretty different from the one I had originally searched for, so I just pretty much asked my friends what they did. So, my friends were the main source for my information.”

R6

“Yeah, the initial idea was from my brother. … Actually, I … managed to go by myself, but basically I … just told him that I chose his idea but … I didn’t really use his help with the project.”

R4

The answers indicate that more time and effort should be dedicated at the start of the course to gathering and discussing ideas about what kind of experimental setups to build.

• Recommendation 3: Support the usage of generative AI

Only two students had prior experience with generative AI tools before the project. The use of AI was encouraged throughout the project. We have found that it offers guidance, especially with coding and basic electronics knowledge acquisition.

According to the feedback form, one participant acknowledged using AI tools when writing the article, though all of them denied using AI tools when preparing the visual materials. Almost all participants mentioned that they had critically evaluated and verified the results whenever they had used AI tools.

The students’ answers about using AI tools to write the code were the most polarized responses in this category. Four students reported using AI tools when writing the code, while the other two categorically denied having carried out so. Interestingly, the participants who did not use AI for coding were not the ones who found coding the most difficult task. One of the four who did use AI for coding said the following about its difficulty:

“…the coding … part, if we want to be very specific, it was the most challenging one.”

R3

• Recommendation 4: Contact teaching and instructor guidance are important

The data revealed that only two participants asked for help from the teachers, which should be encouraged more because they were largely satisfied with the promptness and helpfulness of the teachers’ response. The low number of participants who asked for direct help indicates that teachers need to remind students of the possibility to ask for help more actively and more frequently during the course.

Many aspects of the project that the students found difficult would be much easier with a little help from teachers. For example, the responses indicate that the students differed in terms of how easy or difficult they found the task of selecting the hardware. Four students found the task easy, but two experienced it as challenging.

“I think in the beginning it was hard to, like, think about how to make the … equipment stuff because I didn’t really know anything about … engineering…”

R1

• Recommendation 5: Support for collaboration and connectivity between projects

Lastly, the data reveal that the participants developed a lively spirit of cooperation between themselves during the project. Moreover, the data also indicates a lack of interconnectivity between the individual projects and a failure to share or reuse the partial solutions derived from the different projects. This finding suggests a need to emphasize the interconnectivity and modular nature of experimental setups during the lectures and early stages of the project. In this regard, connectivity would give students reference solutions that might help them with their other projects.

5.2. Design Solution 2: Evaluation Matrix for STEM Projects (RQ2)

In the earlier cycles of the DBR, the authors identified a need for a tool that would facilitate a repeatable and uncomplicated evaluation of the students’ work (Ambrož et al., 2023). Furthermore, the tool should enable transparent evaluations, be fair to the students, and provide the teachers with insights into the evaluation results.

Higher education courses typically require quantitative evaluation, expressed in the form of a numerical grade ranging from 1 to 5. To synthesize the grade, all aspects of a student’s work need to be considered. To achieve this end, the teachers need to prepare a comprehensive list of weighted criteria for evaluation and present it to the students. If the same content within the same course is taught by more than one teacher independently, then it is essential that the criteria are well defined and readily measurable. In many cases, formative criteria can be evaluated and graded more easily than the summative criteria. For this reason, the two groups should be divided and grouped separately.

The above-described tool can be implemented in a spreadsheet (Figure 8), including formulae for multicriteria value analysis of every student (vertical columns). In addition, statistical values for each criterion used for the different students (horizontal rows) can be calculated and evaluated to provide information about the challenges associated with individual tasks. Formative criteria have a green background, while summative criteria have a red background. The evaluation matrix can be downloaded from Appendix C.

The evaluation of the students’ projects was performed using the previously described matrix (see Figure 8) after the course participants had completed all required activities and submitted the required deliverables. The grading was performed jointly by all the course teachers, who reached a consensus on the given grades using a standard 0–5 scale. The individual grades on the scale were defined as shown in

Table 2. Grades based on the grading scale.

Numerical Grade	Descriptive Grade	State of Work
0	Insufficient	Does not meet the minimum standards
1	Sufficient	Meets but does not exceed the minimum standards
2	Satisfactory	Meets and in some ways exceeds the minimum standards
3	Good	Exceeds the minimum standards with some rather significant insufficiencies
4	Very good	Significantly exceeds the minimum standards with some minor insufficiencies
5	Excellent	Exceeds the minimum standards in every criterion without any significant insufficiencies

.

Last, we describe how we conducted the evaluation. This can serve as a valuable example of how to implement an evaluation in higher education. The weighted criteria were adapted by the teachers after giving the grades. After the grading was complete, the students were shown the weighted criteria and their own grades. An individual discussion was held with each student about the use of the grading matrix and the results. The final grades reflect the students’ performance and were acknowledged and fully accepted by the students with respect to their value and the methodology.

Apart from assigning the final grades to the students at the end of the course, the grading matrix tool was also used to analyze the data regarding the study outcome for the individual tasks. Evaluation of the grading matrix in the horizontal direction helped the teachers gain insight into the difficulty of the individual tasks and identify where the students’ performance was lowest in each of those tasks. The tasks with the lowest average grade were “boldness of idea” and “testing,” followed closely by the “soundness of the idea.” This finding is consistent with the results of the subjective evaluation based on the teachers’ notes and with the results of the qualitative analysis. The students derived their ideas from previously realized projects and were for the most part reluctant to try new approaches or combine them with their own original ideas. Also, none of the students excelled in testing their finished product either to evaluate how well it aligned with the requirements or its performance and accuracy.

The matrix was discussed in more depth with each participant in the interview. At the end of the interview, the participants were presented with the grading matrix. The teacher briefly explained the grading methodology and its use for assigning the final grade. After that, each participant was presented with their own grades and asked to express their opinion about the grading matrix as a tool and to reflect on their grades. The participants agreed in their answers that the grading matrix can be an efficient tool for assigning a grade in a fair manner:

“I think it looks … solid, like having both parts, like preparing … the idea and then, like, actually applying the idea. Like, I think there’s a pretty good balance in different … points of view.”

R6

“I think it looks good and it’s, like … it emphasizes that there is so much more than just building the device and having it be ready…”

R2

“It … seems … pretty good since it measures many, many, many parts of the … designing process.”

R1

Some participants emphasized the importance of presenting the grading tool to the students at the very beginning of the course. This important aspect of the project is reflected in the following answers:

“Yeah, it looks like it [is a fair way of grading]. Yeah, at least if the … if it is somehow shared in advance, what is to be evaluated, yeah?”

R4

“I like this chart … and next time also, it would be very good to point out this at the beginning of the course, that there is a valuation for every single phase.”

R6

6. Discussion and Conclusions

Altogether, this DBR had three cycles. Each cycle focused on developing one course iteration and had a different design focus (see Figure 1), which is common for multicycle DBR projects (Edelson, 2002; Sandoval, 2004). The latest iteration of the course was also prepared and conducted based on the findings from the previous research study (Ambrož et al., 2023). In the current version, we used the Arduino microcontrollers as the foundation for the experimental setups instead of the Raspberry Pi single-board computers, which we had used in the first iteration. This allowed us to compare the two systems and generalize the findings. The comparison revealed no significant differences between the systems in terms of the design process, the learning curve, or the usability of the finished product. This result can be explained by the fact that none of the participants in either iteration of the course had any previous knowledge about either system.

The first iteration of the course was prepared and conducted during the COVID-19 pandemic, which posed a unique challenge for the course organizers as well as the participants. Most of the contact sessions had to be performed remotely, and the disruptions in electronics supply chains severely hindered the choice and availability of the components used. Nevertheless, the course organized during such circumstances prepared us for operating in emergency situations and gave us valuable experience in using the tools for remote teaching and communication. Many of the skills acquired by the teachers during the pandemic were readily applicable in the next iteration of the course, carried out in a normal situation.

The post-pandemic return to the face-to-face teaching approach enabled us to give more emphasis to the seminar. One of the important parts of this shift in focus involved student presentations of their projects and joint reflection on the results. Additionally, the possibility of being physically present in the laboratory enabled the course participants to build, refine, and test their experimental setups together and under direct supervision by the teachers, leading to a greater level of development and readiness with respect to the final products.

In cycle 1, we identified five key challenges, which we divided into the following categories: communication, guidance, learning, planning, and collaboration (Ambrož et al., 2023). In the cycle 3 course iteration, we addressed the challenges by providing better formative support. The challenges were quite well aligned with strategies set for formative assessment (Black & Wiliam, 2009). Through empirical problem analysis, we formulated five best practices to support students’ success (RQ1). The data revealed a strong need to focus on supporting step-by-step guidance when adopting an engineering approach (Karakasic et al., 2018; Zadnik et al., 2009); we thus built the Engineering Approach Excel Template (Appendix B). Other formative perspectives, such as teacher guidance, AI support, and peer collaboration through connectivity, can be arranged via pedagogical planning and communication.

Indeed, the overall improvement in communication between the course participants and the teachers and the general feedback strategy included a more energetic expression of the intent to help and more frequent and in-depth discussions. The outcome of this measure was better guidance and better overall adherence to good practices, ultimately leading to better performance by the participants. Still, the participants did not always ask for individual help, which implies a need to give them even more frequent reminders that they could benefit from such help. The challenges related to guidance were addressed more easily due to the ability to organize face-to-face events and the normalization of the electronic parts supply chains. Especially important during cycle 1 was that the teachers placed frequent and well-argued friendly pressure on the participants to finish the tasks required for each step before continuing to the next one. The challenges related to learning that we addressed were mostly targeted at improving the design process and project management skills. Clearer explanations of the steps involved in the design process, more strictly enforcing the agreed-upon timeline, and providing a step-by-step procedure for solving the problems helped students reach those targets.

However, it is important to realize that students have different needs for scaffolding. Some need it more than others. Therefore, we encourage teacher educators to involve students in the process and design together how much and what kind of support they will need during the project (Haatainen & Aksela, 2021), which would help them build an iterative feedback loop, as highlighted by Carless (Carless, 2019). Prior studies have also recommended that teachers reduce the amount of scaffolding provided during the project and that students should begin to take more and more responsibility (Black & Wiliam, 2009; Carless, 2019; Offerdahl et al., 2018).

Also, the future chemistry teachers in our study found the coding difficult because they lacked formal training in it. With respect to addressing the challenge of coding the device software, new opportunities to use the publicly available AI tools have emerged since the first iteration of the course, which poses a challenge in the sense of learning to uncritically use those tools. Thus, we made sure to present those potential challenges to students and ensure that they used the AI tools in a way that supported reaching the learning objectives of the course.

After the first iteration of the course, we identified the need to create a tool for efficiently and fairly grading the participants’ work. In the current iteration, we developed a grading matrix for assigning the students grades based on following a set of well-defined rules. Moreover, the grading matrix also readily provides information on students’ performance in the individual project tasks, which in turn helps the teachers better target their efforts at improving the course outcome. Also, the data indicates that students find formative assessment important in terms of project evaluation. With respect to RQ2, we crafted an Excel template to support the assessment of formative and summative criteria (Appendix C).

More rigorous supervision and a stronger focus on task completion resulted in more well-defined finished products. Improving the instructions on writing the article and providing feedback and assistance to the students resulted in more consistent and higher quality article manuscripts, which is especially important for final-year students because it helps them develop skills for writing their final thesis.

Prior research has convincingly demonstrated that it is challenging to adopt an engineering perspective in STEM education (Christian et al., 2021; García-Carmona & Toma, 2024; Maiorca & Mohr-Schroeder, 2020; Zaher & Damaj, 2018). However, we claim that an authentic engineering approach offers major possibilities for resolving such challenges. In this research study, we crafted several pedagogical resources, including best practices and an evaluation matrix, to support an engineering approach in education. But much research still needs to be performed. In this regard, research is especially needed on the role of an engineering approach in the teaching process and its influence on various aspects of future STEM subject teachers’ education. DBR offers a valid strategy for developing it (Edelson, 2002; Sandoval, 2004). Applying the conclusions drawn from the previous research to the new iteration of the course let us investigate the efficiency of the measures. Apart from that, the results show that the measures introduced helped improve the experience of those students participating in such a course as well as the teachers preparing and conducting the course. The findings will help us further develop the approach through future iterations of the course and assist other teacher educators in implementing authentic engineering practices as part of their higher education in STEM.