Students as Designers of Augmented Reality: Impact on Learning and Motivation in Computer Science

Abstract

In this study, we report findings from the PCBuildAR project, in which students developed augmented reality (AR) artifacts following a guided design-based learning (DBL) approach. Sixty-two students participated in the study and were either in their first year to learn about computer science or were more experienced computer science students. In terms of learning performance, only the first-year students benefited from our guided DBL approach. In contrast, the experienced students were highly motivated to learn computer science not only immediately after the intervention, but also in the long term. For first-year students, this effect was only evident directly after the intervention. Overall, the guided DBL design proved to be effective for both motivation and learning, especially for younger students. For older learners, a better balance between guidance and autonomy is recommended.

1. Introduction

In the digitally networked society, there is an urgent need to motivate and inspire students for science, technology, engineering, and mathematics (STEM) disciplines, as more and more professions related to these disciplines are emerging [1]. In principle, students are expected to be interested in disciplines such as computer science or technology education, but this interest is diminishing as school-based STEM learning is perceived as difficult, complicated, and not relevant to life [2,3]. To address this problem, researchers are calling for practice-based and learner-centered instructional methods that provide opportunities for learners to discover content on their own and design their own learning products [4,5]. Design-based learning (DBL), in which learners plan and develop their own technologies or multimedia artifacts similar to an engineering development process, has become a promising approach in this regard [6,7]. Positive effects of DBL on motivational learning outcomes and learning achievement have mostly been shown in studies about digital game design [8,9] and video creation [10,11]. The relatively new technology of augmented reality (AR) was mainly researched through a learning from media perspective to determine if learning with AR is effective [12]. However, AR technology can also be looked at from a perspective of learning with media, in which learners create AR artifacts to solve educational problems [13]. For example, students in [14] facilitated their collaboration and communication skills through AR game design, and in [15] primary students developed digital literacy skills, e.g., information management and problem solving, through AR artifact creation.

However, researchers also point out that learning with DBL can be overwhelming and then may even hinder the acquisition of knowledge and skills [16]. In addition, regarding the motivational impact of DBL, contradictory results were reported in previous studies for students of different educational levels. In [17] students, self-efficacy and situational interest decreased significantly over the period of a one-semester makerspace DBL approach. The decrease was stronger for higher-level students than for lower-level students. More generally, the authors in [17] maintain that there is little empirical evidence for the motivational effects of DBL and thus more research is needed. Comparisons of DBL between students with different characteristics are also requested in [18], investigating both affective and cognitive learning outcomes.

The aim of this research is to address these calls for more research on DBL while also contributing to the research on AR as an educational technology. Therefore, we first present a rigorously planned instructional design for DBL AR artefact creation in computer science education that aims to promote both learning performance and motivation to learn. Next, we share the results of our intervention study conducted in real classrooms as part of the PCBuildAR project. In this project, lower-level and higher-level students designed and developed their own AR artifacts, which are now available online as open educational resources. The results of the guided AR DBL approach in terms of learning performance and motivation are discussed with reference to previous research, and an outlook on future studies and implications for practice are reported.

2. Literature Review

2.1. Design-Based Learning (DBL)

Design-based learning (DBL) is a learner-centered teaching method that, similar to project-based learning, provides for longer-term engagement with a specific topic or task. At the end of this process, there is usually a product or a learning artifact that is important for the learners and that contributes to the solution of the problem stated at the beginning [3,6]. This result is also presented, discussed, and made available for public use. Learning with DBL is cyclical, nonlinear, and strongly oriented towards engineering-science design processes. The latter distinguishes DBL from project-based learning: The process in DBL requires the development and assessment of various prototypes, each of which is adapted to the newly identified requirements after testing. The goal is to develop an optimal design, but not a product that can meet all requirements [6]. DBL also connects more traditional learning contexts, such as schools and universities, with fields of practice, e.g., start-ups, companies, or research institutions. This is called trialogical learning, in which students, teachers, and experts from the field benefit from each other in developing multidisciplinary knowledge, in addition to soft skills such as communication and project management skills [19]. DBL is most often associated with the creative design of media products, as their planning and development involves both the learning of new knowledge and the acquisition of new skills [8,20]. Many studies to date have investigated how the design of digital games and video affects learning outcomes. For example, Ke [9] demonstrated that the design of digital games positively affects students’ learning outcomes in mathematics. Wake et al. [21] provide evidence for historical learning and Dishon and Kafai [22] in civic education. Regarding digital video creation, Zahn et al. [23] found positive effects on knowledge acquisition and understanding towards obesity, and Stevenson et al. [11] reported positive effects of a collaborative video project on various skills, e.g., ICT skills, dialogic communication, and reflection on teaching and learning processes. Orús et al. [10] found similar results, such as a better performance in a final test for a video-creating experimental group than for a control group in a marketing course. In addition, the students in the experimental group were also more satisfied with the course. Writing skills can be also improved through digital video making [24].

A relatively new technology that is also well suited to be designed by learners in the context of DBL is augmented reality (AR) [15,25]. AR allows digital information or objects to be superimposed on reality [26]. This application of digital objects occurs at a given time and place, and is interactive and geometrically aligned with reality [27]. The digital AR objects can contain different media formats, e.g., videos, audios, 3D models, and interactive simulations, in addition to texts and images [28]. Learners can first create these objects and then augment them (i.e., link the digital objects with real objects) using an AR studio, e.g., Areeka Studio [29]. This was implemented, for example, in [30]. Learners from English classes used smartphones and computers to record short films, photos, and audio files during their trip to Scotland and augmented them in class after their return. This involved first creating posters with trigger images (i.e., images that were linked to the digital objects), which were then scanned with an AR app to reveal the digital objects. The posters were then presented to parents at an event, giving them a much more holistic view of their children’s Scotland trip. The positive effect of such a DBL project in language learning was also found in [31], where students created a multimodal AR museum. Other approaches that combined AR and DBL can be found for interior design [25] and the development of an AR book in primary education [32].

2.2. Potentials of DBL

The potentials of DBL can be traced back to the basic assumptions of constructivist and constructionist learning theories [6,33]. DBL is more effective than traditional learning because learners work on real problems in teams and are allowed to pursue their own ideas and approaches, which in turn leads to higher engagement and thus better learning outcomes [18,34,35]. For example, Yang et al. [34] showed that the creation of course materials by students can contribute to the acquisition of knowledge, and the creation of learning tasks can also positively influence the conceptual understanding of learning content, as Vreman-de Olde et al. [35] demonstrated for students in vocational education. The studies in [7,36,37] also found positive effects of DBL on learning performance.

In addition to learning, DBL appears to have a particularly positive effect on motivational and affective factors during learning [38,39]. Kiili [40] showed that the design of learning materials in the context of an online game can lead to the experience of flow, and in [41], the testing of prototypes was perceived as a very joyful activity. Erol and Kurt [42] were able to demonstrate the positive effect of DBL on the motivational experience in a posttest and in a delayed posttest. In this study, the experimental group created materials with the coding software Scratch and surpassed the control group not only in motivation, but also in learning gains. These results were also confirmed in the studies of Topalli and Cagiltay [43] and Terrón-López et al. [5]. In the latter study, it was also found that the involvement of professional experts was particularly motivating and had a positive effect on attitudes towards a future technical profession. LaForce et al. [44] came to the same conclusion, stating that problem-based projects can increase interest in a future STEM career. Zhang, Markopoulos, and Bekker [39] conducted a systematic review and found that DBL can contribute to self-regulated learning and improves self-efficacy among learners. Furthermore, DBL has a motivating effect, generates interest, and can arouse curiosity about professional topics.

2.3. Challenges Associated with DBL

However, in addition to the potential benefits of DBL and other constructivist learning approaches, researchers expressed doubts about their effectiveness. Analysis of Kirschner et al. [16] showed that learning with DBL and related methods is highly complex, and can thus overwhelm learners. Mostly, the necessary knowledge is not available to allow complex problems to be solved. Therefore, they recommend scaffolding elements in DBL and a reduction in these only after the development of basic knowledge and competences for the treatment of the problem. This is also noted by Hmelo-Silver et al. [45], who argue in favor of guided DBL to reveal the possible potentials. Sweller et al. [46] also added that problem-solving techniques have to be learned first and not through a constructivist teaching approach.

Vongkulluksn et al. [17] found evidence that the complexity of the DBL approach can also influence motivational and affective outcomes in a negative manner. They compared younger and older students learning within a makerspace for one semester and collected data on self-efficacy and situational interest at three points in time. Overall, both factors were found to be above the median of the scales indicating a positive expression during the makerspace instruction. However, a detailed examination of the data showed that both self-efficacy and situational interest scores declined among students over the duration of the semester. This affected the older students more than the younger students. Overall, the older students showed significantly lower self-efficacy scores and more negative emotions over the course of the semester compared to the younger students. Vongkulluksn et al. [17] explained the results as students recognizing and reflecting on the challenges in addition to their own overambitious goals. Zhang et al. [41] focused on emotions during DBL and found overall positive ratings. Here, the detailed results also reveal that learners vary in their experiences of emotions during the different stages of the design process, including negative feelings. This is also reflected in [39], in which the authors found that DBL can also lead to frustration, boredom, and anger, in addition to the more positive emotions of happiness and excitement.

3. The Present Study

Building on previous research on DBL, in this study we examined whether younger (first-year computer science students, 10–11 years) or older students (experienced computer science students, 12–15 years) benefit more from a guided AR DBL approach. For this purpose, we extended the results found in [17] by investigating additional motivational factors relevant to learning in STEM disciplines and by assessing the influence of the approach on learning performance in computer science education. With regard to the motivational factors, we applied the model presented in [47], with self-efficacy, self-determination, intrinsic motivation, and career motivation being relevant components to measure a student’s motivation to learn in STEM disciplines. Regarding learning performance, we investigated learner’s knowledge regarding the components of computer systems and their related functionalities. In addition, we contributed to research on AR as an educational technology by examining the potential of AR from a perspective of learning with media instead of the widespread perspective of learning from media [48,49]. Here, we provide a rigorous planned instructional design for integrating a guided DBL approach in real classroom settings to support a student’s creation of AR artefacts. Our approach was implemented from October 2019 to January 2020. As we were also interested in the long-term effect of our intervention, we asked the students again in May 2020 to complete our questionnaires on learning achievement and motivation towards STEM learning.

The following research questions are investigated:

RQ1: Our instructional design for the AR DBL approach is based on previous research, which revealed the problems of constructivist approaches without guidance (e.g., [16]). Therefore, our design can be characterized as a guided AR DBL approach, i.e., students, teachers, and partners from the professional world work closely together to solve the design problem (details in Section 4.2). Hence, we ask in RQ1 if both first-year and experienced students benefit in terms of learning performance from our guided AR DBL approach.

RQ2: With regard to the motivational effects of our guided AR DBL approach we follow the results found in [17]. Therefore, we investigated if the first-year computer science students in our study also report higher values for the motivational factors self-efficacy, intrinsic motivation, self-determination, and career motivation than older and more experienced students.

4. Method

4.1. Participants and Context

The study is part of the larger research and development project, PCBuildAR [50], in which students, teachers, researchers, and AR development experts collaboratively designed AR-enhanced learning and teaching materials for computer science education. Three experienced and technology-knowledgeable teachers from three different Austrian schools and their classes were recruited as partners for the one-year project. All three schools are designated as so-called “expert schools”, an award given by the Austrian Ministry of Education and Research to schools with a high level of expertise in digital teaching and learning. All three teachers are male, highly motivated, and also engaged in teacher training nationwide. In total, 62 students (42 girls) with an average age of 11.84 years (SD = 1.54) participated in the study. Twenty-five students (17 girls) were part of the first-year computer science class (M_age = 10.16, SD_age = 0.37) and 37 students (25 girls) were part of the experienced computer science class (M_age = 12.97, SD_age = 0.80). For four months, learners engaged with their teachers and the AR experts in the different phases of DBL, creating media products that were augmented and merged at the end to create the PCBuildAR cards for computer science education. The detailed course of this process is presented in the next section.

4.2. Guided DBL for AR Artifact Creation

The guided DBL instructional design for AR artifact creation consisted of three fully planned workshops in which teachers, AR development experts, and educational technology researchers from our lab worked together with the students. Each workshop was scheduled from 8 a.m. to approximately 1 p.m., which is a traditional school day in Austria. Between the workshops, the teachers continued working on the project during their computer science lessons. The AR experts and researchers prepared the next workshops by evaluating students’ ideas and prototype drafts. Our approach is based on the iterative design cycle proposed by Burghardt and Hacker [51] (as cited in [6]):

• Clarifying design specifications and constraints
• Researching and investigating the problem
• Generating alternative designs
• Choosing and justifying the optimal design
• Developing a prototype
• Testing and evaluating the design solution
• Redesigning the solution with modifications
• Communicating the achievements.

In the first workshop in October 2019, the driving question for the project was stated: How should an augmented reality artifact for learning in computer science education in secondary school look? As recommended in the literature, this question is open to various solutions and addresses a real-world problem [3]. Next, we gave a short presentation to introduce AR technology to the students, because they had little or no previous experience with it. Building on this, students tested existing AR materials and evaluated them using a quality check. For example, the students liked the 3D models in an AR history textbook, but they also reported that there was often a need for more in-depth auditory explanations or interaction options such as controls or choices. After discussing their evaluation results, students were organized into groups of four to generate alternative designs for AR artifacts for learning. For this purpose, we used the future technology workshop method [52], which allows learners to collaboratively develop a vison of the AR learning material for the future. The results of the workshop were recorded with the help of posters, digital presentations, drawings, and videos, and then presented to the entire class. Subsequently, the most exciting and promising approaches were selected as a group and transferred into a design of the first prototype.

In December 2019, we held our second workshop in all three project classes. In the interim, the first prototype was already created, which included the results from workshop one. Workshop two initially started with the opportunity to ask questions about workshop one and the further course of the project, and to give feedback on the course of the project thus far.

Then, the students were allowed to test the first prototype. From a technological perspective, the prototype was a marker-based AR product. This means that images act as triggers that contain the AR objects. When the trigger images are scanned with an application, the AR objects become visible on the device’s display. In our case, it was found in the first workshop that the components of a computer were difficult for students to understand because they were not very exciting and poorly visualized in traditional textbooks. For this reason, students selected trigger images with computer components that were linked to three-dimensional AR objects. Students tested the prototype, which at this stage consisted of two components with corresponding 3D models, again in the teams from workshop one, and evaluated the design with a quality check by assigning points from one to four and written comments. Subsequently, there was another panel discussion with the entire class, in which the impressions of the prototype testing were communicated verbally. For example, it was noted that the interactive buttons were still missing and the information on how the components work should be more accurate. Thus, the next step, the redesign of the previous solution, was initiated. First, buttons were added to the prototype sketches and their functions defined. Here, we jointly decided that pressing the buttons should display the information about the respective computer component. What information the various components contain was to be determined by the students. To do this, the students again formed groups and disassembled old computers (Figure 1). With the help of their smartphones and textbooks, they researched the functions and interaction of the components and made a collage at the end (Figure 2). In addition, students created audio, video, and short texts that could be used as AR overlays for the triggers. Both teachers and AR experts were on hand to provide support.

The third workshop was held at the end of January 2020. At the beginning, the focus was on the testing of prototype two. The trigger images were redesigned in color and graphics by the involved team of AR developers and named PCBuildAR cards (Figure 3). The buttons requested by the students were integrated and linked to the 3D objects (Figure 4). Tapping the left button, for example, plays an audio explanation of the component’s function within the computer system. The feedback from the students was extremely positive: they were satisfied with the interactive buttons, the overall design of the cards, and the content they had previously developed. It must be noted again that only two computer components were completed as AR artifacts; a total of eight were to be created. New 3D models were needed for the remaining components. We decided to also involve the students in this. This phase of the DBL project took place in a computer lab, whereas all other workshops were held in a usual classroom. The AR developers showed the students an online tool for 3D modeling. By demonstrating and following along, the learners were guided sequentially in the creation of their first model. After a short time, many were able to design their own simple 3D representations. The models were saved and then linked to trigger images using the Areeka Studio platform [29]. Areeka Studio is a freely accessible online tool that allows the creation of the user’s own AR materials. Required are an image that is predefined as a trigger image and a so-called overlay, in our case the 3D objects created by the students. Areeka Studio links the trigger and 3D object and, with the help of the free Areeka application, the 3D object can be made visible over the trigger image.

The plan was to hold more workshops on 3D modeling. This was no longer possible due to the COVID-19-related school closures shortly after our third workshop. For this reason, it is essential to note here that the 3D models in the final PCBuildAR cards were created by the AR developers. One workshop was not enough to teach the students the necessary skills to design such more complex models.

Nevertheless, we were able to successfully complete the PCBuildAR project with the DBL approach. The finished PCBuildAR cards, also referred to as AR artifacts by us earlier in this paper, are freely available as open educational resources to both teachers and learners [53].

4.3. Data Collection and Instruments

Knowledge of computer components and their functionalities were surveyed at three time points: at the beginning of the project in October 2019 (prior knowledge), after workshop three in January 2020 (time point 1), and in May 2020 (time point 2). Motivational values were collected at time points 1 and 2. At the last appointment, schools were still closed due to the COVID-19 pandemic, so the questionnaire was completed by students during a virtual classroom session.

Knowledge was assessed with two instruments: a questionnaire developed by the project team and an open-ended question. The content of the questionnaire was adapted to the topics covered in class and in the workshops, i.e., knowing the functionalities of the computer components and understanding how they interact in a running computer system. The questionnaire includes four single-choice questions and one multiple-choice question, for example “For which component do you pay attention to gigahertz (GHz) rate information?—Graphic card, CPU, RAM memory, monitor”. The open-ended question was: “What is the purpose of RAM memory in a computer?”. Students answered this question in their own words with a short piece of text.

Motivation to learn computer science was measured through an adapted version of the Science Motivation Questionnaire II (SMQ-II) developed by Glynn et al. [47]. We replaced the term science with computer science in the questionnaire to survey motivation to learn the subject of computer science. This has already been undertaken for other subjects, e.g., chemistry education [54]. The SMQ-II surveys the motivational constructs of intrinsic motivation, self-efficacy, self-determination, and career motivation. The intrinsic motivation scale measures the extent to which learners find learning computer science personally valuable and enjoyable. An example item is “What I learn in computer science is relevant for my life”. The self-efficacy scale measures the extent to which learners believe that they can accomplish computer science-related tasks. An example item is “I am confident I will do well on computer science projects”. The self-determination scale measures the extent to which students experience control over their learning. An example item is “I put enough effort into learning computer science”. The scale career motivation surveys the extent to which students learning in computer science is driven by the extrinsic motivator to obtain a better job. An example item is “Understanding computer science will benefit me in my career”. Each scale contains five items that were answered from 0, strongly disagree, to 4, strongly agree. The full questionnaire is available in German and English languages, as shown in Appendix A.

5. Results

5.1. Scoring, Data Analysis, and Descriptive Statistics

The five questions from the learning performance questionnaire were each assigned one point if correct. All earned points were summed with a maximum of five points to represent the learning performance of the students. The open-ended question was independently rated by two computer scientists from our lab. However, inter-rater reliability was not satisfying for all three measure points with Kappa values of 0.36 (prior knowledge), 0.44 (time point 1), and 0.21 (time point 2). Consequently, we do not include the results of the open-ended question in the further analysis.

The items of the adapted SMQ-II were assigned to their respective scale and reliability was tested. Cronbach’s alpha values for each scale can be considered satisfactory across time points: intrinsic motivation scale α = 0.87–0.89, self-efficacy scale α = 0.82–0.91, self-determination scale α = 0.87–0.90, career motivation scale α = 0.91–0.91. The scores from the SMQ-II scales were summed, for a maximum of 20 for each scale [47].

A Shapiro–Wilk test showed that most of the dependent variable’s distributions differed significantly from the norm; therefore, non-parametric methods were used to analyze the data. All data were computed in SPSS 27. Effect sizes were interpreted according to Cohen [55].

Table 1. Descriptive statistics of both groups for all measured variables.

	First-Year Students (N = 25)		Experienced Students (N = 37)
	M	SD	M	SD
Prior knowledge	2.20	0.87	3.30	1.24
Time point 1:
Learning performance	2.88	1.42	2.35	1.06
Intrinsic motivation	13.72	3.77	13.73	4.91
Self-efficacy	14.56	3.37	14.11	4.59
Self-determination	12.44	4.31	12.03	4.89
Career-motivation	13.12	4.52	12.46	5.27
Time point 2:
Learning performance	2.88	1.20	3.54	1.54
Intrinsic motivation	12.28	5.26	14.49	3.60
Self-efficacy	11.72	6.12	14.97	2.99
Self-determination	8.76	4.80	11.95	3.43
Career-motivation	12.08	4.19	12.49	4.61

represents the descriptive statistics of all variables assessed for first-year and experienced students.

5.2. Learning Performance

To investigate our first research question, we initially checked for differences in prior knowledge. As a Mann–Whitney test shows, younger first-year computer science learners (mean rank = 20.12) differ significantly from older learners (mean rank = 37.84) in terms of prior knowledge, U = 228.00, z = −3.48, p < 0.01. At time point 1, the learning performance between the first-year students (mean rank = 35.48) and the experienced students (mean rank = 28.81) no longer significantly differs, Mann–Whitney test, U = 363.00, z = −1.47, p > 0.05, d = 0.37. In addition, at time point 3, no differences between the two groups occur, Mann–Whitney test, first-year students mean rank = 26.82, experienced students mean rank = 34.66, U = 345.50, z = −1.72, p > 0.05, d = 0.44.

Furthermore, a Wilcoxon signed-rank test shows that the first-year students scored significantly higher in the learning performance test at time point 1 (Mdn = 3.00) compared to their prior knowledge (Mdn = 2.00), z = −2.16, p < 0.05, d = 0.62. This result was also found when comparing first-year students’ prior knowledge with the learning performance at time point 2 (Mdn = 3.00), Wilcoxon signed-rank test, z = −2.05, p < 0.05, d = 0.59.

The older and experienced students significantly worsened their performance at time point 1 (Mdn = 2.00) compared to their prior knowledge (Mdn = 3.00), Wilcoxon signed-rank test, z = −2.98, p < 0.05, d = 0.82. At time point 2, the experienced students improved their performance on the learning test again (Mdn = 3.00); however, compared to their prior knowledge, this improvement was not significant, Wilcoxon signed-rank test, z = −0.54, p > 0.05, d = 0.14.

5.3. Motivation

A Mann–Whitney test was used to determine differences between the first-year and experienced students for both time points.

At time point 1, all students showed similar high values for all four motivational scales, no significant differences were found for intrinsic motivation (first-year students mean rank = 30.66, experienced students mean rank = 32.07, U = 441.50, z = −0.30, p > 0.05, d = 0.08), self-efficacy (first-year students mean rank = 31.24, experienced students mean rank = 31.68, U = 456.00, z = −0.09, p > 0.05, d = 0.02), self-determination (first-year students mean rank = 31.98, experienced students mean rank = 31.18, U = 450.50, z = −0.17, p > 0.05, d = 0.04), and career motivation (first-year students mean rank = 32.86, experienced students mean rank = 30.58, U = 428.50, z = −0.49, p > 0.05, d = 0.12).

At time point 2 no significant differences between the first-year (mean rank = 27.54) and experienced students (mean rank = 34.18) were found for the intrinsic motivation scale (U = 363.50, z = −1.43, p > 0.05, d = 0.37) and the career motivation scale motivation (first-year students mean rank = 29.58, experienced students mean rank = 32.80, U = 414.50, z = −0.69, p > 0.05, d = 0.18). Significant differences were detected for self-efficacy (first-year students mean rank = 26.32, experienced students mean rank = 35.00, U = 333.00, z = −1.87, p < 0.05 one-tailed, d = 0.49), and self-determination (first-year students mean rank = 24.82, experienced students mean rank = 36.01, U = 295.50, z = −2.41, p < 0.05, d = 0.64).

Regarding the long-term effect of the guided AR DBL approach, the Wilcoxon signed-rank test was used to calculate the differences of the motivational values between time points 1 and 2 for both groups. For first-year students, no significant decline or increase in intrinsic motivation (time point 1: Mdn = 15.00, time point 2: Mdn = 13.00, z = −0.92, p > 0.05, d = 0.26), self-efficacy (time point 1: Mdn = 15.00, time point 2: Mdn = 15.00, z = −1.75, p > 0.05, d = 0.50), or career motivation (time point 1: Mdn = 13.00, time point 2: Mdn = 13.00, z = −1.01, p > 0.05, d = 0.28) was found. For self-determination, a significant decline in first-year students’ scorings was found (time point 1: Mdn = 12.00, time point 2: Mdn = 10.00, z = −2.79, p < 0.01, d = 0.83).

For experienced students, none of the motivation scales significantly declined or increased between time points 1 and 2: intrinsic motivation (time point 1: Mdn = 15.00, time point 2: Mdn = 15.00, z = −0.22, p > 0.05, d = 0.06), self-efficacy (time point 1: Mdn = 15.00, time point 2: Mdn = 16.00, z = −0.59, p > 0.05, d = 0.15), career motivation (time point 1: Mdn = 12.00, time point 2: Mdn = 13.00, z = −0.06, p > 0.05, d = 0.02), and self-determination (time point 1: Mdn = 12.00, time point 2: Mdn = 12.00, z = −0.36, p > 0.05, d = 0.09).

6. Discussion

The objective of this study was to extend previous findings of DBL when applied in real classroom settings and to contribute to the research field of AR, by showing that AR can also be looked at from a perspective of learning with media. Therefore, we developed a rigorous planned instructional design for students to design their own AR artefacts, and collected data, related to the learning performance and motivation to learn in computer science, of first-year and experienced students.

In the first research question, we asked whether first-year and experienced students would benefit from our guided AR DBL approach regarding learning. Our results show that the first-year students with less prior knowledge significantly improved their learning performance after the intervention and gained on the experienced students, so that at time point 1 there was no longer a difference between the groups. The effect size for the improvement was medium. In addition, in the long term, the first-year students significantly improved their learning performance compared to their prior knowledge; the effect size was again medium. In comparison to time point 1 (M = 2.88), first-year students achieved the same scores at time point 2 (M = 2.88). Hence, we can conclude that the guided AR DBL approach helped the first-year students with little prior knowledge to improve their learning performance. This is in line with previous research that indicates that guided learner-centered instructional approaches work for students with low prior knowledge [45,56]. In contrast, the experienced students’ performance significantly worsened at time point 1 (M = 2.46) compared to their prior knowledge (M = 3.25); the effect size was medium. One explanation for this effect might be that the guided instructional design does not allowed the experienced students to align their prior knowledge with the newly learned information, resulting in a cognitive conflict [57]. This conflict triggers processes in learners that are cognitively highly demanding: existing mental models have to be reconsidered and the new information has to be reconciled with existing knowledge. If these processes are still in progress during a performance assessment, cognitive overload can occur, resulting in poorer learning performance [57]. The performance results of the experienced students at time point 2 (M = 3.50) can be interpreted as evidence for this explanation. The conflict processes were completed during performance measurement at time point 2, and the learners were able to retrieve their previous performance. However, it must also be noted that neither a significant improvement compared to prior knowledge (small effect size) was found, nor did a significant difference between the experienced and the first-year students learning performance reoccur. Therefore, we conclude that the guided AR DBL approach was not an effective instructional method for the more experienced students with high prior knowledge, which is in line with research on the expertise reversal effect [57].

Our second research question is based on previous findings reported in [17]; hence, we assumed that the motivation of the younger first-year students to learn computer science would be higher compared to the experienced students after the AR DBL intervention. Descriptive statistics support this assumption for time point 1, where first-year students reported higher values for all four motivation scales. However, the differences were not found to be statistically significant, and the effect sizes were small. Hence, we can summarize that the guided AR DBL approach may have helped the experienced students to not overestimate their abilities, which was identified in [17] as a possible factor for the motivational decline.

More interestingly, at time point 2, significant differences with medium effect sizes were found for intrinsic motivation, self-efficacy, and self-determination, and not for career motivation. Here, the experienced students reported higher values four months after the AR DBL intervention, with no decline compared to the results at time point 1. On the contrary, first-year students’ intrinsic motivation, self-efficacy, and career motivation values decreased (not significantly) with small to medium effect sizes. The self-determination scale values even diminished significantly, and the effect size was large. With regard to the experienced students, the results are promising, because otherwise the motivation to learn in STEM-related disciplines gradually drops [58]. In our case, the guided AR DBL approach may have contributed to maintaining high motivation in the longer term and stimulated a sustained interest in computer science education. This is line with previous research on the motivational effect of DBL for STEM-learning [7,39,40]. Regarding the younger first-year students, a positive long-term effect of the guided AR DBL approach is not as clear as that of experienced students. Particularly striking is the significant drop in scores for self-determination (large effect). The intervention was perceived by the first-year students as autonomy-enhancing, whereas the period after the intervention, when “traditional” teaching was resumed, undermined this autonomy again. Self-determination is an essential factor in human motivation that correlates with the other motivational factors measured in this study [47], especially the sense of intrinsic motivation. In the self-determination theory of motivation [59], the experience of autonomy is a central factor of intrinsic motivation, in addition to the experience of competence and relatedness. The guided AR DBL approach presented here contributed to the experience of self-determination, so implementation of this design, especially with younger students, can be strongly recommended.

7. Limitations and Future Research

The results of this study are based on the specifics of our project, such as the highly committed teachers and their selection of the classes. Therefore, a generalization of the results is not possible. In addition, our implementation is a field study under real classroom conditions, rather a controlled randomized experiment. For practice, however, studies such as the one conducted here may be helpful because they provide a blueprint for how DBL can be implemented in school teaching. Another limitation concerns the sole focus on the DBL use, which was separated from what happened before and after this teaching approach. It would be exciting to investigate the point at which the learning of new content DBL would be particularly well suited. For example, it is reasonable to assume that content pre-training could have a positive impact on motivational factors. Such a design could then be compared to a class that was not prepared in terms of content. With regard to our findings, future studies should further investigate the balance between guidance and autonomy or should identify the optimal point in time during a DBL approach to reduce the supportive guidance. Additionally, the small sample size should be mentioned as a limitation of a pure quantitative analysis. Future studies might apply design-oriented mixed-method approaches to gain more detailed insights into how learning happens when using DBL for AR creation.

8. Conclusions and Implications

Based on the results of this study, we can conclude that our guided AR DBL approach for computer science education proved to be more effective in terms of learning performance for younger first-year students with less prior knowledge than for more experienced students with higher prior knowledge. For experienced students, a more balanced design of guidance and autonomy is recommended. Then, these students can use their knowledge base in a more effective way, and thus prevent a cognitive conflict.

In terms of motivation to learn computer science, the results are promising for experienced students, because motivation remained high in the longer term. For the first-year students, a long-term effect was not as clear, but we found evidence that the guided AR DBL approach contributed to student’s motivation to learn computer science after the intervention. This is also true for the older and more experienced students.

For teaching practice, we can recommend the guided AR DBL design presented here, especially for beginners, because they will benefit from the structure. When used with older students, it is recommended to allow more freedom so that cognitive conflict is avoided, and AR design is seen as an exciting challenge.