D-Learning: An Experimental Approach to Determining Student Learning Outcomes Using Augmented Reality (AR) Technology

Abstract

This study investigates the impact of applying digital learning through the use of augmented reality technology in university education on student learning outcomes. The research strategy and design were developed with the aim of answering the following research questions: Do different forms of active learning using augmented reality (AR) influence student knowledge?; Is it possible to assess the impact of AR implementation in teaching on the quality of learning outcomes? The main goal of his research is to identify the information objects Situated, Games, and Research within the Active Learning information package and, based on them, examine and determine the existence, relationship, and intensity of connections of the created information construct Knowledge. The research was conducted through the application of an experimental technique with incorporated survey and knowledge test methods on a sample of 270 participants, in 3 groups of 90 students each. Knowledge was examined through initial tests and final tests for each form of the conducted experiment: Situated, Games, and Research. The obtained results were processed using statistical methods of calculating correlation coefficients and factor analysis. The results indicate that students taught through all forms of AR active learning achieve a statistically significantly higher level of knowledge, thus confirming the hypothesis. In conclusion, the obtained research results provide a basis for further research using the described methodology and the further development of educational application solutions based on new AR technologies.

1. Introduction

New technologies impact learning and teaching outcomes and can be utilized to create interactive learning environments that inject excitement into an active learning process [1,2,3]. Simply put, active learning is suitable for any pedagogical methodology concerned with how students learn, placing the student at the center of the learning process. Bonwell and Eison [4] popularized the concept of active learning by stating that “…in active learning, students will be actively engaged in the learning process”. They also suggested that “active learning involves students in two aspects: doing things and thinking about the things they are doing”. Active learning follows the principles of constructivism, which propose that students construct or build their understanding by “making meaning” of newly acquired information, connecting new ideas and experiences with previously acquired knowledge and practices to form new or enhanced knowledge and achieve deeper levels of understanding [5]. Students are then better able to analyze, evaluate, and synthesize ideas, reaching higher cognitive levels. Active learning also develops students’ autonomy and their ability to learn, helping them become “lifelong learners” as they gain greater control over their learning process.

As interest in new educational methods grows, many academics consider active learning to be an important and effective approach to promoting the development of higher cognitive skills in students, referred to as “deep learning”, as referenced in Bloom’s taxonomy [6]. Active learning methods can be time-consuming for teachers and may present challenges in implementation. Various types of new technologies are now available that could theoretically assist teachers in developing active learning experiences and enable students to become actively engaged both inside and outside the classroom. However, there is a significant problem with this in practice. Repeated research shows that, even after many years of technology adoption, many teachers decide not to engage with it at all or choose to use new technologies in a very limited way. Letina [7] states that “active learning aims to achieve a higher degree of independence, the application of various cognitive strategies, and the development of specific cognitive skills that enable the identification of essentials, analysis and comparison of information, connection with existing knowledge, and critical judgment of their meaning” [7] (p. 8).

The value of the concept for the development of active learning in higher education is also recognized in new technologies such as augmented reality (AR) and virtual reality (VR). Indeed, AR technology has been recognized as one of the more important new technologies for the development of active learning in higher education [8] (pp. 503–520). The fact that some higher education institutions use augmented reality and virtual reality in teaching serves as motivation for their more frequent use in innovative ways that expand educational dimensions [9] (pp. 2540–2547). Furthermore, AR and VR technologies provide an opportunity to overlay images, text, audio, video, and audio–video recordings onto existing AR and VR structures, onto real-time spaces or images, and in communication with students [10] (pp. 249–254).

The application of augmented reality (AR) and virtual reality (VR) technologies in higher education is compatible with certain pedagogical approaches that fit within the framework of active learning: constructivist learning, situated learning, game-based learning, and research-based learning [11] (pp. 8745–8752), [12] (p. 273). Compatibility with constructivist learning is reflected in the interaction between the student and the learning environment and in the construction of connections with the student’s prior knowledge. In line with the approach of situated learning, AR holds particular significance as it allows for the real-time and real-world display of virtual objects that students would not otherwise encounter and that would be difficult to visualize. In the game-based learning approach, students can play an AR game in a role that prepares them for everyday life, while in research-based learning, students explore virtual models and experiment with them. Furthermore, the use of augmented reality can achieve learning experiences beyond the classroom, contextualizing the connection between reality and the learning situation in which students participate, with any physical space becoming a stage for active learning [13] (pp. 1–15), [14] (pp. 394–403). Authors Akçayir and Akçayir [15] highlight the potential of using augmented reality in teaching. Among other things, they mention the positive effects of AR application, which are reflected in a better understanding of the teaching content, emphasizing that the main topics are learned through the effective design of active learning: situated learning, game-based learning, and research-based learning.

Situated learning involves learning in authentic situations, and contextualized learning is enabled by embedding educational experiences into the real environment and bringing the real world into the classroom [16,17,18,19]. Social constructivist theory is a theory in which effective learning occurs in authentic environments and involves social interactions. One well-known approach of social constructivism is situated learning. Situated learning has had a significant impact on educational thinking since it was first expounded by Brown and colleagues [20] (pp. 32–42). Unlike most learning activities in classrooms that involve abstract knowledge detached from context, Lave and Wenger [21] argued that situated learning is embedded in activity, context, and culture, as a process of “legitimate peripheral participation” [21] (p. 29). Among others, Albert Bandura is considered a leading advocate of social learning theory, which suggests that observation and modelling play a primary role in this process, elaborating that learning is a social process as one can “acquire extensive integrated patterns of behaviour without tedious trial-and-error shaping” [22] (p. 12). Bandura also states: “Learning is bidirectional: we learn from the environment and the environment learns and changes through our actions” [22] (p. 12). Dede [23] suggests that tasks that are difficult to learn should be learned through observation. Simply put, situated learning occurs in the same context in which it is applied. Technology can play a significant role in integrating 21st-century skills and mediating authentic experiences in the classroom [24] (pp. 76–101). Simulations and virtual reality provide a basis for one form of situated learning by modeling specific aspects of complex systems in the real world. Users can experiment with the system or manipulate parameters by participating within the system and observing the outcomes of their manipulations. Simulations situated in lush, realistic 3D virtual worlds can be described as “heavy” virtual, but they are less authentic because they depart so much from the real world. In contrast, “light” virtual information learning environments provide less simulated sensory input but remain closer to the real world and can capitalize on their potential for authenticity [25] (pp. 31–45). Mixed reality (MR) technology can be used to balance the strengths and weaknesses of virtual media in creating an authentic learning environment, combining scenes from the real world with virtual objects and live social interactions with other participants. More intuitive ways of interacting with virtual objects using MR help users acquire authentic experiences.

Augmented reality (AR) systems can be used to facilitate immersive game-based learning by creating digital narratives, placing students in roles, providing authentic sources, and embedding contextually relevant information [19,26,27]. The use of augmented reality systems to transform the real world into a gaming environment can often simplify and facilitate the transfer of skills into real-world scenarios [28] (pp. 125–132). Game-based learning approaches have been proposed as a “post-progressive” pedagogy that could situate students in complex cognitive tasks guided by authentic questions, involve multiple tools and resources, rely on learning by doing, guide students through events and modes of thinking, and require complex performances in order to demonstrate skill and competence [29] (pp. 86–107). Advocates of game-based approaches have sought to combine socio-cultural learning approaches with modern computing and video games [30,31,32]. Specifically, students can use these educational games for experiential learning and develop decision-making and problem-solving skills in a dynamic learning environment [33] (pp. 1041–1052). Additionally, students can receive immediate feedback and/or results instead of receiving delayed feedback through traditional assessment methods (e.g., tests and exams) by visiting the teacher’s office for consultation. Applications of these models suggest that the fundamental feature of educational games may involve decision-making cycles, experiencing consequences, interpreting game systems, building periodic experience narratives, multiple experiences within the system, and then building a cognitive model of the game system as a result of game-based learning [34] (pp. 307–318). Furthermore, other software applications that can be used to support education are full-fledged video games, which lead us to the paradigm of gamified learning. Gamification has been at the center of attention in education in recent years. “Gamification is the practice of using game design elements, while game mechanics and game thinking provide activities in the game to motivate participants” [35] (p. 133). Many have suggested full-fledged video games that can effectively support traditional curriculum-based education for use in various educational contexts in higher education [36,37,38] (pp. 250–262 of [37]). Most full-fledged video games deviate from e-learning tools and the basics of edutainment (educational entertainment content for learning) through games.

Furthermore, players immerse themselves in complex and rich environments, allowing them to explore numerous strategies for action and decision-making, which require them to perform challenging tasks set with increasingly difficult goals [39] (pp. 10–66). Some researchers go further, arguing that game-based learning involves processes that differ to such an extent from learning in other forms (such as classroom teaching) that they should be described as a unique model or theory of learning [40] (pp. 258–283). A review of existing games quickly confirms that the uniqueness of game-based learning is difficult to define at the epistemological level. Game designers use behaviorist elements, cognitive elements, and constructivist elements, often combining them in the design of educational games. Others argue that games help develop strategic thinking, group decision-making, and higher cognitive skills [41,42] (pp. 1–17 of [41]). Overall, games seem to be particularly useful for creating a deeper understanding of certain key principles of given topics, mainly when dealing with complex and multiple issues that are difficult to grasp through factual knowledge.

Research-based learning provides a way for electronic data collection for future analysis [19] (pp. 7–22) and offers virtual models situated in the context of the real world that are easily manipulated [43] (pp. 339–345). Augmented reality supports research by providing information that is contextually relevant to the topic being investigated [44] (pp. 15–17). New technologies offer an active potential for facilitating understanding and preventing ambiguities in the scientific domain through visualization [45] (pp. 214–220). Examples of new technologies such as augmented reality (AR) and virtual reality (VR) examined in previous research include animation, virtual environments, and simulation. Dede and colleagues [46] suggest that students can improve their mastery of abstract concepts by using virtual environments designed for learning. Specifically, AR provides an efficient way to present a digital object that requires visualization [47]. AR technology also supports seamless interaction between real and virtual environments and enables the use of tangible interface metaphors for manipulating digital objects. The findings of Tuta and colleagues [48] suggest that AR technology significantly contributes to active learning by stimulating research-based learning. Students in research-based learning explore virtual models and experiment with them. Furthermore, digital objects with interesting data in the real-world context of augmented reality help students understand the results of information analysis. AR technology can be used to address posed challenges, explore digital objects situated in the context of the real world, arrive at new insights, derive results, and thus help students have a better perception of active learning by using augmented reality. AR technology offers flexible learning, meaning it can be used by both teachers and students in the classroom and/or by students independently exploring subjects at home. An example of using research-based augmented reality is the Vuforia application, which uses computer vision technology for scanning, recognizing, and tracking its target. Bernik and colleagues [49] argue “that the advantage of this approach is that the user is at the center of attention, and during research, their own movements and decisions influence the information displayed on the screen.” We can conclude that research-based learning using augmented reality provides direct access to information about digital objects located in the natural environment.

A recent example of current research on the application of AR in higher education points to the significance of teaching effectiveness and students’ ability to develop digital literacy [50] (pp. 267–276) and digital competencies that have a major impact on learning and teaching in university education [51] (pp. 1–12). Augmented reality has been recognized for several years as one of the technologies with a greater impact on higher education. This is particularly important with immersive technologies (AR and VR media) because they rely on parallel effects in the real world. Most of these effects are triggered by students’ actions as AR users, especially in situations where during teaching the student needs to look in a certain direction or use controls to manipulate a virtual object [52] (pp. 89–104). The application of augmented reality in higher education significantly influences contemporary education, not only in the acquisition of knowledge and skills by students but also in the transfer of knowledge and skills from teachers to students [53] (pp. 26–34). Furthermore, based on a review of existing research, it can be concluded that learning in an AR environment aligns with concepts of educational theory that emphasize learning as the result of the connection between stimuli and responses to the stimuli. An AR-based active learning platform provides students with a choice of tools and scenarios for creating models, designed to be easily used (pp. 525–541 of [54]), (pp. 627–636 of [55]). In recent years, learning through interaction with virtual objects integrated into augmented reality has become a modern way of digital learning. Radu [56] also highlights in his research that these practices promote higher performance in active learning and are associated with their creative, motivational, and motor potential, and the strength of the immersive feeling of experience [56] (pp. 1533–1543). Several studies by researchers [57,58,59,60], aiming to analyze the use of augmented reality in the educational context of active learning, highlight positive examples of AR application in teaching, with learning motivation being the most prominent factor. O’Flaherty and Phillips [61] in their work analyze published studies on the implications of using AR technologies and conclude that AR facilitates an understanding of complex phenomena and concepts; promotes contextualization and enrichment of information; enables the individualization of practical teaching and adaptation to different forms of digital intelligence; provides students with the ability to communicate via AR and manipulate real objects; favors ubiquitous and contextualized learning by transforming any physical space into a stimulating academic environment; facilitates the development of constructivist teaching and/or learning methodologies; promotes the development of graphic skills through the perception of spatial content and 3D objects; benefits experiential learning; and increases motivation and improves learning outcomes. Recent research suggests that scientific studies and reports proposing the introduction of XR technology trends into teaching should also be accepted in university settings as soon as possible, with the aim of changing teaching practices [62,63,64]. Additionally, modern research introduces new technologies into university education for students and teachers who must adapt to the challenges and demands of contemporary digital society, taking into account emerging perceptions and trends that students today and professionals tomorrow will encounter in their immediate work environment [65].

Most of the research on active learning has focused on problems, challenges, attitudes, and perceptions [66]. While these studies provide some insight into the problem of implementing active learning in student teaching, they do not offer a solution on how to systematically validate learning outcomes achieved through the use of AR in teaching, nor do they accept the information package of active learning and its information objects, the conceptualization of which will be considered in this paper, the aim of which is to assess the impact of active learning on learning outcomes. The purpose of the research was to critically examine the assessment of the impact of active learning using augmented reality on the knowledge of the cadets at the Military Academy “Dr. Franjo Tuđman”. In order to achieve this purpose, this research aimed to determine the best predictors of academic success of active learning using augmented reality on the cadets’ knowledge and to propose a model for improving academic success. The research goal is to identify the information objects of Situated, Games, and Research from the Active Learning information package and, based on them, examine and determine the existence, relationship, and intensity of the relationships created by the information constructs Perception and Knowledge by assessing the impact of active learning using augmented reality. The strategy and design of the research were developed to address the following research questions and determine the truthfulness of the hypothesis:

Q1. 

Do different forms of active learning using AR affect cadets’ knowledge?

Q2. 

Is it possible to assess the impact of AR on the quality of learning outcomes?

H1. 

Cadets who were taught through all forms of AR active learning (Situated, Games, and Research) achieve a statistically significantly higher level of knowledge compared to cadets in the other two groups.

2. Materials and Methods

The research concerned was conducted with the consent of the Ministry of Defense of the Republic of Croatia, Administration M-2 regarding ethical permissibility. At the beginning of the research, participants were informed that their participation was voluntary and that they had the right to withdraw from the study at any time during the experiment without any sanctions. Procedures, principles, and ethical issues related to the collection, analysis, and interpretation of data from the experiment were carefully monitored to ensure the credibility of the research outcomes in all phases of the research.

In the experiment with cadets (participants), for “Group 1, 2, and 3”, work was carried out on the educational thematic unit “Knowledge of artillery weapon D-30 122 mm Howitzer”, within which the following educational materials and skills were addressed: technical skills (use of the HaubicAR application), photovisual skills (identifying parts of the Howitzer, describing the process of assembling and disassembling the Howitzer), and cognitive skills (response time and accuracy of responses).

The mobile application HaubicAR was developed for the experiment concerned in collaboration with the Faculty of Electrical Engineering and Computing, University of Zagreb [66]. The first version of the application included a display of a simple 3D model of the D-30 122 mm Howitzer in augmented reality, a list of parts of the model with a brief description, a quiz with questions related to the parts of the Howitzer, and a link to a YouTube video demonstrating the assembly of the D-30 122 mm Howitzer from the transport to the combat position. In addition to technical changes, an external solution was implemented to test the cadets before using the application developed for this experiment. The external solution was implemented on the Heroku server to track the time of entry and exit from certain activities within the application. All improvements to the application and implemented external solutions serve as additional means for easier and automated execution of the research concerned. The researcher proposed an information design for the experiment and research on the impact of active learning using augmented reality, which was accepted during the development of the HaubicAR application. Information design refers to the effective presentation of information, which includes creating application solutions that visualize data and present them in ways that allow for their perception and understanding of new technologies. Information design uses a list of parts of the given object, symbols, signs, images, moving images, digital text, a quiz, and a 3D model to describe the digital object in the application solution. The application was developed using the visual scripting method within the Unreal Engine, called Blueprint (2022). Flask’s (2022) web application developed in Python <3.7> (2022) was used as an external solution for tracking user activities, with a PostgreSQL (2022) database hosted on the Heroku (2022) server. Google Forms (2022) quizzes were used for the purpose of research and testing the cadets’ knowledge. Blender’s (2022) tool was used for optimizing the Howitzer model. The application was packaged in the form of a .apk file and transferred to a mobile device (tablet or smartphone), and then the installation was initiated. In the experiment concerned, the researcher used Xiaomi tablets, Pad 5, 6/128 GB. After installation, the user is presented with the initial screen with the application name, a field for entering the user number, and buttons for selecting the group to which they belong. This is important because within the research there are three groups with different materials and forms of learning: Group 1—Situated, Group 2—Situated + Games, and Group 3—Situated + Games + Research. This approach was chosen to determine the impact of different learning methods on the effectiveness of acquired knowledge related to the Howitzer.

The research was conducted using the field method, employing the following techniques: experiment, survey questionnaire, and knowledge test. The study was carried out with cadets of undergraduate and graduate military studies in Military Engineering and Military Leadership and Management, including students from the first, second, third, fourth, and fifth year of study, totaling N = 270. The research plan included four phases: (1) Preparation and implementation of the analysis of information concepts of active learning, (2) Conducting surveys, experiments, and knowledge tests, (3) Data collection and processing, and (4) Interpretation of results and assessment of the impact of active learning through augmented reality on learning outcomes, with the methodology of implementation described below.

2.1. Phase 1: Analysis of Information Concepts of Active Learning

In the first phase of the research, basic student data were collected from cadets currently enrolled in the university-level military undergraduate and graduate studies of Military Engineering and Military Leadership and Management at the University of Zagreb, based on available data from the Research Office in the Cadet Battalion, with the approval and consent of the the Croatian Military Academy “Dr. Franjo Tuđman”, the Headquarters of the Armed Forces of the Republic of Croatia, the Personnel Directorate-J1, and the Ministry of Defense of the Republic of Croatia, Sector for Human Resources Development and Management M-2. A survey questionnaire was created using Google Forms with independent variables of gender, study program, and year of study, and dependent variables of immersive experience, familiarity with (XR) technologies, perception of the application of augmented reality in teaching, prior knowledge of (AR) applications, and experience in using the HaubicAR application. The learning outcomes that cadets were expected to achieve after the sample classroom exercise were also defined, and a knowledge test was created.

2.2. Phase 2: Implementation of the Survey, Experiment, and Knowledge Test

In the second phase of the research, a survey was conducted on the defined sample of participants, along with an experiment using the HaubicAR application and a knowledge test. The initial test measured the knowledge acquired before exposure to AR active learning, while the final test measured the knowledge acquired after actively participating in a specific form of AR active learning. Randomization was achieved for the purpose of conducting the survey, where cadets were randomly divided alphabetically into three groups of participants, with an equal number of members for each form of the conducted experiment: Group 1: Situated learning, Group 2: Learning through games, and Group 3: Research-based learning. The purpose and goal of the pilot classroom exercise were presented. The survey, initial test, experiment, final knowledge test, and test results were conducted through a pilot classroom exercise using ten Xiaomi Pad 5 tablets, 6/128 GB. The first part of the experiment consisted of a survey, comprising open-ended and closed-ended questions, which examined the cadets’ knowledge of the capabilities and facts related to the application of augmented reality. In the second part, the cadets completed the initial test, aimed at verifying previously acquired knowledge. The initial test consisted of a total of eleven questions related to the topic “Technical and Operational Characteristics of the D-30 122 mm Howitzer”, which participants answered using the application previously installed on the tablets. The second part involved the experiment for “Groups 1, 2, and 3” using the HaubicAR application. Each group (1, 2, 3) consisted of nine subgroups, each with ten participants. A total of 270 participants took part in the experiment. The experiment took place in a classroom where ten participants were present simultaneously. Each group was assigned different forms of active learning through the pilot classroom exercise on tablets using the HaubicAR application, but the topic for all three groups was the same: “Technical and Operational Characteristics of the D-30 122 mm Howitzer”.

2.2.1. Group 1: Situated

Group 1: Situated was given the task of studying digital material on the given topic in PDF format. Through situated learning, participants were provided with the opportunity to learn about a topic embedded in augmented reality through educational activities in a real environment. Participants had a specific time to study the digital material in order to proceed to the final test after the experiment.

• 00:00–00:05 = Presentation of the purpose and objectives of the classroom exercise;
• 00:05–00:15 = Conducting the survey;
• 00:15–00:20 = Initial test of knowledge;
• 00:20–00:40 = Experiment: Situated;
• 00:40–00:45 = Final test of knowledge;
• 00:45–00:50 = Test results.

2.2.2. Group 2: Situated + Games

Group 2: Situated + Games engaged in a quiz game. Participants utilized immersive learning on the given topic through a quiz game where authentic sources and contextually relevant information were embedded. Participants had a specific time for the quiz game in order to proceed to the final test after the experiment.

• 00:00–00:05 = Presentation of the purpose and objectives of the classroom exercise;
• 00:05–00:15 = Conducting the survey;
• 00:15–00:20 = Initial test of knowledge;
• 00:20–00:45 = Experiment: Situated + Games;
• 00:45–00:50 = Final test of knowledge;
• 00:50–00:55 = Test results.

2.2.3. Group 3: Situated + Games + Research

Group 3: Situated + Games + Research had the Howitzer D-30 122 mm in the form of a digital object in augmented reality while learning through research, and they explored and experimented with it. Participants engaged in immersive learning through visualization that was contextual to the given research topic. Participants had a specific time for research in order to proceed to the final test after the experiment.

• 00:00–00:05 = Presentation of the purpose and objectives of the classroom exercise;
• 00:05–00:15 = Implementation of the survey;
• 00:15–00:20 = Initial test of knowledge;
• 00:20–00:50 = Experiment: Situated + Games + Research;
• 00:50–00:55 = Final test of knowledge;
• 00:55–01:00 = Test results.

2.3. Phase 3: Data Collection and Processing

The data were collected through the HaubicAR application, in which the Google Forms tool was integrated. This allowed us to collect data using the survey questionnaire, the initial test of knowledge acquired before exposure to AR active learning, the experiment, and the final test of knowledge acquired after active participation in a specific form of AR active learning. In this study, the collected data were analyzed using quantitative methods. After preparing the data from the HaubicAR application, the researcher analyzed the obtained data using the statistical package IBM SPSS 25. The Cronbach’s alpha coefficient was applied to the study to determine the validity and reliability of the instruments used. The Cronbach’s alpha coefficient of internal consistency was 0.72 and indicated satisfactory reliability of the scale. Descriptive statistics (arithmetic mean—M and standard deviation SD) were calculated for individual items of the perception scale; correlations with the total score on the scale were corrected (r_it—corrected item-total correlation); and Cronbach’s alpha was calculated after excluding an individual item. It is recommended that the corrected correlations with the total score on the scale are greater than 0.30, which is satisfied in this analysis. By excluding particle A5, the Cronbach alpha coefficient would increase at the second decimal point from 0.72 to 0.73. Given that the reliability coefficient is satisfactory and that the exclusion of particle A5 would not result in a greater increase in the reliability of the scale, this particle was retained.

In the data processing phase and interpretation of results, the method of frequency of occurrence was used through the application of inferential and descriptive statistics (frequencies, percentages, measures of mean value, and variability). Based on the analysis of the results obtained from the survey, initial test, and final test of knowledge, informational constructs were created: Perception and Knowledge. Perception was examined through the survey using the following factors: resistance to the use of augmented reality, the significance of augmented reality application in teaching, and perception of the role of AR in acquiring knowledge and skills. A Likert scale and descriptive statistical methods were used for analysis. Knowledge was examined through initial tests and final tests of knowledge for each form of the conducted experiment: Situated-, Games-, and Research-based learning. The dependent variables Perception and Knowledge were determined using the following factors: technical skills (use of the HaubicAR application), photovisual skills (naming parts of the Howitzer, naming the assembly and disassembly process of the Howitzer, and cognitive skills (response time, accuracy of responses). The processing of the obtained results was carried out using statistical methods such as correlation coefficient calculation and factor analysis.

2.4. Phase 4: Impact Assessment of Active Learning Using AR

In the final phase of the research implementation, an assessment of the impact of active learning using augmented reality on learning outcomes was undertaken. The impact was described using the information objects Situated, Games, and Research, and was estimated to be determined by the sequence of the information constructs Perception→Knowledge. The procedure was carried out using defined learning outcomes for the experimental classroom exercise using HaubicAR. By applying a model whose constructs were Perception and Knowledge, the learning outcomes were validated. Perception was examined through a survey using the following factors: resistance to the use of augmented reality, the significance of augmented reality application in teaching, and perception of the role of AR in acquiring knowledge and skills. A Likert scale and descriptive statistical methods were utilized for analysis. Knowledge was assessed through initial tests and final tests of knowledge for each form of the conducted experiment: Situated, Games, and Research.

3. Results

3.1. Initial Test

The results of the initial test of knowledge were obtained by summing the achieved scores, where each correct answer to a question scored one point and incorrect answers scored zero points. In this regard, cadets from the Situated + Games group achieved the lowest average score on the initial knowledge test (M = 5.12, SE = 0.17, 95% IP [4.78, 5.46]). According to the average number of points, cadets from the Situated + Games + Research group followed (M = 5.43, SE = 0.17, 95% IP [5.10, 5.77]). The best result (the highest average score) on the initial test of knowledge was achieved by cadets from the Situated group (M = 5.54, SE = 0.17, 95% IP [5.20, 5.88]). From the results shown in

Table 1. Results on the initial knowledge test in the initial measurement for three groups of cadets.

Group	Measurement	M	SE	95% IP
Situated	Initially	5.54	0.17	[5.20, 5.88]
Situated + Games	Initially	5.12	0.17	[4.78, 5.46]
Situated + Games + Research	Initially	5.43	0.17	[5.10, 5.77]

, it is evident that there was little difference in the average results obtained by cadets from different groups on the initial test of knowledge.

For the examination conducted while undertaking the initial test of knowledge (measured in seconds) among cadets from three groups/conditions (Situated; Situated + Games; Situated + Games + Research), the results indicate a statistically significant difference in the time spent on the initial test of knowledge (Kruskal–Wallis H = 13.85, df = 2, p < 0.01). Furthermore, the results showed that cadets from the Situated + Games group spent significantly less time on the initial test of knowledge compared to cadets from the Situated + Games + Research group, as well as compared to cadets from the Situated group. Between the remaining groups (Situated and Situated + Games + Research), there was no significant difference in the time spent on the initial test of knowledge, as is evident from the results shown in

Table 2. Time spent solving the initial knowledge test with respect to the group.

Group	N	Average Rank 1	Kruskal–Wallis H	df	p
Situated	90	154.23	13.85	2	0.001
Situated + Games	90	111.79
Situated + Games + Research	90	140.48

¹ The time is in seconds.

.

3.2. Final Test

The results of the final knowledge test were obtained by summing up the achieved scores, where each correct answer to individual questions scored one point and incorrect answers scored zero points. Cadets from the Situated group, on average, achieved the lowest number of points on the final knowledge test (M = 7.72, SE = 0.15, 95% IP [7.42, 8.02]). Following this, average scores were achieved by cadets from the Situated + Games group (M = 8.72, SE = 0.15, 95% IP [8.42, 9.02]). The best result (the highest average points) on the final knowledge test was achieved by cadets from the Situated + Games + Research group (M = 10.27, SE = 0.15, 95% IP [9.97, 10.56]). The results obtained by cadets from different groups on the final knowledge test differ significantly. Furthermore, from the results shown in

Table 3. Results on the final knowledge test in the final measurement for three groups of cadets.

Group	Measurement	M	SE	95% IP
Situated	Final	7.72	0.15	[7.42, 8.02]
Situated + Games	Final	8.72	0.15	[8.42, 9.02]
Situated + Games + Research	Final	10.27	0.15	[9.97, 10.56]

, it is evident that each group of cadets achieved a significantly higher average number of points on the final knowledge test compared to the average number of points achieved on the initial knowledge test.

For the examination conducted while solving the final knowledge test (measured in seconds) among cadets from three groups/conditions (Situated; Situated + Games; Situated + Games + Research), the results indicate a statistically significant difference in the time spent on the final knowledge test (Kruskal–Wallis H = 168.78, df = 2, p < 0.001). Furthermore, the results showed that all three groups significantly differed in the time spent on the final knowledge test. Specifically, cadets from the Situated + Games group spent the least time solving the final test, followed by cadets from the Situated group, while cadets from the Situated + Games + Research group spent the most time solving the final test, as evidenced by the data presented in

Table 4. Time spent solving the final knowledge test with respect to the group.

Group	N	Average Rank 1	Kruskal–Wallis H	df	p
Situated	90	125.35	168.78	2	0.000
Situated + Games	90	65.48
Situated + Games + Research	90	215.67

¹ The time is in seconds.

.

3.3. Integrative Results of Conducted Knowledge Test

The cadets’ knowledge was assessed by a knowledge test consisting of 11 questions. The results of the knowledge test were obtained by summing the achieved scores, where each correct answer to a particular question scored one point and each incorrect answer scored zero points. Therefore, the theoretical range of scores is from a minimum of zero to a maximum of eleven points.

Table 5. Descriptive statistics for the points achieved on the knowledge test in two measurements.

	N	Min	Max	M	SD	Sk	Ku
Points in initial measurement	270	0.00	11.00	5.37	1.63	−0.01	−0.05
Points in the final measurement	270	4.00	11.00	8.90	1.77	−0.59	−0.49

shows the descriptive statistics for the results of the knowledge test in the initial and final measurements. In the total sample of cadets (N = 270), the average score (M) on the knowledge test in the initial measurement was 5.37 points (SD = 1.63), and in the final measurement, it was 8.90 points (SD = 1.77).

A statistically significant interaction between measurement and group on the achieved number of points on the knowledge test was obtained (F(2, 267) = 35.21, p < 0.001, ηp² = 0.21). From the results (with 95% reliability intervals) shown in

Table 6. Results on the knowledge test in the initial and final measurement for three groups of cadets.

Group	Measurement	M	SE	95% IP
Situated	Initially	5.54	0.17	[5.20, 5.88]
	Final	7.72	0.15	[7.42, 8.02]
Situated + Games	Initially	5.12	0.17	[4.78, 5.46]
	Final	8.72	0.15	[8.42, 9.02]
Situated + Games + Research	Initially	5.43	0.17	[5.10, 5.77]
	Final	10.27	0.15	[9.97, 10.56]

, it is evident that there was little difference in the average scores obtained by cadets from different groups on the initial knowledge test. However, the results obtained by cadets from different groups on the final knowledge test significantly differ. To this end, cadets from the Situated group on average achieved the lowest number of points on the final knowledge test (M = 7.72, SE = 0.15, 95% CI [7.42, 8.02]).

According to the average number of points, cadets from the Situated + Games group follow (M = 8.72, SE = 0.15, 95% IP [8.42, 9.02]). The best result (the highest average number of points) on the final knowledge test was achieved by cadets from the Situated + Games + Research group (M = 10.27, SE = 0.15, 95% IP [9.97, 10.56]). Additionally, it is evident from the results that each group of cadets achieved a significantly higher average number of points on the final knowledge test compared to the average number of points achieved on the initial knowledge test.

Related to the three forms of AR active learning (Situated; Situated + Games; Situated + Games + Research), the difference in cadet scores on the knowledge test between two measurements (initial test and final test) was examined. In Figure 1, the blue broken lines indicate that cadets in all three forms of AR active learning achieved significantly lower average scores on the initial test (M = 5.37) than on the final test (M = 8.90). It is evident that there were no differences in the average scores obtained by cadets in different groups on the initial test. Cadets in Group 2: Situated + Games had the lowest average scores on the initial test (M = 5.12), followed by cadets in Group 3: Situated + Games + Research (M = 5.43). The best result (the highest average scores) on the initial test was achieved by cadets in Group 1: Situated (M = 5.54). After controlling the cadets’ perception of the application of augmented reality technology in teaching, a statistically significant interaction of measurement and group was obtained based on the cadets’ knowledge (F(2, 264) = 36.13, p < 0.001, ηp²= 0.21). Cadets from different groups did not differ in terms of mean scores on the initial knowledge test, as shown in Figure 1 by the blue dashed lines, but they did differ in mean scores on the final knowledge test. After controlling for perception, a significant effect of the group on the score on the knowledge test was obtained (F(2, 264) = 31.15, p < 0.001, ηp² = 0.19), whereby the analysis of multiple comparisons with the Bonferroni correction indicates a significantly higher average score of the cadets from Group 3: Situated + Games + Research (M = 7.88) compared to the score of the cadets from Group 2: Situated + Games (M = 6.92) and the score of the cadets from Group 1: Situated (M = 6.65). This is shown in Figure 1 by the purple lines.

However, the results obtained by cadets from different groups on the final knowledge test significantly differ, as can be seen in Diagram 1 where the purple lines represent the difference in the achieved results. Therefore, cadets from Group 1: Situated on average achieved the lowest number of points on the final knowledge test (M = 7.72). In terms of an average number of points, cadets from Group 2: Situated + Games (M = 8.72) follow. The best result (the highest average number of points) on the final knowledge test was achieved by cadets from Group 3: Situated + Games + Research (M = 10.27) [67].

4. Discussion

Cadets who were taught through all forms of (AR) active learning (Situated + Games + Research) achieved a statistically significantly higher level of knowledge compared to cadets of the other two groups. The obtained results of the conducted research indicate that cadets who participated in the form of AR active learning (Situated + Games + Research) using the HaubicAR application showed a higher level of knowledge, as follows:

• There was a correlation between the initial knowledge test and final knowledge test scores in the cadet sample (N = 270), where the average score (M) on the knowledge test in the initial measurement (initial test) was 5.37 points (SD = 1.63), while in the final measurement (final test), it was 8.90 points (SD = 1.77). The results indicate a significant effect of measurement (F(1, 267) = 748.30, p < 0.001, ηp² = 0.74) on the knowledge test score, with cadets achieving significantly lower average scores in the initial measurement (M = 5.37, SE = 0.10) than in the final measurement (M = 8.90, SE = 0.09).
• The correlation between the different groups on the initial knowledge test, a statistically significant interaction of the measurement and the group on the achieved number of points on the knowledge test was obtained (F(2, 267) = 35.21, p <0.001, ηp² = 0.21) where it is evident that no differences were obtained in the average results achieved by cadets from different groups on the initial knowledge test. There is no statistically significant difference between the groups.
• The correlation on the accuracy of responses between different groups on the final knowledge test differs significantly. The Situated group achieved the lowest average number of points on the final knowledge test (M = 7.72, SE = 0.15, 95% IP [7.42, 8.02]). Following in terms of average points are cadets from the Situated + Games group (M = 8.72, SE = 0.15, 95% IP [8.42, 9.02]). The best result (the highest average number of points) on the final knowledge test was achieved by cadets from the Situated + Games + Research group (M = 10.27, SE = 0.15, 95% IP [9.97, 10.56]).
• The correlation between different groups does not differ in the perception of the application of AR technology in teaching (F2, 265 = 0.61, p = 0.55), thus satisfying the prerequisite of independence of covariate and independent variable. Covariance analysis results indicate a significant effect of measurement (F(1, 264) = 5.20, p < 0.05, ηp² = 0.02) on the knowledge test score, where cadets achieved a significantly lower average number of points in the initial measurement (M = 5.37, SE = 0.10) than in the final measurement (M = 8.93, SE = 0.09). A statistically significant interaction between measurement and group on cadets’ knowledge was obtained (F(2, 264) = 36.13, p < 0.001, ηp² = 0.21).
• The correlation between different groups does not differ in the average cadet perception score on the initial knowledge test but differs in the average score on the final knowledge test. Cadets from the Situated group achieved the lowest average score on the final knowledge test (M = 7.75, SE = 0.15, 95% IP [7.46, 8.04]). A better result on the final knowledge test was achieved by cadets in the Situated + Games group (M = 8.71, SE = 0.15, 95% IP [8.42, 9.00]), while the highest score was achieved by cadets in the Situated + Games + Research group (M = 10.33, SE = 0.15, 95% IP [10.04, 10.62]). The correlation between different groups is positive and statistically significant.
• In terms of the correlation between different groups on the accuracy of answers after controlling for perception, a significant group effect on the knowledge test score was obtained (F(2, 264) = 31.15, p < 0.001, ηp² = 0.19), with multiple comparison analysis indicating a significantly higher average score for cadets in the Situated + Games + Research group (M = 7.88, SE = 0.12). The correlation between different groups is positive and statistically significant compared to the results of cadets from the Situated + Games group (M = 6.92, SE = 0.12) and the Situated group (M = 6.65, SE = 0.12).
• The correlation between different groups in the time spent on the initial knowledge test was analyzed using the Kruskal–Wallis test (H = 13.85, df = 2, p < 0.01). The post hoc analysis revealed that cadets from the Situated + Games group spent significantly less time solving the initial knowledge test compared to cadets from the Situated + Games + Research group, as well as compared to cadets from the Situated group. There was no significant difference in the time spent on the initial knowledge test between the remaining groups (Situated and Situated + Games + Research). These results indicate a statistically significant difference in the time spent on the initial knowledge test.
• The correlation between different groups in the time spent on the final knowledge test was examined using the Kruskal–Wallis test (H = 168.78, df = 2, p < 0.001). The post hoc analysis revealed that all three groups significantly differed in the time spent on the final knowledge test. Specifically, cadets from the Situated + Games group spent the smallest amount of time, followed by cadets from the Situated group, while cadets from the Situated + Games + Research group spent the most time on the final knowledge test. These results indicate a statistically significant difference in the time spent on the final knowledge test.
• In the correlation analysis between the three groups regarding the average time spent within the application, with regard to the group (F2, 267 = 274.23, p < 0.001, ηp² = 0.67), a significant effect was found. The results indicate that cadets from the Situated + Games + Research group (M = 1739.58, SD = 205.74) spent significantly more time within the application on average compared to cadets from the Situated + Games group (M = 1223.87, SD = 102.11) and cadets from the Situated group (M = 1157.23, SD = 27.40). There was no significant difference in the time spent within the application between the remaining groups (Situated and Situated + Games). These results indicate a statistically significant difference in the average time spent within the application.

The obtained results of the conducted research indicate the success on the knowledge test, wherein cadets achieved significantly lower average scores in the initial measurement compared to the final measurement. No differences were found in the average results obtained by cadets in different groups in the initial knowledge test. The best result (the highest average scores) on the final knowledge test was achieved by cadets in Group 3: Situated + Games + Research. This confirms hypothesis H1: Cadets instructed through all forms of AR active learning; Situated, Games, Research achieve a statistically significantly higher level of knowledge compared to cadets from the other two groups.

The relevance of the results and setting of the applied research model are related to the context of the existing literature. The authors Pantelić and Plantak [11] in their research emphasized the use of augmented reality technology in teaching in higher education as it is compatible with certain pedagogical approaches that fit into the framework of active learning: constructivist learning, situated learning, game-based learning, and learning based on research. Furthermore, they found that respondents had better results when using active research-based learning, which is in line with the relevant results of this research. Authors Akçayir and Akçayir [15] point out the potential of using augmented reality in teaching to teach the main topics through effective active learning design: situated learning, game-based learning, and research learning. Tuta et al. [48] in their research and results demonstrated that AR technology greatly contributes to active learning because it stimulates inquiry-based learning. In inquiry-based learning, students explore and experiment with virtual models. Therefore, inquiry-based learning provides a way to electronically collect data for future analysis and provide virtual models placed in a real-world context that are easily manipulated.

5. Conclusions

With the emergence of new technologies and the transition to virtual teaching environments, the processes of learning, teaching methods, and the overall view of the future of education at all levels and in all segments have fundamentally changed. The research described in this paper represents a form of contribution to determining and evaluating the learning outcomes of students through the application of active learning using augmented reality technology in education. The research under consideration, through a literature review, analysis of recent sources, conceptual framework, and experimental approach to assessing the impact of active learning using augmented reality on knowledge, represents a step forward in the acceptance of educational solutions. Furthermore, our scientific contribution at the methodological level was achieved through the original design of the research implementation, where three scientific methods were synergistically connected. For these reasons, it is appropriate to consider continuing the research concerned in a manner that applies the described methodology to conduct research on other related and unrelated study programs. In conclusion, the obtained research results provide a basis for the further development of technological possibilities for future applicative solutions based on new AR technologies, thus demonstrating their application potential.