A Learning Analytics Conceptual Framework for Augmented Reality-Supported Educational Case Studies

Abstract

The deployment of augmented reality (AR) has attracted educators’ interest and introduced new opportunities in education. Additionally, the advancement of artificial intelligence has enabled educational researchers to apply innovative methods and techniques for the monitoring and evaluation of the teaching and learning process. The so-called learning analytics (LA) discipline emerged with the promise to revolutionize traditional instructional practices by introducing systematic and multidimensional ways to improve the effectiveness of the instructional process. However, the implementation of LA methods is usually associated with web-based platforms, which offer direct access to learners’ data with minimal effort or adjustments. On the other hand, the complex nature of immersive technologies and the diverse instructional approaches which are utilized in different scientific domains have limited the opportunities for research and development in this direction. Within these research contexts, we present a conceptual framework that describes the elements of an LA process tailored to the information that can be gathered from the use of educational applications, and further provide an indicative case study for AR-supported educational interventions. The current work contributes by elucidating and concretizing the design elements of AR-supported applications and provides researchers and designers with guidelines on how to apply instructional strategies in (augmented) real-world projects.

1. Introduction

Contemporary educational technology requires researchers and instructional designers to propose purposeful and effective solutions to eliminate significant points of failure that are associated with reliable computing systems. Well-designed simulated instructional environments, with high representational fidelity and adaptability, can assist in the application of the theoretical knowledge via experimental (“hands-on”) practices and tasks [1]. This is also aligned with the new statistical data, which show the rapid expansion of different educational fields, especially in developed countries [2].

However, instructional designers are still facing a wide range of constraints and limitations when applying laboratory exercises, experiments, and practice-based learning tasks, regardless of the educational level or subject [1]. The barriers that create such drawbacks include the transportation of students/instructors to different locations (out-of-class exercises or extra-curricular activities), the time required to set up and prepare the laboratory tasks/experiments, and, of course, the respective costs. Another factor affecting the educational experience negatively—which has been linked to students’ frustration and dissatisfaction—concerns the lack of support/guidance that learners receive from their instructors either due to time constraints or the limited availability of human resources (teachers/assistants) [3,4].

From another point of view, the aforementioned aspects may affect those students who originate from low-income countries. In such contexts, the lack of financial resources and/or technological infrastructure to design and set up training sessions makes it almost prohibitive for both practitioners and students to practice “hands-on” activities. Another aspect of major importance regards the diverse educational outcomes that are usually reported in experimental studies and the never-ending efforts that scholars make to better understand the impact of the chosen instructional practices on learners (individually or collectively). Hence, the importance of analyzing the potential and the shortcomings of teaching and learning practices is becoming even more crucial and imperative [5].

The recent advancements in the field of computing have enabled educators and practitioners to increase the effectiveness of instructional design and allowed them to collect and analyze large-scale digital datasets. One proposed solution to investigate and explore such topics involves the collection and interpretation of so-called ‘big data’, which are processed with the aid of machine learning (ML) models and educational data mining (EDM) techniques [6,7,8]. The so-called learning analytics (LA) discipline is an emerging field, with rapid growth over the last five years, which aims at assisting researchers and instructors to better understand learners’ needs by collecting and interpreting large-scale data in a systematic and independent way. The core philosophy of LA is focused on students’ positions at the center of attention, wherein their digital traces of interaction are recorded, analyzed, and explored in order to reveal insights related to their cognitive patterns and behavioral decisions.

Early attempts to introduce LA into practice have describe the collection of data which have been utilized to study in detail the complex nature of interactions that learners have when undertaking technologically supported learning activities [9]. However, as interest in this direction of research has grown additional drivers and facilitators have joined the field. For instance, [10] introduced tools that could support instructional designers to develop learning scenarios and orchestrate teaching strategies for multilayered activities. Others [11,12] focused on the improvement of the services that massive online open courses (MOOCs) offer. Some researchers [13] have introduced review tools and analysis protocols for students’ and teachers’ interactions to support the process of reflection on the applied teaching strategies. More recent studies [14] explored multimodal data derived from sensors and wearable trackers, to provide adaptive and enhanced feedback for learners. Nevertheless, despite the growing interest in evidence-informed instruction, the efforts to introduce LA practices in augmented reality (AR)-supported instruction have been limited and scarce.

1.1. Augmented Reality in Education

In contrast to virtual reality (VR), which “transfers” the real-world learning experience to a computer-generated simulated environment, AR “augments” the learning process by enriching the spatial environment with digital information that is rendered over physical objects in real-time [3,15]. The projection of the virtual content in real-world settings is achieved via mobile devices (e.g., smartphones, tablets), (semi)transparent surfaces (e.g., tabletops) or accessories (e.g., AR glasses) [16]. Under this consideration, [17] places AR solutions in the spectrum of immersive technologies, as they merge the physical with the virtual environment. Thus, AR can enable learners to augment their learning experience based on the ideas and concepts that are under investigation and visualized over physical objects [3].

Even though immersive technologies share many features in common, studies that have examined comparatively the use of VR and AR in education persistently documented the superiority of the latter, especially with regard to the ease of use (independently of learners’ age) and the considerably lower operational cost, as there is no need for expensive or specialized equipment. This is further evident after considering the call for research related to AR applications that [18] made, with particular focus on topics related to applied sciences [19,20,21,22,23]. To this end, a number of studies have already well-documented the potential of using AR in different “hands-on” learning practices, such as: (a) to support collaborative-, problem-solving-, or inquiry-based contexts, which aim to help educators to contextualize the learning process and students to link the concepts under investigation with their prior knowledge and experiences [24]; (b) to help students better understand what kind of information is provided within a specific context, how this information can be translated into knowledge acquisition, and how the constructed knowledge can be applied more effectively into practice [25]; and (c) to improve students’ performance (e.g., knowledge development, behavior) and outcomes (e.g., satisfaction, achievements, acquired knowledge/skills, competencies) while also identifying the fluctuations that learners have during the different stages of the interventions [26].

In accordance with the abovementioned studies and conclusions drawn by other researchers [3,16], the greatest potential of AR is generally attributed to the visually appealing graphics and the vivid multimedia features, which offer diverse opportunities for interplay and engagement, as well as on the immediate (real-time) feedback that users receive in response to their actions. Additionally, an ongoing body of literature [7,27,28] has also provided possible solutions to enhance the potential of AR via LA. Such a combination can assist researchers and educators to recognize new ways of understanding the key concepts of action, as well as students’ competencies and outcomes.

1.2. Integration of Learning Analytics in Education

The rapid deployment of educational technology in modern classrooms and the deliberate efforts of educators to improve their formal and informal educational practices led to the emergence of a new field which relates to the collection of large datasets for the evaluation of the learning process and the assessment of the competencies of the respective programs [29].

The term “big data” can be classified based on the following broad dimensions: (a) volume—the quantity of the data; (b) speed—the rate at which the data are imported/exported; and (c) variety—the sources from which the data originate. However, when it comes to LA, a more context-specific and student-centered perspective is required. For instance, [30] distinguished the applications of LA from the perspective of educational level/field and proposed a three-dimensional taxonomy:

• Micro-level: data gathered by recording a specific module or learning activity in-class.
• Intermediate level: data gathered by recording a complete training program or unit.
• Macro-level: data gathered by recording a set of educational programs or modules.

Other studies [31,32] carried out the classification of LA in accordance with the questions they seek to answer:

• Descriptive analytics, centering on what has already happened and answering the question of discovering patterns based on the aggregation of students’ data.
• Predictive analytics, focusing on what is going to happen and attempting to predict evolutionary trends in student’s future progress.
• Regulatory analytics, aiming at what needs are important and what factors are affecting student learning performance and proposing recommendations for future activities.
• Management analytics, converging on the financial cost of the operational/technical equipment and attempting to predict the future use of the present resources and decisions to ensure the quality of educational units and/or modules.

Those who have examined the added value of LA from the instructor’s point of view have focused on the reinforcement of educational objectives based on the patterns that students develop after undertaking repeated specific exercises [33,34,35].

According to Chatti et al. [36], LA is an iterative three-step cycle consisting of data collection and pre-processing, analytics and action, and post-processing. Several approaches have been proposed for LA applications, and these have been reviewed by Sciarrone and Temperini [37]. Campbell et al. [38] proposed the five-step Oblinger model—which is composed of five sequential steps (capturing, reporting, predicting, acting and refining)—in which researchers had to answer specific questions. According to this model the data were captured at the first step, and reports were created, along with predictions that helped researchers take the appropriate actions in the learning environments to improve the learning process. Clow [39] added the concept of feedback and described LA as a circular process. He proposed the four-step LA Cycle Model, represented by a four-step cycle based on learners, data, metrics, and intervention. The metrics step provided researchers with visualized information on the basis of which they could proceed to the interventions. During the intervention step, based on the metric outcomes, the researchers could explore the results and improve the educational process. Greller and Drachsler [40] proposed a generic framework for LA composed of six dimensions (stakeholders, objectives, data, instruments, external constraints, and internal limitations). Similarly, Chatti et al. [41] proposed a four-dimensional LA Model. According to this model, LA data could be distinguished into four dimensions, answering questions about “what” (data and environments), “who” (stakeholders), “why” (objectives) and how (methods). Another LA Model was proposed by Siemens [42], which included seven components: collection, storage, data cleaning, integration, analysis, representation and visualization, and action. Finally, Sergis and Sampson [43] proposed another model which focused on teaching analytics, consisting of six looping steps (problem identification, developing inquiry questions, educational design, delivering educational design and collecting data, analyzing educational data, reflecting on data). LA is still developing as a field. New and promising technologies that are now used in education, such as AR/VR, and the increase in recorded data, both in terms of quantity and quality, require special attention on the application of LA in different educational settings. This paper proposes a novel LA framework that can be applied, among others, in immersive technology systems, and provides guidelines on how this can be implemented for AR-supported educational interventions.

2. Rationale and Purpose

The pervasive technological evolution has brought multiple benefits to the societal foreground, as well as introducing the need to reconsider many practices, including educational ones (e.g., the development of digital literacy and competence skills) [44,45,46]. For instance, constructivism—one of the most fundamental and authentic didactic approaches—is nowadays being aided by advanced and visually rich computing solutions (e.g., VR), which support self-directed and collaborative instructional methods. However, especially within the last decade, an increase in the number of studies which explore the use of AR in both K–12 (primary/secondary) and higher education (HE) settings can be observed [47,48]. By utilizing such technological solutions, learners of all ages and educational contexts are not only developing their digital competencies and skills, but also their cognitive and practical reasoning [49,50].

However, the experimental nature of immersive technologies, the specialized equipment usually required for such interventions and the situationally unique nature of the training sessions/programs make it difficult to use innovative technologies in the context of everyday teaching and learning practices, and even less in large scale projects. The reasons on how such efforts can be integrated within LA practices in the context of immersive technologies are rather scarce are not surprising.

Previous research has addressed different issues and trends related to the use of AR and the application of LA in different learning subjects and contexts. However, the development of emerging learning scenarios requires new design frameworks that are capable of improving the potential of AR technology in both formal and informal educational scenarios. To this end, this study seeks to outline an innovative way to support trainees (both in K–12 and HE) to develop their knowledge and expand their skills in a personalized and mutually interactive way, while undertaking training experiences aided by immersive technologies.

Motivated by the research challenges that surround the application of LA in immersive education, we propose a conceptual design framework which combines key elements and factors from both ends, and further theorize how such advanced techniques can be utilized to capture the elements that influence the learning process when AR technology is used. Therein, the contribution of this work is twofold: (a) it supports the multimodal nature of LA and its potential to aid the design and development of interactive learning experiences, and (b) it provides guidance and insights on how such ideas can be integrated into the real-world context of AR-based interventions.

3. Framework Overview

The proposed framework consists of four main components, namely: design, development, analysis and assessment. All components act as a part of a continues cycle process that can be adjusted over the time according to the present needs. The overall architecture of the proposed framework is presented in Figure 1. The proposed framework will advise and inform educators, instructors, and data analysts about the topics related to the design, implementation, and evaluation of LA in teaching and learning, and will provide guidelines on the application of this framework in AR-supported educational innervations.

3.1. Design

The design of each procedural process is crucial, as it currently affects the outcomes and the impact of the results. A well-designed process will provide the involved stakeholders with safe and clear scientific results that can be easily used by the wider scientific community. On the other hand, an LA process that is not well-designed can lead to misleading or controversial outcomes and may result in lost working hours.

We believe that special emphasis should be placed on the correct design of the LA process. Therefore, based on related research on LA and instructional design methods, the proposed framework provides detailed instructions by dividing the design component into seven distinct stages (

Table 1. Design stages.

No	Stage	Aim	Type
1	Identification of the key stakeholders	Identify all the possible key stake holders	Conceptual
2	Identification of needs	Provide guidelines on the needs, identification according to the previous stages	Conceptual
3	Mapping the available data	Study the available data and the possible ways that they can be analyzed	Technical
4	Definition of metrics and indicators	Choose or create the metrics and indices that will be used for the LA process	Technical
5	Data collection approach	Adopt an adequate collection approach	Technical
6	Data analysis methods	Decide on the methods that will be used	Technical
7	Visualization techniques	Decide on the visualization techniques	Technical

) of two types (i.e., technical and conceptual) that researchers should follow in order to design an effective LA application for education.

3.1.1. Identification of the Key Stakeholders

First, researchers need to identify the main stakeholders (e.g., learners, educators, instructional designers, policy makers) and summarize the key areas that can be influenced by the integration of LA. The aim of this stage is to explore the unique characteristics of the stakeholders who will be involved in the teaching and learning process. It should include an analysis of the characteristics of the key stakeholders, their role in the instructional process, the actions that execute, and their relationship to the educational content, the educational context, and the learners. They should also consider how educational AR applications can affect the above information. For example, what actions do the educators need to execute to apply AR technology in their content? How can they use AR in education? Researchers can explore possible information that could help stakeholders to improve the educational process from their perspective and position. The outcome of this stage will be used as input for the next stage.

3.1.2. Identification of Needs

The second stage of the design component is the most important. This stage includes the identification and analysis of the research questions that need to be addressed, as well as the determination of the problems and challenges that may occur. It can be also divided into two phases that should be covered during the LA process.

Identification of Concepts for Analysis

To identify the concepts for analysis, researchers should decide on the analysis type that they will implement, based on the questions that need to be answered. Consequently, the first step is to conclude whether the LA will be descriptive, predictive, regulatory or managerial. Each of the predefined categories will lead to different analysis methods and results, as introduced earlier, and may require different sets of data as an input. Researchers also need to conclude if models and theories (e.g., the technology acceptance model (ΤAΜ); analysis, design, development, implementation, and evaluation (ADDIE)) will be considered during the analysis stage. The adopted models and theories may indicate whether additional concepts need to be analyzed, as well as clarifying the types of data that are necessary for the analysis of each concept. Additionally, they need to consider the use of AR and which information would be beneficial in relation to the technological application.

This analysis and assessment can be applied through three axes according to the data classification and the stakeholders, as described below.

Educational Content

The online educational content itself produces specific data that can be further analyzed and assessed. Considering the use of AR, the educational content is more likely to consist of an online learning platform and the use of an AR mobile application that will enable students to view the provided augmentations. The educational content will contain a lot of activities instead of static resources and researchers should decide which characteristics of the content can be used for the analysis. For example, the number or the type of augmentations could be one example of this type of data. LA based on these data is useful for instructional designers, scholars, and educators. These data itself are independent of the students’ actions and can be gathered from the course’s e-learning platform or the AR application. The educational content may affect students’ behavior and performance. Therefore, it is very useful to know the quality and quantity of the educational content since both define students’ actions through the learning process. Frequently updated and high-quality content could maintain students’ interest in the course and motivate each one to concentrate and study. Similarly, a course with an adequate quantity of content will cover all the necessary concepts that students need to learn according to the course learning objectives. Any lack of quantity or quality could reduce user learning performance and experience. For example, content with less activities and more static resources is more likely to be lacking in terms of quality, since “hands-on” practices involve the active participation of students. Therefore, any feedback to the content authors could provide an estimation of the course characteristics and help in identifying further improvements.

Quantity can be defined by metrics related to the e-learning platform, such as the provided file numbers and sizes, the number of links to other resources and the number of interactive exercises. For AR technology, quantity can be defined by the number of applied augmentations, the total length of possible video augmentations, and so on.

Quality can be captured based on the type of educational content and how “rich” it could be. We can assume that video content has increased quality compared to a plain text document. For AR technology, quality could be defined in terms of AR tracking, AR visualization and AR interaction. AR tracking is about the triggering process that allows the system to provide the augmented content. This can be achieved via a marker (e.g., a QR Code) or in markerless fashion (e.g., through geo-location triggering, images, or real 3D objects). A markerless AR application, in most cases, provides a better learning experience. AR visualization relates to the media type of the provided augmentation (e.g., text, link, video, image, sound, 3D object) and can also be an indicator of AR content quality. An augmentation that provides a video may result in a better learning experience compared to an image augmentation. AR interaction is about how users can interact with the augmentations. Different types of interactions can be applied for each type of content. For example, augmentations may be static, without interactions, or they may allow users to take simple actions such as determining the size of a photo, managing the flow of a video, moving a 3D model across its three axes or to perform more complicated interactions such as choosing specific parts of a 3D model and exploring them virtually, answering multiple questions and simulating lab processes. The latter type of interactions could increase the quality of content. In the educational context, this could be translated into measures of the percentage of interactive content compared to the total content of a course, the objectives of each exercise and whether or not critical thinking is required.

Learner Profiles and Behavior/Activity

These data can indicate learners’ state before, during and after the learning process. Consequently, it can contain three types of data. The first type is students’ academic characteristics, which are constant during the learning process, such as age, gender, learning style, previous knowledge, and experience with used technologies (such as AR-aided educational content). These data allow educators to know the initial knowledge, preferences, needs and even to forecast the performance [51] of their students. Accordingly, they can help them decide how to proceed with the educational process based on students’ individual needs. Additionally, online platforms may adapt their content according to these initial data [52] and may provide personalized educational content.

The second type of data is related to the students’ activity and course (learning management systems (LMS) and AR application usage). The data produced during the learning process can include students’ number of visits, the duration of each online course, preferences in educational content type, utilization of AR content and exercises. This type of data is very important since, on the one side, it can enable educators to discover possible problems that students face during their study and allow educators to proceed to the appropriate actions to improve the learning process and, on the other hand, educational platforms need to provide students with personalized indications, content and messages to motivate them to proceed with their study [53]. In addition, other studies have shown that students’ usage data can range from course ranking [54] and classification [55] to automated suggestions for improvements of the educational content [56]. For example, if users prefer video instead of image augmentations, this may help the course authors to design their courses according to this specific learning preference. Likewise, if students spent a considerable amount of time in a course with specific content and very limited time in another similar online course, this may indicate that the second course needs improvement in terms of quality or quantity.

The third type of data is related to students’ learning performance, and this can be retrieved through an online questionnaire, a quiz, an interactive assessment activity or through the completion of the course. These data can reveal the knowledge outcomes of the students in an automated way. Additionally, researchers may discover possible interrelations of students’ actions and learning outcomes or proceed to a two-fold evaluation [57] for better results. These results could potentially enable authors to redesign their courses and change the implemented learning strategies in the future.

Regarding the AR application, the abovementioned types of data can reveal the degree of context immersion in AR environments, such as involvement, engagement, motivation, mobility, reality or parameters associated with psychology, including self-efficacy, self-esteem, situational interest, or general information about overall user experience, satisfaction and course interactivity [3,15,28,58].

Technology Utilization Context in the Educational Process

The third assessment axis is related to the context in which online learning and technologies are applied. These data may not be recorded in electronic media; however, they must be considered in conjunction with the other evaluated data. These data can include information that can be retrieved by the course curriculum like the type of the course (formal/informal), whether completely distant or blended learning is applied, the place (location), the provided technologies used in this context (classroom, laboratory, field, home), and whether course attendance is compulsory or optional. All these factors may affect the behavior and actions of learners concerning the provided educational material and should always be taken into consideration. For example, students in a blended learning course may have reduced activity and may not make use of the AR content or the provided interactive exercises due to their interaction with the teacher. On the other hand, in a distance learning online course, the use of AR and/or other interactive content is necessary to fulfil the course objectives. If researchers do not consider these elements, incorrect conclusions may be drawn.

Identification of Data Shortages

The previous stage may reveal some possible data shortages. At this stage, researchers define which data shortages should be covered during the LA process. These shortages can be classified into three general categories—problems, challenges, and limitations.

We define these shortages as ‘problems’ when it is easy to address them. A common example of this category is the lack of recording of personalized data in an online learning course when the visit and duration is recorded but not the student ID. This is a problem that, in most cases, can be easily addressed through minor technical actions.

Challenges may require more complicated actions and resources to be addressed. For example, if one wants to measure student motivation regarding the AR content, they may first need to propose a new index that is based on current measures and check if this can reveal the requested outcome. Another example is how one can record the utilization of the AR content by the students. Such a process requires some specific steps, such as: (a) conceptual analysis, where researchers propose a new index; (b) technical analysis, where programmers make the appropriate changes in the code to retrieve the appropriate data; (c) component analysis, where researchers need to analyze the provided data, and (d) assessment analysis, where researchers confirm the validity of the proposed index and the correctness of the analysis results.

In relation to this notion, there is a broad agreement that some research questions cannot be answered due to several limitations. This is the third category of shortages. A common example is the size of the research sample or the need for the use of expensive infrastructures that blend fields like educational neuroscience [59]. These constraints should be recorded and explicitly presented by the researchers.

3.1.3. Mapping the Available Data

The next stage is mainly technical and concerns the mapping of the available data. The first step is to assess what kind of data is available. An e-learning platform can provide different types of data in different forms and formats (e.g., log files, dynamic reports), whereas other data, such as final exam grades, can also be available from other resources. AR applications can also provide specific data formats that allow researchers to study users’ behavior. Researchers should review the literature and categorize the available data. They need to identify and name the concept covered by each data type. They should also examine the available options to analyze the data that can be provided. For example, some LMSs provide visual data on demand (e.g., monthly reports) or enable the users to export the raw data in various formats for analysis using third-party programs. Therefore, an output at this stage could be a report that contains a list of the various data, the concepts that each one can cover, their type, and the possible methods of analysis.

Researchers need to either use current indices/metrics or propose some new indices that can better address the predefined needs of LA.

Mapping the available data stage answers the question of what data should be used to meet the needs identified in the second stage of design. Metrics refer to primary data, such as the number of users or the usage time, demonstrating in a clear way what they represent. The indices are derived from the processed metrics by means of appropriate equations and formulas (e.g., average daily study time per user, average use of augmentations per content size). To answer any of the abovementioned questions, researchers may need to use existing indices and metrics or propose new ones. Therefore, three actions could be carried out:

• Studying the literature for available indices and metrics and their relationship to LA concepts: Many researchers have suggested different indices and metrics to study different concepts related to e-learning. Often, researchers can adapt them to the specific LA process. In this case, they can suggest changes in the e-learning platform or the online course to obtain the data needed for the subsequent analysis. For example, if a system only records the login/logout time of its users, researchers may ask to record the time users spend on certain online resources of the course.
• Using appropriate existing indices and metrics: If researchers have found that existing indices and metrics can be used in the LA process, they may need to adapt them to the specific needs and characteristics of the learners. For example [60] uses indices and metrics proposed by Laudon and Traver [61] from e-commerce and business analytics for LA processes.
• Proposal of new indices: Researchers can propose new indices that allow them to study specific concepts. They must study the literature carefully and decide whether they can link the newly proposed indices with specific concepts. In most cases, this process requires validation to prove the correlation between the indices and the research concepts.

3.1.4. Data Collection Approach

At this stage, researchers should decide on the way that they will collect the appropriate data. According to [62], the data can be divided into four types, namely, (a) social, (b) educational, (c) learning, and (d) behavioral. Each type reveals several parameters and requires different methods for data collection such as survey, registry data, LMS logs, and motion tracking data.

The data collection approach should be designed carefully to ensure the accuracy of the results and allow authors to use all the available data and tools. Special care should be taken in cases where this data is recorded by users’ mobile smartphones, as this is very likely when AR technology is applied. Data should be retrieved in respect to user anonymity rights and national/international regulations, such as the General Data Protection Regulation (GDPR).

3.1.5. Data Analysis Methods

Researchers need to decide on the applied data analysis methods and the necessary programs and tools that need to be used. Some questions that need to be addressed at this stage are the following: Which methods could be potentially used for each type of analysis? Are there any limitations or drawbacks that should have been considered previously? Could authors cross-check their results using different types of analysis methods? Do the chosen methods require third-party programs or tools to be applied? What actions could be undertaken to increase the accuracy of the results? Are the adopted methods adequate for analyzing data related to AR content for education? For example, according to [62], data mining techniques can be used such as clustering, classification, association rules and regression.

3.1.6. Visualization Techniques

The produced outcomes of the data analysis may be complicated and difficult to interpreted by the final recipients. For example, some technical tables that are provided by statistical software, like SPSS, require knowledge of statistics to be understood. To address this problem, many learning platforms (such as Moodle), perform some initial analysis of their log data and provide educators with practical graphs that enable them to easily come to specific conclusions about their students. Consequently, researchers should decide on the visualization techniques that need to be applied to present the results of the learning analytics. A correct choice of visualization techniques will fully enable stakeholders to better understand the LA results and utilize the provided information. Special care should be taken especially when interpreting AR-generated results. It should be noted that it is not enough to provide generic outcomes about the course as a set of assets. Educators need to know how the AR content has been used and if it has positive effects on the students’ study. Therefore, visualization techniques should provide data for each kind of content and not only for the course.

3.2. Development

This is the second component of the proposed framework. It includes all the actions that researchers should perform to proceed with the analysis. Any changes that need to be applied to the learning platform or the server are applied as part of this component. In addition, programmers may need to develop new tools, applications and software programs that are necessary for the analysis. AR applications are very unlikely to record usage data. Therefore, developers should proceed with changes both in the AR application and the AR platform that manages the educational content and the augmentations. At the end of this component, researchers should have all the appropriate tools and programs available that are needed for the next component (analysis).

3.3. Analysis

During the analysis component, researchers should proceed carefully with the analysis of the data according to the design component outcomes and using the tools and programs that may have been developed in the previous component. Generally, they should follow the best practices based on the design of the adopted analysis.

The analysis of the data will produce useful information for the involved stakeholders. Researchers should provide this information to the stakeholders using adequate visualization techniques. However, they should also provide them with interpretations of the results and possible actions that the latter could carry out to further utilize the analysis results. For example, researchers (or platforms that perform automated analysis) may provide teachers with LA results and an additional report with suggestions for actions that need to be taken, such as: (a) Which actions should be done in relation to students? (b) How to further improve their online courses? [63]. (c) Is there a need to provide more AR content or interactive exercises?

3.4. Assessment

The last component of the proposed framework aims to further improve LA methods and results. Researchers should assess the quality of the produced results, the sample that has been used, and any specific characteristics, such as the use of AR or educational applications, or other issues related to the implemented analysis. Additionally, they should record any limitations or shortages and propose possible improvements to the LA process. Such an action will allow researchers, over the course of time, to improve the LA process and consequently provide better outcomes. Additionally, according to the LA results, they can improve either the applied pedagogy (such as learning models and theories), the provided educational content and the way technology (such as AR) is used in the educational intervention.

4. Framework Implementation for AR-supported Interventions

The proposed framework can be applied to any LA application. This section provides guidelines and presents an overview of the proposed framework’s application for educational AR-supported innervations. Specifically, Figure 2 depicts an educational AR intervention for the current case study consisting of three main dimensions: pedagogy, educational content and technology.

The instructor initially prepares the way he or she will proceed with the educational intervention and the applied pedagogy (instructional models or techniques and learning theories). In most of the cases the instructor creates or provides the students with specific educational content in either traditional (books, notes, etc.) or digital (pdf, PowerPoint, etc.) formats and implements them using an AR application that allows students to “augment” the initial educational content with appropriate additional information.

Students, on their side, have access to the educational content and can use their mobile devices (mobile phones, tablets, AR glasses, headsets) to access the augmentations and interact with the educational content. Every action of the students is stored in an external database. These data can be used as an input for the LA process. The instructor follows the proposed framework phases and designs the LA application, uses technology for the development component (if this is necessary) and proceeds with the analysis of the recorded data according to the design component outcomes. Finally, the instructor assesses the quality of the produced results and proposes improvements in both the LA process and applied pedagogy, educational content and/or AR application.

5. Discussion

As educational programs continue to gain ground in curriculum-aligned contexts globally, there is a persistent need for educators and instructors to offer their support. Student engagement across different educational fields is dependent on the appropriateness of the learning tasks that inform their academic activities and thus shape their future professional practices [64,65,66]. Likewise, given the professional character that such programs have, the development of a better understanding of the self-directed and collaborative efforts that students make is needed. In achieving this goal, researchers, educators, and instructional designers will be able to capture the multifaceted nature of such alternative learning experiences, as well as the patterns and the interrelationships that occur between the trainees and the learning context.

Despite the growing number of studies in educational technology and the evidence gained by combining different domains [7,12], the investigation of the potential of AR combined with LA is still lacking. Similarly, no tools have been identified that can contribute toward the evaluation of such practices. Therefore, in this work, we have proposed the supplementary use of AR in education as it has also been linked with positive gains in students’ cognitive thinking and practical skills [3]. In addition, we have outlined and synthesized the processes that should be performed prior to, during and after the carrying out of the respective interventions.

By connecting LA practices [67,68] and AR-supported instruction [3], a pathway is shaped wherein connections between the academic and professional world are made and the elements of implementation are revealed. In addition, the configuration of an interactive environment capable of supporting student learning through reflection and application is proposed. Furthermore, the analysis of such programs with LA can provide remarkable information, which is difficult to recognize due to the diversity of academic and professional practices over the long term. This work mitigates this issue and provides a way to understand whether AR-supported instruction can affect trainees’ engagement in academic and professional practices. Nonetheless, there is still much work to be done regarding the effectiveness of the AR-supported instructional methods, as a wide range of learning opportunities are warranted.

6. Implications

Digital literacy and skill acquisition have been recognised as key competencies in the digital age. The present work contributes to the application of AR technology in educational research in the following ways: (a) it proposes an ongoing framework which suggests the primary importance of using AR for developing students’ competencies and (b) it provides a deeper understanding of how to assess such digital competencies based on LA measurements.

From the instructional designer’s perspective, we highly recommend educational technologists to focus on the design and development of AR-oriented training programs to support and/or enhance trainees’ digital competences and boost their learning performance. To this end, the user interface of the applications needs to be simple, but with insightful and “fading scaffolding” exercises, in accordance with the instructors’ recommendations. Likewise, from the educators’ perspective, some of the most key strategies presented by [24] encompass project-based, activity-based, and problem-based learning conditions that can assist trainees’ productivity in practice and can be utilized by instructors and educators to develop valued academic and disciplinary practices in different educational fields. To this end, the following implications for practice and/or policy are proposed:

• Instructional designers should be trained in how to use appropriate software and hardware related to AR technology.
• Application developers and learning technologists should explore design solutions related to the use of AR technology in “hands-on” learning practices.
• Policymakers should not neglect the socio-cognitive and cultural effects of using interactive AR applications combined with LA to inform trainees and practitioners about their performance and outcomes.

7. Conclusions and Future Work

The present work aimed at responding to a timely challenge by proposing a conceptual framework related to the combination of innovative technologies (AR and LA) in both formal and informal educational contexts. Based on the theoretical foundations of a design framework [7], the proposed conceptual design can assist educators, instructors and scholars to investigate potential connections and complexities of educational practices after considering the diversity of the instructional contexts, the available resources and the design characteristics of the educational activities. The integration of LA methods can enhance the education continuum by introducing alternative instructional and assessment approaches.

However, regardless of the context and the utilized tools, the instructor’s contribution is always necessary, especially as far as the guidance and the assessment processes are concerned. From this point of view, instructors and instructional designers should receive appropriate training to effectively support the design and deployment of interactive and engaging professional and/or academic learning scenarios.

Finally, although the proposed design framework can be widely applied for designing, developing, and implementing AR-supported instruction for trainees and practitioners who want to apply their theoretical knowledge via interactive environments to improve learning, LA integration can raise several managerial issues in terms of privacy and ethics [69]. This is particularly important for trainees with different socio-cognitive backgrounds, health problems or intellectual disabilities, and thus there is a reasonable need to require informed consent.

Future works should consider the elements related to the evaluation of the instructional approaches that concern the training of students with learning difficulties or disabilities. In this regard, instructors need to know a priori if there are students who have difficulties in terms of providing additional support and guidance. In addition, it would be useful to explore teachers’ perspectives, ideally in the context of longitudinal studies, regarding the trajectories and transitions of temporary online learning contexts to the new blended learning setting that will most likely be in place in the forthcoming years.