Transforming Interactive Educational Content into Immersive Virtual Reality Learning Objects

Abstract

Currently, the integration of immersive virtual reality applications into learning management systems is limited, which disadvantages kinesthetic learners, as the majority of learning material is mostly written or visual in nature. This research presents a method on how to transform existing conventional learning material into immersive virtual reality and integrate it into learning management systems, demonstrating it with a pilot study and an immersive virtual reality application prototype. The prototype application uses WebXR technology and provides integration into the Moodle learning management system using the Sharable Content Object Reference Model standard. The developed learning object was evaluated using the System Usability Scale, Web-based Learning Tool evaluation scale and the igroup Presence Questionnaire. The results of the pilot study revealed that the learning object performed above average in all tested fields and is preferable to established non-IVR solutions.

1. Introduction

Every individual has a preferred way of thinking, processing, and understanding information, and this diversity of learning styles has attracted considerable attention in educational research. The four primary learning styles commonly identified are aural, visual, read/write, and kinesthetic (tactile) learning styles [1]. However, most of the learning material and information presented in e-learning are tailored primarily for visual and auditory learners, often a disadvantage for kinesthetic learners [2]. Kinesthetic learners are better suited for active learning environments, where physical activities and hands-on experiences play a crucial role in their understanding and retention of knowledge [2].

Thankfully, immersive virtual reality (IVR) technology underwent increasing technological developments in recent years [3]. The reduction of its cost throughout the years made it an attractive alternative solution to traditional laboratories and equipment [4]. Initially popularized for entertainment, gaming, and training purposes, IVR has extended into the realm of education, where IVR is being recognized as a tool that can transform approaches to learning [5], allowing users not only to experience virtual environments but engage multiple senses simultaneously, including visual, auditory, kinesthetic interactions. This level of immersion introduces new opportunities for presenting learning material and facilitating learning for kinesthetic learners [6]. Subsequently for kinesthetics, IVR integration in educational environments is beneficial, as it befits their preferred learning style encompassing hands-on experiences and physical activities [4]. In IVR environments, students can actively interact with virtual objects, manipulate scenarios, and conduct practical simulations, thus promoting active learning experiences. Such IVR applications can increase the motivation [7], interest [8], and retention of students’ knowledge [9,10], ultimately improving their learning outcomes [11].

Numerous systematic reviews found in scientific literature have analyzed various research works that apply IVR technology for different educational [5,12,13,14,15,16,17] or industrial training purposes [18]. These papers range from proposals to full studies of developed solutions, or their effectiveness evaluations. However, not all works are specifically focused on the implementation of kinesthetic learning using IVR technology. Additionally, while IVR does bring promising potential into education [12,14,18], there are pressing challenges that need to be addressed. One challenge that requires consideration includes the development and integration of desktop or IVR applications into an existing learning management system (LMS) [19]. Another challenge is that it is expensive to develop tailored IVR educational applications [6,12,17]. Depending on the complexity of the application, it may require a wider set of design features or detailed 3D environments, which in turn can increase the cost of development. This can be highly probable with IVR applications adapted for kinesthetic learners as it requires a higher level of interactivity, as one way to reduce or share the cost of development and management of any educational content was through the concept of learning objects [20,21].

Learning objects (LOs) are reusable and sharable packages containing educational content and its metadata description [21]. There is no clear definition on what type or scope of educational content should be included in a learning object, meaning that it can contain any volume of text, image, audio or video resources, interactive content [22,23], etc. However, LOs’ metadata must be properly annotated as they help categorize LOs and allow users to find suitable content for a specific context more efficiently, which is key to an LO’s reusable and sharable nature [20,24]. A correct structure of LOs and their metadata are defined by standards such as Learning Object Metadata (LOM) or the Sharable Content Object Reference Model (SCORM) [20].

It is worth to consider developing IVR educational applications as LOs since they are beneficial to IVR use in education [17]. First, LOs can be stored in LMSs and directly delivered to learners through them [25]. The process of using IVR educational applications becomes more convenient as it does not require any tedious solutions or workarounds for learners to use them [5]. The content is available in an already familiar environment. Secondly, when a learner uses an IVR educational application, their actions and results can be recorded and passed onto the LMS. Educators can then review an automatically composed report of the learner’s performance and evaluate them in the integrated familiar environment. Lastly, following LOs’ principals of modularity and reusability, IVR educational applications can be developed as smaller pieces of content, each covering a single specific objective or task of a wider educational context. That way, different courses can be supplemented with small reusable IVR educational applications according to their needs.

With an established development pipeline, new IVR educational applications can be created as LOs and help solve the aforementioned challenges of IVR educational application development and integration. Improving the accessibility of IVR through familiar LMSs can encourage its use in education even more. In turn, kinesthetic and other types of learning that leverage IVR can become more viable and easier to use.

The aim of this research is to evaluate the usability of the developed IVR kinesthetic-based learning system as a learning tool in the Lithuanian higher education sector using standardized questionnaires and determine whether such transformation of interactive learning tools into innovative IVR learning systems is an attractive alternative. In the paper, we discuss the process of enhancing the general learning process with immersive virtual reality. We conducted a pilot study that evaluates a developed prototype and performs quantitative analysis of developed methods and implemented solutions. Section 2 provides an overview of the general trends of IVR usage in the education process and presents the current state of research in this area. The concepts and general process of including immersive virtual reality technologies in higher education teaching are presented in Section 3. The research methodology is described in Section 4. Implementation that integrates developed IVR application into learning management system is defined in Section 5. Finally, the settings for the experiment and experiment results are defined in Section 6 and Section 7, respectively. The conclusions of research results, findings and future research plans are discussed in Section 8.

2. Related Works

In the context of education, IVR has been applied in many different areas including science, engineering, healthcare, social sciences, emergency and safety, manufacturing, and others [13,15,16,18]. Wide application of IVR technology can be attributed to the positive benefits that it brings to education. It can provide learners with immersive and engaging experiences [6]. As the content and scenarios are virtual, they can be completed repeatedly and safely, avoiding real risk factors [16]. Through IVR, learners can interact with virtual environments and imitate various hands-on experiments [5]. With all of these positive properties, there is still a need to identify how they affect the learning process and which design features or technology characteristics fit best in specific educational circumstances [17].

Checa and Bustillo [14] identified four game typologies used in IVR educational applications: explorative interactive, explorative, interactive experiences, and passive experiences. The authors highlight that the most common typology was the interactive experience that permits interactions with restricted movement. They also noted that explorative games were underrepresented as they do not provide clear objectives to the user and are thus less meaningful in a learning environment. Pellas et al. [5] pointed out that simulations were the most common type of instructional design used in IVR applications for higher education as they allow users to gain knowledge through a more problem-based approach. Won et al. [17] had a similar observation, where most studies used IVR applications as simulations, offering the ability to perform tasks for practice, or virtual world representations, visualizing objects or events related to the learning context.

Identified evaluations of different IVR educational solutions also varied. Radhakrishnan et al. [18] separated IVR educational solution evaluations based on data objective and subjective measures, where any factual data were designated to the former category (such as completion time, reaction time, errors, etc.), while measures based on individual participant experiences and how they felt during experiments and exposure to IVR were included in the latter category. The authors categorized the subjective measurements even more, into four different categories (cybersickness, usability, task load, and immersion) highlighting some standard evaluation tools (such as System Usability Scale (SUS) [26], task load index, igroup Presence Questionnaire (IPQ) [27], and others). Though, based on their results, they noted that less than one-fifth of the reviewed works used any kind of standardized tool to evaluate their IVR educational solutions based on the set subjective measurement categories. In their systematic review, Checa and Bustillo [14] noted a lack of usability and immersion evaluation, which have an impact on a successful experience in IVR, while user satisfaction evaluation was conducted in most studies. Kamińska et al. [13] also noted a similar observation, stating that the use of well-documented and easy-to-use tools such as SUS are rarely used or not used enough to evaluate IVR educational solutions.

IVR educational solutions that utilize kinesthetic learning or evaluate its use were identified further. For kinesthetic and immersive plant facility operation safety training, Bae et al. [9] proposed a system defined as a haptic augmented virtuality system with an encounter-type haptic display system. As a problem, the authors state that current haptic devices used in conjunction with IVR cannot provide the necessary haptic feedback for such training scenarios because the variation in different types of tools and their placements inside plant facilities are too great. The proposed system could use a robot to prepare and place haptic feedback devices in front of the trainee depending on what tool he is supposed to interact with within the IVR environment at the exact timeframe.

In a follow-up study, Bae et al. [28] attempted to evaluate the effectiveness of their proposed collocation solution for real and virtual tools by comparing the user performance between virtual and real environments and the effectiveness of enactment for memory-based tasks. The collocation effectiveness was measured based on reaction time and accuracy and had a combination of four experimental conditions (with HMD, without HMD, and sitting down, standing up conditions). For the experiment, all participants had to perform a task that required pressing a button and a task that required grasping a wheel in all four conditions. For the evaluation of enactment effectiveness, three different experiment conditions were used: virtual reality-based environment, video-based environment, and text-based environment. Participants were randomly selected for each condition and had to perform and/or memorize a list of procedures in their designated environments, followed by a recall test. The authors conclude that based on the results of the collocation effectiveness experiment, their proposed method is a viable solution to use for the purpose of imitating real tools in a virtual environment. In the enactment experiment, the results showed that memory recall from the enactment virtual environment was weaker than other environments, but it was more accurate in a follow-up recall test.

A proposal by Kaluschke et al. [29] presented an IVR system with haptic feedback for dental surgical skill training. The goal of the system was to create an IVR training solution that makes it possible to transfer and teach haptic or kinesthetic sensations that are required during dental or other surgical procedures. The system allows the teacher and a student to share a virtual environment in which they are provided with a virtual dental station, a patient, and a virtual dental drill. Using a Phantom Omni haptic device, the teacher can perform dental procedures on the virtual patient while a student observes. The forces that are used during the procedure are transferred to the student’s haptic device, mimicking the teacher’s performed actions with the virtual drill. Alternatively, a student can perform the specified procedures and the teacher receives the haptic feedback from the student’s actions. Based on this, the teacher can evaluate the student.

In the study on the IVR solution, Kaluschke et al. [11] evaluated the impact stereoscopic view and tool alignment had on learning effectiveness and skill transfer in IVR training. The authors hypothesized that stereoscopic view and hand-tool alignment has a positive impact on learning effectiveness, and that these conditions also increase the suitability of a simulator for skill assessments. The study used four separate participant groups with different pairs of IVR stereoscopic or monoscopic view and hand-tool alignment or misalignment conditions. The stereoscopic condition used the standard IVR rendering, where a different image within the HMD was displayed to each eye, creating a natural-feeling stereoscopic vision, while the monoscopic view condition displayed the same rendered image to both eyes. The hand-tool alignment condition had the real haptic controller device aligned with the virtual drilling tool that was seen within the virtual environment, while the misalignment condition had an added offset to the haptic controller device, creating a sense of discrepancy similar to a desktop setup.

Overall, the authors of the study [11] concluded that hand-tool alignment and stereoscopic view had a positive effect on skill transferability from real life to the simulator, as there was a positive correlation with the participant pre-training scores, meaning that such type of IVR could be used as an effective tool for skill assessments. On the other hand, stereoscopic view had no statistically significant impact on real-world learning gain because it required the participants to work on small elements within the virtual environment, which proved to be complex with the HMD’s small-resolution screens.

The work of Koutsabasis and Vosinakis [30] presents a desktop-based educational application based on kinesthetic interactions in which users can digitally create a historical figurine with the goal to learn about the process, tools, and materials used in such a craft. The whole process of creating the figurine is separated into three different stages. Before each stage, the user must select the proper tool for the required task and use it on the figurine. These interactions are implemented by using the Leap Motion device, which tracks the user’s hand position and gestures. By physically moving their hand and performing a grasping gesture, the user can pick up tools and use them by mimicking the motion of their real use. In this case, instead of reading or observing other types of media about such a process, the user learns about it by imitating the actions that are needed for the simulated task to be completed, which makes the learning process more kinesthetic.

From their usability evaluation study, Koutsabasis and Vosinakis [30] highlighted a depth perception issue that occurred, where some users faced difficulties when trying to work on the virtual figurine because at the beginning, they were holding the virtual tools too far away from it, thinking that they were performing the action correctly. As one of the possible solutions for this, the authors proposed the use of some sort of stereoscopy such as an IVR HMD.

In a study comparing desktop-based and HMD-based displays, Krokos et al. [10] attempted to find out if participants could recall information better from a virtual memory palace using an IVR HMD than a traditional desktop setup. The authors hypothesized that because of the way an HMD can immerse a person into a virtual environment, allowing them to perceive the environment as if they were in it, it could assist them in recalling observed information better. In the study, participants were presented with famous people or character faces as information to memorize and recall. The results of the study really did show better information recall when observing the environment through an HMD instead of a desktop-based display. Such positive results can portray IVR technology as a useful and viable tool for better information retention or cognitive improvements. Although, because interactivity was limited within the applications used in this study and it was not the aim of the study, there were no insights of using IVR technology for kinesthetic learning through physical activities.

Vázquez et al. [31], in their work, presented a kinesthetic IVR learning solution called Words in Motion, which could be used to enhance second-language vocabulary learning lessons. The solution works as a recognition system, where each word is paired with a corresponding action. Students perform actions to specify words and create an association between words and their body movement. The system can be trained to make new pairs of words and actions, and lists of available actions can be created that could be used for engaging gamified activities.

To evaluate the effectiveness of the kinesthetic learning method in their proposed solution, Vázquez et al. [31] conducted a between-subject study comparing three groups of participants with different conditions. These conditions included learning through text only, learning in IVR with no kinesthetics, and learning in IVR with kinesthetics. At first, each participant was presented with a sequence of 20 words to memorize in their respective conditions. The words were presented twice, 15 s each. Text-only condition participants were shown pairs of words in Spanish and English on a monitor. Both IVR condition participants were presented with the words in IVR, the difference being that with kinesthetics, participants had to perform a movement that was shown with a motion signifier and repeat it to create an association with the word, while the participants with no kinesthetics had to only observe the action signifier. Immediately after the word memorization, each participant was prompted with Spanish words which they had to translate to English. They had to complete the same test exactly after a week. The results showed that text-only participants outperformed both IVR groups in the immediate post-test. On the other hand, the test results after a week showed that the number of recalled words from the text-only and IVR with kinesthetic groups were essentially the same, while the IVR without kinesthetic group results were even worse. Although IVR did not surpass the results of a more traditional way of learning, the authors highlighted that overall, the retention rate of words in the kinesthetic IVR was better than the text-only group.

Hoang et al.’s work [32] presented an empirical evaluation comparing the use of an IVR kinesthetic learning application with traditional training using slides, photos, and videos for industrial hydraulics machine maintenance training. The training application used in the comparison was developed for the Oculus Quest 1 and 2 platforms using the Unity3D game engine. Within the application, a virtual environment was built replicating a real-life hydraulic power unit. Three maintenance tasks were prepared for the training which included changing the pressure filter, offline filter, and inspecting the accumulator pressure. These tasks were designed based on kinesthetic learning. To complete them, trainees had to follow real-life procedures and perform real-like actions on the task relevant components of the power unit by interacting with them with physical hand movements and virtual tools. To use a tool, a trainee must first pick it up by moving their hand close to the tool and pressing and holding the Oculus Quest controller’s trigger button to grip it. Then, while holding the tool and keeping it close to a component, the trainee can physically move their hand in a way that mimics a real use of the tool. As an example, the authors describe the use of a virtual spanner where a trainee would pick up the spanner and, while maintaining contact between the spanner’s head and a correct bolt, they would motion the hand back and forth to use it.

For their study, Hoang et al. [32] recruited 2 groups of participants. One group was made up of university students with various degrees of experience. The other group was made up of industry professionals with prior experience in the field. For the student group, a between-subject design was used, so the group was split into two additional subgroups where one subgroup received training through the IVR application while the other one received training in the traditional way through slides, photos, and component samples. The industry group was not split and received both types of training with reduced time duration and learning content for each of the types. For the comparison, data were collected through training experience and environment, and satisfaction and self-confidence in learning surveys, educational practices questionnaire, NASA task load index, practical performance of the maintenance task assessment, and a sense of presence and sense of embodiment assessments. According to the analyzed results, the authors concluded that there was no significant improvement in maintenance task completion performance between training with the IVR training application and traditional training. On the other hand, the authors highlighted that the participants from the student IVR subgroup felt less time-pressured when learning than the participants from the traditional subgroup, stating that more training could be achieved when using IVR.

Research by Pfeiffer et al. [33] presents a brief study and an IVR training solution that uses electrical muscle stimulation to give feedback to a trainee. The concept of the solution is to help and guide trainees through any virtual training scenario task by providing them with cues and hints using haptic feedback through electrical muscle stimulation. The feedback is provided through an electrical muscle stimulation by either encouraging the users hand to perform an action or discouraging them by preventing them to perform the action. The designed solution in the study used Unity3D, SteamVR, and HTC Vive to implement and use the training scenario with a STIM-PRO x9+ to generate the required electrical muscle stimulation pulse for the feedback. A within-subject design was used for the work’s user study. Each participant had to complete a task that required memorizing and pressing a sequence of virtual buttons. Four feedback and three sequence conditions were used in the experiment. Feedback conditions included no feedback, reenforcing feedback, preventing feedback, and a combination of reenforcing and preventing feedback. Sequence conditions required a different number of buttons to press in task sequences: three, five, or seven. To evaluate the solution, a custom questioner was used inquiring about the comfortability of the electrical muscle stimulation and how much each of its feedback supported the user in their task completion. Overall, the users did not find the stimulation uncomfortable, and mostly all feedback types supported them in their task, though the combination of reenforcing and preventing feedback was favored the most. As per the task completion results, users failed to complete the task as the number of task sequence increased, while there was no feedback provided. With feedback, only one user failed to complete the task.

Dürr et al. [34] presented an IVR solution for learning how to handle patient transfer during different conditions such as unique patient impartments. The solution is separated into two phases: instruction and training phases. During the instruction phase, learners are presented with an animated virtual patient and instructions in text that demonstrate and explain how to perform patient transfers in certain situations. The learner can interactively control the animation of the demonstration, audio, and highlights of the correct patient pose. During the training phase, the learner can interact with the virtual patient by moving their body parts and conducting the patient transfer. Also, the learner can control the patient transfer steps and other aspects of the training, like in the instruction phase.

A summary of reviewed related works and our work is provided in

Table 1. Summary of related works.

Reviewed Research	Implementation	Performance Evaluation	User Experience Evaluation
Bae et al. [9,28]	A system that uses a robot to automatically prepare haptic devices as industrial tools to use during training for users with an HMD.	Measured reaction time and accuracy for collocation evaluation. Measured memorized and performed actions for enactment evaluation.	Did not conduct a user experience study.
Kaluschke et al. [11,29]	A system using a Phantom Omni haptic device with an HMD for dental surgery training.	Measured participant performance scores after conducting pre-training with different IVR conditions.	Did not conduct a user experience study.
Koutsabasis and Vosinakis [30]	Desktop-based learning application using a Leap Motion controller to interact and perform figure carving actions with hands.	Measured task completion time and error count.	Evaluated the application through user observation, feedback, and fun toolkit questionnaire.
Krokos et al. [10]	Desktop- and IVR-based virtual palace applications for information recall.	Compared recall performance and confidence between using the desktop or the IVR application.	Did not conduct a user experience study.
Vázquez et al. [31]	Application for learning second language words through kinesthetics by performing hand motions associated with the words.	Compared how well participants recalled new second language words between learning through text only, IVR by only observing motions, and IVR by performing motions.	Did not conduct a user experience study.
Hoang et al. [32]	IVR kinesthetic learning application for industrial hydraulics machine maintenance training.	Measured participant scores in a real-life performance evaluation after conducting training in IVR.	Used VARK questionnaire to measure participant learning style before the experiment. Training experience and environment, satisfaction and self-confidence surveys, educational practice questionnaire, and NASA task load index used to evaluate user experiences.
Pfeiffer et al. [33]	An IVR training system that gives feedback by stimulating muscles through electrical pulses.	Measured the completeness of a given task.	Evaluated how comfortable and helpful the system is with a custom questionnaire.
Dürr et al. [34]	IVR training application for handling patients with impartments during their transfer.	No study was conducted.	No study was conducted.
This work	IVR educational application integrated into an LMS for learning human skeletal structure and anatomy.	Measured completion time, error, and usage of additional learning material.	Used web-based learning tool scale, IPQ Group Presence Questionnaire, System Usability Scale to evaluate user experience.

. Most of the analyzed related works were focused on applying kinesthetic learning through IVR in specific domains. The only exceptions are the Koutsabasis and Vosinakis [30] and Krokos et al. [10] studies. The former used a desktop application to apply kinesthetic learning, not an IVR application. The latter compared an IVR application with a desktop application for memory recall task, not a kinesthetic task. Though these works were included as one attempt to implement kinesthetic learning, the other compares the use of a desktop application with an IVR application for educational or cognitive purposes. However, none of the works attempted to integrate their IVR applications into an LMS.

In case of learning effectiveness or user performance measuring, all studies except that by Dürr et al. [34] used similar metrics or a combination of them such as task completion time, task completion accuracy, and evaluation score to evaluate their solutions. However, only three of the reviewed works conducted any kind of user experience study. Of those three, Koutsabasis and Vosinakis [30] and Hoang et al. [32] used standardized tools to evaluate user experience, while Pfeiffer et al. [33] used a custom questionnaire. A lack of user experience studies based on standardized tools matches the observations of other works [13,18].

The analyzed related works mostly present their own developed stand-alone applications and performance evaluations. A lack of solutions integrating educational IVR applications into LMS was identified. Such an integration could simplify the use of IVR in education by enabling it through a familiar environment for learners and educators. A process of developing an educational IVR application and integrating it into an LMS could be determined and defined by implementing such an application.

3. Proposal

Currently, in the field of higher education the integration of desktop or IVR applications into LMS is limited [19]. The conceptual model for the entities exiting in this transformation is presented in Figure 1. The general concepts can be divided into two main parts: (1) general education concepts such as study course, teacher, students, and learning material; (2) IVR integration concepts such as the VR learning object which represents the development of the IVR application, environment which represents the 3D learning environment, task which defines learning tasks performed within the IVR applications, and interactive objects with which these tasks are performed.

To modify the current learning material with IVR learning objects, the transformation process of the current learning material detailed in Figure 2 is to be used. This process starts by identifying learning goals with a focus on the interactive technology. When the learning goals are defined, the study course syllabus should be developed, denoting teaching and evaluation methods as well as creating a list of things that could be implemented in IVR. If learning goals or syllabus already exist, they should be revised and updated by including tasks and steps, requirements that would benefit from interactivity and increased availability of teaching resources and environments in interactive virtual reality. With completion of syllabus, the interactive virtual reality application development starts. The teacher prepares the requirements that denote required activities, 3D environments, and required interactive objects. The requirements should contain enough details to allow 3D artists and immersive virtual reality developers to take over the development process if the teacher lacks such competences. These details may include information on the domain area itself, what type of tasks should be performed, in what way they need to be completed, what counts as a mistake, what models should be used, what data should be recorded, and so on.

The IVR application development starts with 3D environment modelling. The development of such environments commonly uses 3D game engine software so the result for this step is a 3D environment which facilitates the interactive learning tasks. If the selected learning method requires interactive objects in its defined tasks, the second development step should contain 3D modelling of said objectives and implementation of logic powering the interaction in the immersive virtual world. This development cycle of interactive objects should be performed until all the required objects are ready to be included into the task. The final step is to integrate interactive objects with developed 3D learning environments.

After the initial development of the IVR learning object is complete, the usage tracking integration should be performed. Such integration allows monitoring of the students’ performance within the IVR environment and sending the logged information to the LMS. This enables individual tuning of task requirements and simplifies grading. The performance tracking starts by specifying the required metrics and implementing the API calls to the LMS.

The final step in the transformation process requires using the developed IVR solution and wrapping it into an application supported by the learning management system. Commonly, the wrapping requires transformation of the developed solution to be supported in web-based environments with support for information transfer to third-party resources.

To test the feasibility of the proposed transformation process, the pilot study implementing integration into LMS systems and prototype IVR application was developed. The developed study course was deployed to the Kaunas University of Technology Moodle LMS [35] and used for experimental evaluation. The implementation details of the prototype application and integration are presented in Section 5. The following section defines the research methodology that the work was based on.

4. Methodology

A single existing interactive learning application was transformed into an IVR LO using the defined method and integrated into an LMS, in an effort to evaluate the IVR LO’s usability attractiveness as an alternative to current interactive learning applications.

The task on the topic of human anatomy was selected due to the fact that transforming spatial bone structures into a two-dimensional plane oversimplifies the topic by reducing and flattening details, which might hinder learning and understanding. The physical movement in the IVR environment potentially improves the learning because of the introduction of the kinesthetic aspect, while also requiring to visualize and remember the shape and the position of bones.

Two data collection methods were chosen—one to evaluate participant IVR performance, and another to help evaluate the LOs based on established scales via questionnaires. For the performance evaluation, three metrics were outlined: number of mistakes made, time spent to complete the task, and number of hints used. These metrics could help determine how well participants manage to deal with a given task under potentially new and unusual conditions. The completion time and number of mistakes could indicate ease of use of IVR, while higher times and a larger number of mistakes could suggest difficulties. To evaluate the LO’s Web-based learning tool (WBLT) scale, SUS and IPQ surveys were selected. WBLT was selected as it is used to evaluate learning tools based on learning, engagement, and design metrics. SUS and IPQ were selected as they can be used to measure the developed solution’s usability, immersion, and visual quality.

The experiment compared the standardized learning object and IVR experience. The experiment was conducted with participants who provided verbal consent. Each participant performed learning tasks in a non-IVR application, and then the proposed IVR implementation. Later, their experience was recorded in an anonymous survey. The WBLT questionnaire had to be answered twice by the participants—for IVR and non-IVR experience. Finally, statistical analysis of the experiment results was performed to explore the findings.

5. Implementation

The resulting prototype is a kinesthetic educational IVR application developed using Godot version 3.5 [36], designed to familiarize the user with the structure of the human skeleton (Figure 3). The goal of the application is to assemble a spatial puzzle—a human skeleton—from the bones scattered on the floor by manually picking them up and inserting them into the correct place within the silhouette of a human body. The bones are fragmented into 29 groups, instead of each bone being separate—larger bones, such as the arm and leg bones are separate, but smaller bones like the ribs, spine, wrist, and feet bones are grouped together. This makes the application approachable for a quick introduction or a refresher, without having to delegate a large portion of time for one task.

Objects can be picked up from the ground by physically leaning down towards the floor and pressing the top trigger on the IVR controller while the controller is near an object. Subsequently, releasing the trigger also drops the picked-up object. The prototype also includes additional objects which are used to aid the user with their primary task: a scanner to display the name of a scanned bone, an anatomy atlas to give more information about its function and relation to the rest of the body, a scoreboard to display how many bones are left, how many mistakes were made, and so on. These objects are located at different places in the virtual environment, which encourages the player to move in various ways—bend, crouch, turn around, reach up—thus promoting kinesthetic learning.

The core tenets of LO are discoverability, reusability, and interoperability [20,22], meaning that in order to turn a IVR application into an LO, it is necessary to make sure that the created program can be reused many times across various different LMSs, while ensuring integration and compatibility between the two.

One way to establish communication between an LMS and an IVR application is by launching the IVR application through the LMS itself. Since most commonly used LMSs are accessed without using proprietary software, it would be best to have the IVR application work from within the web browser as well. This is most easily accomplished by exporting the final IVR application into an HTML5 format, which is possible using most multiplatform game engines capable of creating IVR applications. The developed IVR prototype was built into an HTML5 format through Godot version 3.5.

In order to access IVR in HTML5, the project incorporates WebXR [37] technology, an open standard which allows web applications to work with virtual and augmented reality hardware. This means that if the application is exported in a supported format that a web application uses, such as HTML5, it is possible to enable access to an IVR application straight from the web browser. This has the added benefit of not having to download or install anything on the machine, making it convenient for testing the project during development, thus easily accessible on various platforms.

However, to fully integrate a standalone HTML5 application into an LMS, it is necessary to create some sort of communication interface between the two. Since each LMS is different from one another and LOs are meant to be reusable, it is necessary to adopt some kind of standard for compatibility. Thankfully, there are several standards to choose from, such as SCORM, H5P content, and IMS content packages, though not every LMS supports every kind of standard. For the purposes of this research, SCORM suited best (Figure 4).

A SCORM package is a Package Interchange File that contains specific contents as defined by the SCORM standard [20] and has all the relevant data needed to transfer learning content to an LMS. SCORM is a set of specifications that provide interoperability for accessing and reusing web-based learning tracking. Using this standard, one may create a reusable LO that automatically records the results of certain activities, much like a test.

Converting an IVR application into a SCORM package is a relatively simple process, requiring only a few steps. Once an application is exported into an HTML5 format, it must be archived into a .zip package along with several SCORM-specific files. These files include several .xsd manifest files, a library file in .xml format defining the package structure and listing all of its contents, as well as a JavaScript API wrapper containing all of the functions used to communicate with the LMS, as defined using the SCORM standard [38]. To create the SCORM package, the developed IVR prototype in HTML5 format was archived with the required additional .xsd, .xml, and JavaScript files. The .xml file was modified based on existing examples in the official SCORM website [38], changing the title, organization, identifier, and resource tags. SCORM 1.2 was used for this purpose.

These functions allow the use of various commands, such as allowing the LO to parse data from the LMS environment, such as the player’s name and how many attempts they have made, or allowing the LO to submit data to the LMS directly, like setting a question as answered correctly or not, noting how much time was spent while completing tasks, determining the final grade estimate, etc. Using these functions, the LO automatically writes data into the LMS, similar to an online test, which later allows educators to review student performance, how much time they used to complete a task, and any other information that gave a better understanding on what could be improved and better emphasized during the learning process. These JavaScript functions can be called from within the IVR application’s code directly, which allows for parsing and submitting these data at any point in real time.

The LMS chosen to perform the study was Moodle [35]. Moodle is consistently regarded as one of the top-rated and most popular LMSs [39]. The reason why it was chosen for the study is because Moodle is compatible with SCORM, unlike other popular alternatives such as Google Classroom or Edmodo.

Inserting the IVR application into a Moodle course is performed by creating a SCORM activity in a Moodle course and uploading the .zip package interchange file. If everything is performed correctly, the designated users would be able to access the activity and begin the IVR application and access the virtual environment, provided they are equipped with compatible IVR equipment. Once they complete their tasks within the application, the educator can access the results in the ‘Reports’ tab, shown in Figure 5. This view only gives an overview of the participants’ results, showing the number of attempts, their date and time, as well as the final score for each one of the attempts. More detailed information for each attempt as specified during development, such as information on each task, time taken, mistakes made, and so on, can be accessed by pressing on any attempt number in the ‘Attempt’ column.

After developing the specified IVR educational application prototype and integrating it into the Moodle LMS, the application was prepared for experimental evaluation with select users.

6. Experiment Setup

A quantitative exploratory study of the developed IVR application in educational activities was performed. To gain insights into the application and usability of the developed prototype, a strategy was chosen to include a selected group of information technology undergraduate students from the course “Information Systems Graphical User Interface” due to their familiarity with the concept of system usability, as well as expertise in the principles in website and GUI evaluation. Any student from this course falls in this category—students’ ages range from 20 to 25 years old; they have finished secondary education that includes a biology course that introduces knowledge about basic human anatomy. Even so, though the undergraduate students are computer literate, the majority of them did not have any experience with IVR. The choice to include participants with no prior IVR experience was made because the experiment evaluates an IVR LO as a tool to be included into appropriate learning curricula.

Permission to conduct the research during the course’s lectures and invite its students to participate in it was given by the course’s professor. Participating students were volunteers, no reward incentives for participating were offered. The developed educational IVR application prototype was not part of the “Information Systems Graphical User Interface” study course, it was only included as additional material for conducting the experiment.

The experiment was conducted according to the following procedure. Each volunteering participant was briefly introduced to the experiment and its research purpose, they were informed about the procedure and what they will have to complete. The volunteers were also notified that they could opt out of the experiment at any time. After receiving their consent to participate, the experiment began.

First, the participants had to test the developed educational IVR application (Figure 3) by completing its task—constructing a human skeleton. Each participant was instructed on how to use the application, how to pick up objects, and how to use the available learning material. The IVR hardware that was used for the experiment was the Meta Quest 2 headset with its controllers, using the default Horizon OS v65 (formerly known as the Android Open-Source Project operating system software). The application itself was hosted on the Kaunas University of Technology Moodle environment (Moodle v4.1) and launched through the Meta Quest Browser. No time limit was set to complete the task.

After completing the task in the educational IVR application, each participant had to construct a human skeleton in a certified desktop 2D interactive LO (Figure 6) from “Emokykla.lt”—a website authorized by the National Education Agency of Lithuania, hosting various books, programs, recommendations, digital teaching aids, and tools for various age schoolchildren. The interactive LO was used in the Google Chrome web browser on a computer running the Windows 10 operating system. When participants had completed tasks in both applications, they were asked to fill in a form containing a series of questions about their experiences using the applications.

The research instrument is an anonymous questionnaire consisting of 54 questions designed to evaluate the developed IVR educational application prototype and compare it to the desktop alternative. The questionnaire was constructed and issued using Google Forms. Question blocks of the questionnaire:

• Sociodemographic data (IVR experience)—one question
• WBLT questionnaire prepared according to the Web-based Learning Tool (WBLT) Evaluation Scale [40]. This evaluation scale is intended to determine the effectiveness of learning tools and objects. The WBLT Questionnaire consists of 13 questions representing 3 constructs: learning, design, and engagement.
• IPQ [27]—igroup Presence Questionnaire, designed to assess the sense of presence experienced in a virtual environment. This questionnaire measures the subjective sense of being in a virtual environment. The IPQ consists of 14 questions representing 3 subscales: sense of presence, involvement, and realism.
• SUS [26]—the System Usability Scale questionnaire is a 10-question questionnaire used to assess the usability of a product, system, or interface.
• Open questions—at the end of the questionnaire, participants were asked three open-ended questions asking them to share their experiences and provide insights, feedback, or comments.

The WBLT questionnaire had to be answered twice by the participants—once to evaluate the developed IVR educational application prototype and a second time to evaluate the desktop 2D interactive LO. Since the latter did not have an immersive virtual environment, only the WBLT questionnaire was employed for comparison between the two.

In addition to the questionnaire, the result of each participant in the IVR application was recorded. The metrics included completion time, which was tracked from the start of the task until it was completed, number of mistakes made, which counted how many times a participant attempted to place a bone in an incorrect location, and number of hints used, which counted how many times a participant used the atlas to read about a specific bone. Finally, outcomes with a focus on evaluating the engagement, usability, and quality of the IVR LOs were measured.

7. Experiment Results

The experiment procedure was completed by 32 participants. All participants completed the developed educational IVR application’s assignment. On average, each participant spent 6 min 57 s, used supplementary learning material 2.5 times, and made 12.5 mistakes while completing the task. The time spent completing the task within the application varied from 3 min 31 s to 12 min 3 s and was mostly influenced by the experience in using IVR technology and the general knowledge of the human skeleton anatomy. Note that 93% of the participants indicated that they had little to no virtual reality technology experience. The low number of supplementary material usage was influenced by general knowledge of human skeleton. The high number of mistakes can be accredited to the participants’ overconfidence in the topic material and inexperience with IVR technology.

Estimates from the WBLT scale of both tested objects (IVR educational application prototype and 2D desktop interactive LO) were first compared, then a more detailed analysis of the developed application was performed. The WBLT scale scores for both objects are presented in the box and whisker graph below (Figure 7). The educational IVR application developed for the study received higher evaluations in all constructs: learning, design, and engagement. The most significant differences between the evaluation of the studied objects were found in engagement (0.943 vs. 0.472) and learning (0.894 vs. 0.557).

The relatively small difference in the design field (0.917 vs. 0.779) can be explained by the fact that both the IVR application and the 2D interactive LO cover the same learning material and aim to achieve similar learning outcomes, since the core of the IVR application is simply an adaptation of the material from the 2D interactive LO into an IVR environment.

The only statement in the 2D interactive LO’s WBLT questionnaire that received a higher rating than the corresponding statement from the IVR application’s questionnaire is item No. 8, in the design construct: “The learning object was easy to use” (0.919 vs. 0.926). This is not surprising, as IVR applications are generally more difficult to use compared to standard desktop applications, especially for users lacking experience in virtual reality technologies. However, the highest rating for a single statement in the IVR application’s WBLT questionnaire was received by item No. 12, in the engagement construct: “The learning object made learning fun” (0.971 vs. 0.412). This reveals that even though the IVR application was more difficult to use, overall, the users were more engaged, which should increase motivation and participation, potentially leading to higher grades.

The p-value calculated using the Wilcoxon Signed Ranks test for all tested fields is less than 0.0001, signifying that the comparative results are of high statistical significance (

Table 2. Wilcoxon Signed Ranks test.

Measure 1	Measure 2	Z	p
IVR Learning	2D Learning	4.319 a	<0.001
IVR Design	2D Design	3.917 b	<0.001
IVR Engagement	2D Engagement	4.834 c	<0.001
IVR Total	2D Total	4.690 d	<0.001

Note. ^b based on positive ranks. H_a μ Measure 1 − Measure 2 > 0. ^a 4 pairs of values were tied; ^b 6 pairs of values were tied; ^c 2 pairs of values were tied; ^d 1 pair of values was tied.

).

Analyzing the results of the IPQ scale by each subscale, the highest score for the developed IVR application was for the overall general estimate (4.72), and the lowest for realism (2.56).

When compared to the raw IPQ data from the igroup database [41] (n = 619), the IVR application performs better than average in all fields: general (4.72 vs. 3.40), sense of presence (4.138 vs. 3.751), involvement (3.452 vs. 2.992), and realism (2.56 vs. 2.10). When compared to only the IPQ data of game applications using stereo head-mounted displays for video (n = 29), the results stay consistent: general (4.7 vs. 2.93), sense of presence (4.138 vs. 3.475), involvement (3.452 vs. 2.560), and realism (2.56 vs. 2.068).

After calculating the SUS results from the study according to the SUS methodology, an estimated score of 85.88 was obtained (Figure 8). This is a high score, indicating that the developed prototype in this area is superior to over 96 percent of evaluated systems, with the average score being 68. It was observed that the respondent’s experience with IVR technologies can positively influence the evaluation of the program—those who were already familiar with IVR technologies may have found it easier to complete the given tasks on average.

The user evaluation experiment of the IVR educational application was conducted using three different evaluation scales. All three scale results showed that the application was rated as above average, meaning that the users had a positive opinion of the application. When comparing the WBLT scale scores of the developed IVR application and the 2D interactive LO, the former also comes out on top. This can be attributed to the interactivity and immersion of the IVR application denoting that this type of application can be an attractive alternative to established interactive learning tools.

8. Conclusions and Discussion

An educational IVR application prototype was developed and successfully integrated into Moodle, allowing for automatic recording of results in the LMS. The prototype was evaluated as part of a study by respondents who are familiar with system usability and have expertise in website and GUI evaluation. Due to the difficulties of performing usability testing of IVR applications on a large scale caused by requiring specific hardware, a total of 32 respondents participated in the study. It is possible that the study results may have been biased since all of the study participants had an IT background, are more familiar with technology, and may be quicker to adapt. Even so, the study yielded positive results. The system usability evaluation carried out according to the SUS methodology indicates that the system is of high quality, surpassing 96% of systems tested using the SUS scale, on an estimated score of 85.88 out of 100. The high SUS score indicates that the application’s usability had little to no negative effect on the rating of the application’s immersion and learning tool evaluation.

The IPQ scale results revealed that the developed prototype performed better than average in all subscales, compared to the data from the igroup database [41]: general (4.72 vs. 3.40), sense of presence (4.138 vs. 3.751), involvement (3.452 vs. 2.992), and realism (2.56 vs. 2.10). The low evaluation in the ‘Realism’ field of the IPQ questionnaire can be explained by the fact that the 3D environment was stylized to fit the purpose of the project and was not meant to be realistic. This was intentionally done for several reasons—the lack of complex 3D models and visual texture detail ensures that the LO is light in its memory use and does not increase the load on the LMS, while having the environment more detailed or open and explorative would distract from the main focus of the application—the learning material itself. Overall, the higher-than-average scores indicated that participants preferred the developed prototype application. With higher rating of immersion, a higher level of engagement may be expected, resulting in better learning material assimilation.

With the WBLT scale, the created IVR educational application prototype was evaluated higher than the established LO in all categories, comparing learning (0.894 vs. 0.557), design (0.917 vs. 0.779), and engagement (0.943 vs. 0.472). These results proved to be statistically significant (p < 0.001). The proposed IVR application scored higher in all WBLT scale parameters, indicating that the application was a preferable solution to the study participants. Higher scores of learning and engagement metrics denote increased students’ motivation to participate in the task. The relatively small difference between the compared 2D and IVR learning objects in the ‘Design’ field of the WBLT questionnaire can be explained by the fact that the IVR application is primarily meant to re-use the 2D application’s learning material while being provided in a different medium, since one of the goals of the project was to transform existing learning material into IVR. Overall, the comparison results indicate higher satisfaction when using the developed IVR application.

Considering that the research was evaluated using only a pilot study, the experiment could be expanded to include more participants to ensure that the results in each tested field are statistically significant. It would be beneficial to implement the created LO into an existing course or study program and examine the rate of acquisition of learning material among students before and after using it. As there was no evaluation of learning outcomes or results to compare against, it is difficult to comprehensively conclude whether and how much the IVR LO assists in further understanding the learning material. These issues should be addressed in future testing.