Design, Implementation and Evaluation of Deusto XRL, a Reference Architecture for Extended Remote Laboratories (XRLs)

Abstract

Extended Remote Laboratories (XRLs) have emerged as a potential solution to address the lack of science laboratories in educational institutions, a deficit that hinders the attainment of a Quality Education outlined in the 2030 Agenda for Sustainable Development. By integrating physical assets with Extended Reality (XR) technologies, XRLs provide an immersive remote laboratory experience. This study presents the development of Deusto XRL, a reference architecture for XRLs, and investigates User eXperience (UX) via a survey involving 150 high-school students. The participants were divided into groups and exposed to different online laboratory experiences: traditional Remote Laboratories (RLs) and XRLs. Feedback scores from the UX survey indicated that the XRLs received the highest ratings, reflecting a positive UX. The results highlight the potential of XRLs to enhance the accessibility and quality of science education, enabling students to engage in immersive learning experiences despite physical laboratory limitations. Future research endeavors will focus on exploring the educational implications and learning outcomes associated with XRL technology. By further investigating the pedagogical aspects of XRLs, this study aims to uncover their potential benefits and educational value. Implementing XRLs in educational settings can revolutionize science education, providing students with interactive and engaging opportunities to deepen their understanding of scientific concepts.

1. Introduction

The 2030 Agenda for Sustainable Development, adopted by all UN Member States in 2015, outlines a shared vision for global peace and prosperity [1]. At its core are the 17 Sustainable Development Goals (SDGs), which serve as a universal call to action [2]. This paper focuses specifically on the fourth SDG, Quality Education, which aims to provide inclusive and equitable access to high-quality education and promote lifelong learning opportunities for all individuals.

In the pursuit of Quality Education within the 2030 Agenda, social inequalities present significant challenges, particularly in schools with an inadequate infrastructure, including a lack of science laboratories. South American and African countries are those that suffer the most from a lack of school infrastructure. In Brazil, for instance, only 9% of K-12 schools are equipped with science labs [3], hampering effective STEM teaching, which relies on hands-on experimentation [4].

Remote Laboratories (RLs) are real experiments that can be accessed from anywhere; instead of being in a physical lab, students use a computer or a device to control equipment and see the results [5]. People can learn and explore different scientific concepts without leaving their homes or classrooms. RLs have a positive effect on students’ learning progress [6,7]. However, Nickerson et al. [7] point out that RLs present problems related to a lack of immersion through User eXperience (UX).

Extended Reality (XR), including Augmented Reality (AR) and Virtual Reality (VR), enhances the user’s visual experience [8,9], provides immersion, and motivates students through innovative learning approaches [10,11]. Nevertheless, educational experiments developed with XR do not provide real data, since they are, in principle, based on mathematical logical models.

The concept of a digital twin was first introduced by Grieves [12] in 2002, while the definition first appeared in NASA’s draft version of the technological road map in 2010 [13]. A digital twin is a virtual representation of a physical object or system based on real data, which can be used for analysis and monitoring purposes [12]. It has been widely used in the engineering industry, and in 2022 began to be employed in STEM education, focused on the development of more immersive simulations [14].

RLs lack immersion despite the fact that they use real data, while Extended Reality (XR) is immersive but lacks real data integration. Digital twins offer both immersion and real data utilization but are underutilized in education. To address these limitations, the development of Extended Remote Laboratories (XRLs) emerged as an innovative solution, integrating immersive experiences and the concept of digital twins [15]. XRLs have been developed and used since 2001 [16]; however, the existing literature on XRLs highlights various shortcomings, including the lack of reusable architectures and dependence on specific software, leading to obsolescence.

To address the identified requirements for an innovative XRL architecture, we have developed a reference architecture called “Deusto XRL”, which integrates XRL features and uses a digital twin. To validate our approach, we created a prototype based on this architecture and conducted a user study involving 150 high-school students from a public school. The students were divided into two groups, with 75 accessing the RL and 75 accessing the XRL. Subsequently, we administered a questionnaire to evaluate UX and compare the answers from each group.

After introducing the paper, we describe the characteristics, requirements, and challenges involved in developing XRLs (Section 2). We then present the proposed architecture (Section 3) and a case study on the design, implementation, and validation of the XRL pendulum and a prototype based on the architecture (Section 4). Finally, we present our conclusions (Section 5).

2. Characteristics, Requirements, and Challenges of Extended Remote Laboratories

Based on a systematic review of Extended Remote Laboratories from 2000 to 2022 provided by Silva et al. [15], we present the most relevant architectures for XRLs published in the last five years (2019–2023) in order to obtain a set of characteristics intrinsic to XRLs, including requirements for their development and challenges that must be solved.

Trentsios et al. [17], in 2020, proposed an architecture that leverages the current infrastructure and compatible front ends in both desktop- and VR-based modes (Figure 1). They introduce two visualization methods to accommodate varying degrees of immersion and independence from specialized virtual reality hardware. The first method utilizes a combination of 360 degree images to replicate the physical laboratory environment, while the second method presents the RL as a 3D-model-based virtual replica. The Unity engine is employed for both approaches, enabling the utilization of the same protocols and algorithms. The authors highlight the advantage of the Unity cross-platform compatibility.

The work by Trentsios et al. [17] offers advantages including compatibility with web browsers and VR equipment, minimizing obsolescence risk through current technology usage. However, it is limited by its specific focus on an RL scenario, excluding AR remote labs, which reduces its adaptability to various technologies. Additionally, data communication relies on Labview’s integration with Unity, potentially leading to disadvantages like hardware incompatibility, maintenance, and a learning curve.

Palmer et al. [18], in 2021, proposed a digital twin platform as a generic architecture for implementing digital twins in Remote Laboratories (RLs). The platform consists of three main elements: a Digital Twin Builder application for specifying laboratory tutorials, three processing pipelines for data, geometry, and processes, and a Digital Twin Player for the VR learning application. The platform allows for the easy creation and reuse of digital twin specifications through a drag-and-drop interface. It supports multiple hardware platforms, such as VR glasses and mobile devices.

The advantages of Palmer et al.’s [18] architecture include its consideration of current technology and the incorporation of the digital twin concept. It is also applicable to various types of RLs, making it reusable. However, a limitation of their architecture is the lack of support for AR technology, which restricts its universality. Moreover, data from the remote lab are not collected in real time but are previously collected in spreadsheets, which are inserted into the digital twin to simulate the RL’s behavior. Palmer et al. affirm that for future work, they want to develop an API to improve communication and transmit results in real time.

Nicolete [19], in 2022, developed an architecture for an Augmented Remote Laboratory using Unity and C#. The mobile application integrates AR technologies through the Vuforia framework, which enables the addition of AR markers. The application uses Vuforia’s API to detect markers through the device’s camera and generate corresponding 3D models on the screen. To achieve bidirectional interaction between AR and the RL’s hardware, a communication protocol was developed between the AR app and the RL.

Communication was established through a WebSockets system. User interactions in the AR app triggered the sending of information packages from the client to the server, which processed and applied the changes in the real experiment. The implementation utilized SocketIO, a JavaScript library, and a C# library in Unity to facilitate real-time communication between clients and servers over the web. Figure 2 presents a prototype based on Nicolete’s architecture [19].

Nicolete’s [19] architecture for AR remote labs has certain drawbacks. Firstly, it is limited to AR remote labs, and thus lacks universality. Additionally, the reliance on an external framework and the use of WebSockets introduce potential issues. WebSockets requires a stable network connection, and poor or unreliable network conditions can compromise the UX, leading to delays or disruptions in the RL interaction. Moreover, there are security concerns, particularly when handling sensitive data or conducting experiments that require stringent access controls. Proper security measures should be implemented to address these risks.

Alsaleh et al. [20], in 2022, introduced ReImagine Lab as a framework for integrating hands-on, virtual, and Remote Laboratories into digital twins. The framework consists of three levels: replicating Remote Laboratories as high-frequency synchronized digital twins, periodically synchronized locally hosted virtual laboratories, and leveraging XR technologies for enhanced interaction and visualization in the digital twin representation.

The framework by Alsaleh et al. [20] employs the constantly updated Unreal game engine, but it is essential to recognize that it is a framework, not a comprehensive architecture. Additionally, the lack of AR support reduces its universality, despite integrating XR technologies. Figure 3 illustrates Alsaleh et al.’s [20] schematic diagram for the ReImagine laboratory framework. Concerning RL and Unreal communication, they utilize the UDP communication plugin, which has several drawbacks, including unreliable delivery, no error checking or correction, a higher packet loss rate, potential data duplication, and limited security features.

The characteristics presented in

Table 1. Characteristics and classification of XRLs’ architectures found in the recent literature.

Characteristics	Architectures
	Trentsios et al. [17]	Palmer et al. [18]	Nicolete [19]	Alsaleh et al. [20]
Affordability
Availability
Communication
Deployment
Digital Twin
Full XR
Multilingual
Security
Universality
Up-to-date
Usability

The table highlights the fulfillment of each characteristic using colors: red for missing, yellow for partial fulfillment, and green for complete fulfillment.

were defined based on the XRLs found in the literature. The characteristics are Affordability, Availability, Communication, Deployability, Digital Twin, Full XR, Multilingual, Security, Universality, Up-to-date, and Usability. These characteristics can be translated into challenges, which are addressed by requirements. Table 1 classifies the most recent XRL architectures in the literature based on the defined characteristics.

The challenges found are presented in Table 2, while the requirements are listed as follows:

• Universality: XRLs should be developed with a view to scalability and widespread adoption, so a cross-platform approach is necessary. Therefore, the XRL should be up to date in order to facilitate access.
• Full XR: An XRL architecture should include both AR and VR, as well as technologies such as digital twins.
• Affordability: Cost-effective solutions are needed. Therefore, the issue of deployment should be addressed.
• Multilingual: A multilingual approach improves the XRL availability, accessibility, and possibilities of replication.
• Usability: Usability is vital for effective and engaging learning. Availability should be considered in order to achieve adequate usability.
• Security: XRLs should be secure to ensure the protection of data, users, and system integrity. This includes issues concerning communication.

Table 2. XRL challenges and their descriptions.

XRLs’ Challenges	Description
Up-to-date	XRLs tackle obsolescence by relying on rapidly evolving technologies; adaptability is crucial.
Availability	Ensuring ongoing availability and sustainability of XRLs is indispensable.
Digital Twin	A digital twin enhances interactivity and improves visualization and engagement.
Deployment	Deployment should be considered to facilitate the reuse of architecture for as many XRLs as possible.
Communication	Communication issues include bandwidth, security, and reliability.

Table 2. XRL challenges and their descriptions.

XRLs’ Challenges	Description
Up-to-date	XRLs tackle obsolescence by relying on rapidly evolving technologies; adaptability is crucial.
Availability	Ensuring ongoing availability and sustainability of XRLs is indispensable.
Digital Twin	A digital twin enhances interactivity and improves visualization and engagement.
Deployment	Deployment should be considered to facilitate the reuse of architecture for as many XRLs as possible.
Communication	Communication issues include bandwidth, security, and reliability.

Upon analyzing the architectures, we verified that none of them fulfilled all the requirements or addressed the challenges pertinent to the XRL characteristics.

In terms of universality, only Trentsios et al. [17] and Palmer et al. [18] offered cross-platform support for both desktop and VR devices. Affordability was a common trait among all architectures, as they did not rely on paid software or hardware. However, none provided a complete XR solution, with each focusing on either AR or VR remote labs but not both.

All the architectures employed up-to-date technologies and utilized game engines like Unity or Unreal, reducing the risk of obsolescence. However, none considered multilingual support, limiting the XRL to the developers’ native language.

Regarding usability, Nicolete [19] and Alsaleh et al. [20] validated their architectures with users, though with a limited number of students. Availability wise, most XRLs were either accessible or in active development, except for Nicolete’s [19] XRL, which is no longer being updated.

Regarding the incorporation of digital twins, Trentsios et al. [17] and Palmer et al. [18] focused on this approach, while Nicolete [19] and Alsaleh et al. [20] did not. In terms of deployability, all architectures allowed for relatively easy deployment but required adaptations for a more integrated deployment, given their lack of universality and full XR support.

Concerning communication, Trentsios et al. [17], Nicolete [19], and Alsaleh et al. [20] provided real-time communication, each with its own approach, while Palmer et al. [18] did not offer real-time communication. However, each approach had its own drawbacks, such as security and reliability issues. In Palmer et al.’s [18] case, they used simulated data based on RL-provided data instead of real-time data.

In conclusion, the existing literature lacks an XRL architecture that satisfies the requirements and presents challenges to be solved. In the following section, we will present our novel architecture.

3. Deusto XRL: The XRL Architecture

This section details Deusto XRL, an architecture that we developed based on the issues discussed in Section 2. The following subsections introduce the description of the proposed architecture, and explain how it addresses the requirements and challenges on which it is based. Finally, Section 4 is focused on the validation of the architecture through the development of a prototype and its posterior testing phase in a real scenario.

3.1. Description of the Architecture

This subsection presents a description of the architecture proposed for the design and development of the XRL, denominated Deusto XRL. The structure of this architecture is introduced graphically in Figure 4.

The architecture proposed in Figure 4 does not include specific technologies, as it is expected in a reference architecture. However, when explaining it, we will rely on some technologies to clarify the objectives and tasks of each of the components of the architecture. A more detailed technological description is provided in Section 4.

Deusto XRL was designed in order to be adaptable to as many types of RL as possible. According to our architecture, the process of designing the XRL starts with the collection of data from the physical twin. In the proposed approach, the physical twin includes both the RL that is to be virtualized and converted into a digital twin and the software module necessary to obtain the real data generated during the execution of the real experiment. This software module emulates the behavior of a user when interacting with the Remote Laboratory through the internet. The difference is that this software allows for automatic execution of all possible combinations of the input variables and, through the sensors arranged for this purpose in the RL, collects the response of the experiment and stores it in digital format. As in an RL, this software does not affect the behavior or response of the experiment, which depends only on the configuration of the input variables.

Control of the RL is executed by a Python script, which sends data from the RL to a server through different types of interfaces (such as TCP, USB, and others). This script has been developed in a modular way so that it is interoperable and functional with different types of RL. One must bear in mind that each RL can be controlled using different technologies and protocols. That is why this script has a simple structure that can be adapted to the needs of the RL under study. Once it is executed, the real data generated by the RL are stored in an SQL database (DB), in which a CSV file is generated for each of the possible combinations of laboratory control variable settings. This format has been chosen for its simplicity and its interoperability with multiple programming languages. The CSV files are subsequently processed so that they can be easily consumed by the digital twin modules that need them.

Next, the real data are converted into a data structure understandable by Unity, through the Data Converter module. This consists of inserting the CSV files into the Unity project and associating them with the respective C# scripts that should be called by GameObjects depending on the input sent by the user. Once the real data are adapted, they can be consumed by the different models of the Unity project that make up the XRL, which is composed of two main modules, the XRL model and the interface, both of which are displayed in Figure 4.

The XRL model module has two main components, the 3D model and the Engine Model, which contains data on the physical twin (the RL) execution. The 3D model is a digital copy of the available elements that the RL has. It can be modeled in Unity or any other 3D modeling software.

The 3D model contains all the elements that make it possible to generate/compose a digital model of the physical twin. It is therefore the module in which the developer creates each of these parts, who is able to use different software tools for this, such as the facilities offered by Unity or others such as TinkerCad, Blender, and Maya for this purpose.

As an example, if an RL allows experimentation with the laws of kinematics with an inclined plane, the 3D model will contain at least the inclined plane; the motor that enables it to be positioned; the object that moves along the plane; and the element that makes it possible to release said object so that it slides across the plane.

Thus, this module will receive information from the interface’s Digital Twin Configuration module. This will allow the user, for example, to select the length of the inclined plane, the weight of the object, or the type of surface on which it slides. In fact, these parameters must also be parameterizable in the RL. These parameters must be the same as those used in the Python Script to configure and collect actual data from the physical twin.

Meanwhile, the Engine Model contains data on the physical twin’s behavior. These data are received by the Data Converter module, which, as noted earlier, prepares real data from the physical twin to be replicated by the digital twin. The Engine Model is responsible for combining the information received from the physical twin together with the configurations made by the user in the interface. Thus, the Engine Model commands each of the elements of the 3D model to reproduce the real behavior of the RL in the virtual model.

While the 3D model is a virtual representation of the digital twin, the Engine Model represents its behavior. When a user configures the 3D model through Digital Twin Configuration, a script inserted into the GameObject corresponding to the 3D model will call different actions depending on the input it has received. This script is the result of the process carried out in the aforementioned module, the Data Converter.

This script, through the Digital Twin Control module, will check the physical twin’s responses to the settings and reproduces the behavior in the 3D model, through the Digital Twin Execution module.

The module interface is composed of four main components: Immersion Controls, Digital Twin Configuration, Digital Twin Control, and Digital Twin Execution.

Immersion Controls provides resources with which the user can navigate within the XRL environment. There are three blocks belonging to this component: AR, VR, and navigation. AR and VR refer to configurations added in the project in order to adapt the XRL into an AR or VR experience, as on Unity, it is necessary to install and build the project according to specific configurations in order to make it compatible with certain resources. It also refers to how AR or VR concepts are going to be integrated into the RL experience.

AR functionalities could be related to adding contextualized information during the execution of an experiment, for example, adding information during execution time on the velocity of the fall of an inclined plane experiment, adding data on the voltage of lamps in an electric panel, adding information on the pH of a substance, among other possibilities.

VR functionalities could be related to conversational agents that can interact with the user while engaging in the experience to explain certain concepts or that can allow users to use their hands to interact with an experiment such as a pendulum or circuit board, among others. VR would also be positive in terms of the wait time for the RL based on queues, as students could access it together as a group and even interact with one another, as in a physical science laboratory at a school.

Still in relation to the Immersion Controls module, Navigation refers in general to how the user will interact and navigate within the environment, considering the use of AR/VR gadgets or a web-browser-based environment.

Digital Twin Configuration refers to the data set by the users vis-à-vis how the digital twin should be parametrized, for example, the length of a pendulum, the angle of an inclined plane, or the mass/density of a target that the user wants to submerge to verify Archimedes’ principle in a laboratory. These parameters are the same ones that can be configured in the C# script when collecting the actual data on the physical twin. This process is accomplished by components of Digital Twin Control and Digital Twin Execution.

Digital Twin Control permits the control of the execution of the XRL, for example, if the user has the controls to start/launch the experiment or control the instruments that allow them to take measurements during the experiment. Digital Twin Control is achieved by collecting data from Digital Twin Configuration, calling a C# script relevant to the configuration set. Digital Twin Execution thus consists of the execution of said script on the digital twin, which can be executed in several ways, such as by activating animations, adding game objects to the scene, and changing the object’s masses and positions, among other possibilities, depending on the experiment’s nature.

In summary, this subsection introduces the Deusto XRL architecture, which offers advantages and innovations related to the topics discussed throughout the section. Notably, it includes specific modules designed to enable AR and VR functionalities for XRL. The following subsection will provide a more detailed analysis of how this solution aligns with the specifications introduced earlier.

3.2. Conformity to XRL Characteristics

As regards the conformity of the architecture to the characteristics described in Section 2,

Table 3. Comparison between Deusto XRL and other architectures.

Characteristics	Trentsios et al. [17]	Palmer et al. [18]	Nicolete [19]	Alsaleh et al. [20]	Deusto XRL
Affordability
Availability
Communication
Deployment
Digital twin
Full XR
Multilingual
Security
Universality
Up-to-date
Usability

The table highlights the fulfillment of each characteristic using colors: red for missing, yellow for partial fulfillment, and green for complete fulfillment.

presents a comparison between Deusto XRL and other related architectures found in the literature. As highlighted in the table, our architecture accomplishes all requirements and addresses all challenges pertinent to XRL.

It is universal since it considers a cross-platform approach by providing an interface that embraces responsible interfaces as well as use through other kinds of platforms. It is affordable since does not rely on paid software and hardware. Deusto XRL is Full XR, as it does not exclude AR or VR; it can be adapted to both scenarios or even integrate both of them together. Furthermore, it is up to date, as it relies on constantly updated technologies and platforms, such as the Unity game engine. It also facilitates multilingual resources.

Deusto XRL considers aspects of usability, and was tested and validated with users via a UX questionnaire considering usability, immersion, utility, and satisfaction—this validation will be detailed in Section 4. It also considers availability, by being offered online in order for people to access it; although it is not currently openly online, since it is still in the prototype phase. Students who validated the prototype accessed it through a temporary link.

The architecture we propose is based on the concept of a digital twin, as it sees the RL as a physical twin that communicates with the digital twin within Unity. Therefore, it is in line with the new trends in technology. It has high deployability, as, in principle, it can be adapted to guide the development of any XRL. With regard to communication, it currently provides the best option for XRLs. It provides safer and more reliable data by creating a bridge between the server and the Unity application that does not compromise security or direct access to the server. It is innovative because it is not prone to risks that other approaches may involve while allowing the use of real data in real time.

Following the introduction of the proposed architecture in this section, along with its key advantages and differences compared to similar approaches, Section 4 describes an example of applying this solution to a real-life case.

4. The XRL Pendulum: Implementation and Validation

Firstly, in Section 4.1, the Deusto XRL architecture is technically validated through the implementation of an operational prototype, resulting in the XRL Pendulum. Subsequently, in Section 4.2, the prototype’s UX is analyzed in a real usage scenario.

4.1. Implementation

This subsection provides an overview of the technical validation, or the implementation, of the XRL Pendulum prototype, which is built upon Deusto XRL, the XRL architecture introduced in the previous section. Figure 5 illustrates the interface of the XRL Pendulum.

In terms of its interface, the XRL Pendulum is currently available in Spanish, English, and Portuguese, with the possibility of additional language translations upon request. It offers cross-platform compatibility, allowing access through various devices such as mobile devices, web browsers, or VR devices.

While Deusto XRL encompasses both AR and VR approaches, we opted to develop the prototype using a VR approach for this study, as according to Silva et al. [15], VRLs (Virtual Reality Remote Laboratories) have increased in popularity in the last three years thanks to concepts such as the Metaverse and digital twins. It is important to note that the architecture supports a full Extended Reality (XR) approach, but this study exclusively validates the VR component. Future research may explore the integration of AR or a combined AR/VR platform.

The physical twin chosen for the prototype was a real laboratory developed by BIFI [21] and purchased by the University of Deusto in 2016. It is a simple pendulum, the angle of which can be configured from 10 to 30 degrees, and has three options for mass configuration: standard, with weight attached above (a soft drink can), and with the same weight attached below. Figure 6 presents the physical twin and its three different configurations.

This laboratory was adapted into an RL, which is controlled using a Python script. This script generates a sequence of commands to manage the execution of the RL, including configuring the position from which the pendulum is launched. Within this script, there is an image analysis module that captures the pendulum’s angular position three times per second. Consequently, it is possible accurately to determine the pendulum’s actual position during its execution. This script generates a CSV file for each configuration, comprising two columns; the first one records the elapsed time, while the second one records the corresponding position for each time stamp.

The Data Converter module adapts the CSV into a format that can be read by Unity. To achieve this, the CSV files generated by the server should be inserted into the Unity project, for example, inside a folder. In the case of the XRL Pendulum, as it was a prototype, CSV files were generated with the responses for inputs 10, 20, and 30, which refer to the pendulum’s angle. These CSV files were called by a script, which parsed the CSV file data from string to float and created two arrays of float values: one dedicated to the column reporting the time in seconds, and the other to the column reporting the angle at each second. Then, the array reporting the angles transforms the object’s position/rotation, while the array reporting time ensures that the object is in a certain position at the right time.

The 3D model was designed using the ThinkerCad platform along with Unity and scaled on the basis of the RL in accordance with the three types of configurations available for length, size, and thickness. Considering the three different configurations, three 3D models were actually created, to be viewed according to the settings selected by the user. ThinkerCad was used to create the standard pendulum, while other configurations used the model developed in Thinkercad modified in Unity with prefabs. The 3D model for the standard pendulum consists of three elements: a pin, called the base; a rod; and an added weight. The added weight is held by the rod, while the rod is held by the base. In the Unity hierarchy, the rod and the added weight are children of the base. In the case of the pendulum with the weight added above, there is an extra weight above the added weight, which is a three-dimensional model of a can. In the case of the pendulum with the weight added below, the extra weight is the same as the can model, but positioned in front of the added weight. Figure 7 presents the developed 3D models.

The Engine Model is reflected by the scripts associated with each 3D model configuration. It reads the data converted from the CSV file and allows the 3D model to behave in the same way as the physical twin, therefore configuring it as a digital twin within the concept of an XRL, which requires real data.

The interface was designed to resemble the RL environment. For instance, the RL environment included a knob controller that allows users to visualize the angle. Another similarity to the RL environment is that the platform also presents the RL environment in a smaller wing, allowing users to observe it alongside the digital twin. Additionally, the prototype’s interface includes a timer to help users observe the pendulum’s movements. While the prototype’s interface shares similarities with the RL environment, it also offers additional features. Users can input the desired angle manually and reset the application as needed. Furthermore, the 3D model provides a clearer visualization of the experiment.

Control of the digital twin is facilitated through buttons in the interface. Users enter the desired angle in an input field, and when they press ‘start’, this value is sent to a script. For example, if the selected configuration is the standard pendulum, the 3D model is associated with the ‘STControl’ script. This script reads the input and selects a behavior based on the chosen angle. The 3D model then reflects the corresponding behavior. Users also have the option to reset the experiment, restoring all values to their original positions.

To manage the timer, users can interact with it using three buttons: reset, play, and pause. ‘Play’ initiates the timer, while ‘pause’ stops it and ‘reset’ resets it to zero. Additionally, a button allows users to return to the main menu, where they can select a different pendulum configuration, including attached weights.

After the user configures the digital twin, the latter not only executes the corresponding animation regarding the data specified in the CSV, but it also plays a video showing the physical twin’s execution. Therefore, the users can compare both elements, since they are not restricted to the 3D representation alone.

Regarding immersion controls, the platform allows the user to zoom into or zoom out of the experiment by mouse scrolling, thus facilitating the visualization of movements—something that would not be possible in a conventional RL environment.

This section has provided a detailed description of the implementation of the XRL Pendulum. The following section will explore its validation process.

4.2. Validation

This subsection presents the UX validation of the XRL Pendulum and it is divided into two topics: a case study and results from a questionnaire.

The validation was conducted in order to answer the hypothesis: “The user eXperience (UX) in the XRL Pendulum is better than the UX when using the RL Pendulum”.

4.2.1. Case Study

The case study involved 150 high-school students from Engineer Sebastião Toledo dos Santos High School, a public school in Criciúma, Santa Catarina, Brazil.

The students, aged 14 to 17, shared similar socio-economic backgrounds and were enrolled in the first year of high school. They also all had the same physics teacher, whom we contacted and who agreed to participate in the research. This specific teacher was contacted because in the past she had participated in other research with us, and was therefore familiar with the activities to be conducted.

The case study was conducted in five steps.

First, we contacted the teacher and proposed that she and the school where she works participate in the research. Then, we had a meeting to decide which students could participate in the study, and she chose all her first-year high school students, namely 150 boys and girls aged 14 to 17 from six physics classes.

The second step was to send terms of agreement for the students and their parents to sign, allowing them to participate in the research. The research was approved by the Research Ethics Committee of the University of Deusto, which provided the terms of agreements signed by the students and their parents.

During the third step, the teacher conducted a 50-minute lesson about pendulums over six classes over the course of one week. The lesson was given without the aid of the internet, simulations, remote labs, etc., and was conducted in a traditional classroom.

In the fourth step, students were divided into two groups: Group RL (75 students, or three classes) used a traditional RL, while Group XRL (75 students, or 3 classes) utilized our prototype. Although we have the information that, of 150 students, 78 were girls and 72 were boys, we do not know the number of boys and girls assigned to each group, as data protection would not allow us to know their gender.

The students were unaware of the type of laboratory to which they were assigned, as both were referred to as “laboratory”. This step took place one or two weeks after step three, as it depended on the availability of the school’s computer laboratory, where students would use the prototype.

Figure 8 shows students using the XRL Pendulum.

The fifth step occurred on the same day as the fourth step, with students completing online UX questionnaires immediately after performing their assigned experiments.

On the day they used the laboratory, the students had a 90-minute lesson divided as follows: for the first 30 min, the teacher reviewed what they learned during the lesson about pendulums; then, for 30 min, they used the laboratory; and finally, during the last 30 min, they answered the questionnaire.

The following section will detail the questionnaire and its results.

4.2.2. Questionnaire Results

The validation process for the XRL Pendulum was conducted based on UXQ4RL version 1.0 (User Experience Questionnaire for Remote Labs), as detailed in [22]. UXQ4RL 1.0 is a seven-point Likert questionnaire designed to assess UX in remote labs. It comprises nine questions categorized into three scales, immersion, usability, and utility, with three questions dedicated to each scale. The original questionnaire on which UXQ4RL 1.0 was based encompassed four scales, including satisfaction. Initially, this questionnaire consisted of 16 questions, but it was eventually replaced by UXQ4RL 1.0 after thorough analysis and validation.

Based on UXQ4RL 1.0, UXQ4OL version 1.0 (User eXperience Questionnaire for Online Labs) was developed. This questionnaire also employs a seven-point Likert scale and maintains the same three scales as the original version. However, there are three key differences to note. First, the UX encompasses both RL and XRL, making it more appropriate to refer to it as Online Labs (OLs) to reflect this broader scope. Moreover, UXQ4OL 1.0 incorporates two additional questions within each of the three scales, allowing for a more detailed analysis of the UX with each type of laboratory. In addition, the former questionnaire is available in English and Spanish, while UXQ4OL is available in English and Portuguese, as it was applied to Brazilian users. The complete questionnaire can be found in the Appendix A labeled “User eXperience Questionnaire for Online Laboratories (UXQ4OL) v. 1.0”.

Table 4. Comparison between UXQ4RL 1.0 and UXQ4OL 1.0 questionnaires.

Questionnaire	UXQ4RL 1.0	UXQ40L 1.0
Scale	Questions
Usability	Q00: The remote lab is easy to use.	Q00: The lab is easy to use.
	Q05: I can predict the result of using each element in the remote lab interface.	Q05: I can predict the result of using each element in the online lab interface.
	Q06: The remote lab layout is sufficiently apparent so that help is not needed to use it.	Q06: The online lab layout is sufficiently apparent so that help is not needed to use it.
		Q11: The look of the online lab makes it easy to use.
		Q13: The dynamics of the online lab make the interaction easy.
Utility	Q03: The remote lab helps me learn.	Q03: The online lab helps me learn.
	Q07: The remote lab meets my requirements.	Q07: The online lab meets my requirements.
	Q08: The remote lab will help me pass the course.	Q08: The online lab will help me pass the course.
		Q10: The look of the online lab helps its comprehension.
		Q12: The online lab flexibility helps me learn.
Immersion	Q01: Using the remote lab feels like using a real lab.	Q01: Using the online lab feels like using a real lab.
	Q02: My interactions with the remote lab seem real.	Q02: My interactions with the online lab seem real.
	Q04: When working on the remote lab I concentrate on the assigned tasks.	Q04: When working on the online lab I concentrate on the assigned tasks.
		Q09: The look of the online lab seems as it is real.
		Q14: The dynamics observed in the online lab seem real.

presents a comparative overview of the questionnaires. The first column presents each scale and the second column displays the questions from the original questionnaire in [22], while the third column presents the questions included in UXQ4OL 1.0. The numbers indicate the position of each question in each respective questionnaire.

Compared with the first version of the questionnaire [22], the present questionnaire demonstrates significantly improved reliability. This enhancement indicates a notable refinement of the questionnaire’s measurement properties, with the newly introduced questions effectively aligning with their respective scales.

The examination of Cronbach’s alpha coefficient within the questionnaire reveals a high level of internal consistency, with an overall reliability coefficient of 0.94, indicating the instrument’s exceptional capacity to consistently measure usability, immersion, and utility in online laboratory settings. It has a higher coefficient than UXQ4RL, which had an overall reliability coefficient of 0.83. The reliability analysis conducted on each one of the scales has revealed varying degrees of internal consistency within the questionnaires used to assess different constructs. Usability demonstrated a moderate internal consistency (0.77), suggesting some potential for improvement in reliability. However, it had a higher coefficient in comparison with UXQ4RL, which had a coefficient of 0.70 regarding the usability scale. Conversely, immersion and utility exhibited good to excellent levels of internal consistency, reaching scores of 0.85 and 0.89, respectively, affirming the robustness of their respective questionnaires, which was higher than for UXQ4RL, which attained coefficients of 0.73 and 0.81, respectively, for the same scales. In relation to the results for UXQ4OL, usability may benefit from further refinement to enhance its reliability, while the results for immersion and utility can be regarded as highly reliable instruments for measuring their targeted constructs.

A Confirmatory Factor Analysis (CFA) is a method used to check whether questionnaires or research tools effectively measure the constructs intended for assessment, analyzing the different scales defined. For UXQ4OL, the CFA results indicate that the measurement model fits the data well. Specifically, the Comparative Fit Index (CFI) and the Tucker–Lewis Index (TLI) are both higher than the standard threshold of 0.90, with values of 0.929 and 0.914, respectively. The Root Mean Square Error of Approximation (RMSEA) was above the ideal threshold at 0.086 but had an upper confidence interval (CI) that approached 0.08 (CI: 0.068 to 0.104), indicating an acceptable fit. In summary, the overall evidence from multiple fit indices suggests that the model fits the data reasonably well.

After evaluating the consistency and reliability of the data, we tested the hypothesis about the effect of including virtual elements on UX. The results of a Wilcoxon rank-sum test (W = 42,829, p < 0.001) indicate that the UXs in both labs are different, with the users of the XRL Pendulum having a more positive experience than users of the RL. Non-parametric statistical techniques were chosen to avoid assuming an interval level of measurement for the UXQ4OL scales.

In the exploratory analysis of the responses to the UXQ, Figure 9 shows that the UX of the groups that used the XRL is higher than the UX of the group working with the RL.

A comparison of the answers between students from the RL group and Xhe RL group is presented in Figure 10. As shown in the figure, in general, the XRL Pendulum was better accepted by the students, generating better UX scores. Question 3, from the group of questions regarding immersion, i.e., “My interactions with the online lab seem real”, was the one with the biggest impact, where students that used the XRL Pendulum agreed more than students that used the RL. Question 1 about usability, i.e., “The lab is easy to use”, showed the smallest difference between the groups. It can therefore be inferred that the XRL Pendulum provides a more immersive experience, but both options are easy to use.

A more detailed analysis of the three different scales regarding the comparison of answers between the RL and XRL groups, is presented in Figure 11, Figure 12 and Figure 13. In all questions, the XRL Pendulum generated better feedback than the RL, although some groups of questions had higher results than others.

As presented in Figure 11, questions related to group usability revealed a smaller difference. Regarding the seven-point scale, the average answer that students from the RL group gave to questions from this group was 4.82, while the answers of students from the XRL group had an average of 5.56. Therefore, usability was the least significant factor in relation to the improvements brought by the XRL Pendulum.

Figure 12 presents the comparison of answers for each group regarding immersion. The RL had the weakest immersion feedback, achieving an average of 3.91, while the XRL Pendulum attained an average of 4.82 points. This proves that the XRL Pendulum provides a more immersive experience compared to a conventional RL, which is positive, as Ma and Nickerson [7] state that the feeling of immersion is fundamental in order to take full advantage of the remote experiment.

Figure 13 presents a comparison of answers for each group regarding utility, which had the better performance when compared to the other groups of questions. The average answer that students gave concerning the RL’s utility was 4.34, while for the XRL Pendulum, it was 5.77. Thus, students regard the XRL Pendulum as more useful than the RL.

To summarize, after the process of evaluating the XRL Pendulum’s UX, it is possible to draw conclusions regarding both the consistency and reliability of the questionnaire developed, as well as concerning the UX itself.

Regarding the questionnaire UXQ4OL, the validation process showcased internal consistency and reliability across the three scales. The analysis demonstrated that while usability exhibited potential for refinement, immersion and utility emerged as highly reliable measures of their respective constructs. Furthermore, the CFA results provided substantial evidence of a reasonable model fit, reinforcing the validity of the measurement model for latent constructs.

As highlighted earlier, the motivation behind the development of XRLs stemmed from the recognized deficiency in user immersion in RLs, as noted by Nickerson et al. [23]. The Wilcoxon test and a detailed analysis of participant responses consistently favored the XRL Pendulum, revealing that it not only offers a more immersive experience but also significantly enhances utility compared to conventional RLs. This research underscores the potential of XRLs for improving UX in online laboratory settings and suggests a shift towards their adoption for enhanced educational outcomes.

5. Conclusions

In this paper, we have presented the design, implementation, and evaluation of an XRL reference architecture. Our work began with the identification of essential requirements and challenges for an innovative XRL architecture, resulting in a comprehensive XRL architecture called Deusto XRL, exemplified by the XRL Pendulum prototype. To validate our approach, we employed a questionnaire covering aspects of usability, immersion, and utility.

We identified ten crucial characteristics for XRLs, with five serving as requirements (universality, affordability, multilingual support, usability, and security) and the remaining five as challenges (up-to-date data, availability, digital twin integration, deployment, and communication). Our architecture encompasses all these characteristics, with the current XRL prototype fulfilling all but one—full XR—which it partially achieves by offering a VR experience.

Deusto XRL offers accessibility through standard web browsers, enabling interactions via computers, mobile devices, or AR/VR devices. The interface is built on an integration of Unity and WebGL, providing users with access to a digital twin and enabling command input. Data from user interactions are relayed to the Remote Laboratory, and the resulting real-world data are logged and exported to a CSV file. These data are then processed in C# to be rendered in Unity, accurately replicating the behavior of the Remote Laboratory in the digital twin.

For validation, we chose a pendulum RL for the XRL Pendulum prototype and conducted a study with 150 high-school students, where 75 interacted with a conventional RL and 75 with the XRL Pendulum. The validation process was supported by the “User eXperience Questionnaire for Online Laboratories (UXQ4OL) v. 2.0”, developed specifically for this research. Our results confirmed the questionnaire’s high reliability and consistency. Moreover, the XRL Pendulum demonstrated superior performance by providing a more immersive experience and enhancing utility compared to conventional RLs.

A limitation of Deusto XRL is that it was not validated in terms of the use of AR, integration with AR/VR gadgets, or learning aspects.

The implementation of new remote experiments that integrate VR (Virtual Reality) and AR (Augmented Reality) will showcase the quality of the architecture or its limitations in real-life scenarios, which may require adaptation of the XRL architecture to new interfaces and devices, such as haptic devices, sound devices, and other sensors and actuators, in line with references [24,25].

Additionally, an important point is to test the new architecture through labs implemented by other designers. In addition, as this paper presents the first implementation of the architecture with a specific RL, at this moment there are no other limitations, but as the architecture is implemented with more RLs, it is probable that more limitations will be revealed.

Furthermore, the technical validation presented in this work is of a functional nature. In future work, it will be necessary to analyze the computational aspects in more detail, such as latency, computational resource consumption, bandwidth, etc. This way, the architecture would not only be defined by its purpose, but also by its computational cost.

In future research, we aim to validate the performance of XRL Pendulum regarding student’s learning outcomes to test it as a learning tool. A pretest/post-test scenario is being developed at this time as a part of future work.

Another avenue for future work involves further enhancing usability, as our findings indicate that while the XRL Pendulum excels in immersion and utility, there is room for improvement in usability to ensure a well-rounded UX.