The Inﬂuence of Immersive and Collaborative Virtual Environments in Improving Spatial Skills

: The use of Virtual Reality (VR) is growing in popularity in educational contexts. In this work, we present a novel software application based on immersive VR in which users can interact simultaneously with a model in a shared virtual scene and maintain audiovisual communication. The 3D model-building activity within the application was designed to improve spatial skills. Twenty-nine industrial engineering students completed the modeling activity in groups of two or three. Post-test scores showed signiﬁcant improvement over pre-test scores for both spatial tests used to measure the effectiveness of the instrument in improving spatial skills. Participants rated the application favorably in terms of usability and functionality. Results suggest that spatial training in a collaborative immersive virtual environment has the potential to be an effective approach to improving spatial skills.


Introduction
Virtual Reality (VR) has grown in popularity as a learning tool over the past decade, including among academics and educators [1]. The improved user experience in terms of interaction, immersion, and quality of information [2], together with the decreasing cost of the technology, have propelled this expansion. Many others have highlighted the advantages of using VR for education, including learner satisfaction [3], improved motivation and problem-solving abilities [4,5], and the advantages of sensory-based learning [6][7][8]. Studies of using VR-based platforms for the acquisition of knowledge are now found across a wide range of disciplines, including medicine [9], rehabilitation [10], industrial training [11], evacuation plans [12], urban planning [13], journalism [14], sports [15], geometry [16], design [17], biology [18], physics [19], and engineering [20].
In the past decade, many researchers have compared students' learning in VR with learning through other mediums. Some examples include the work of Nadan et al. [21], [22], Slavova and Mu [23], and Zakaria et al. [24]. One area of education that is particularly interesting for VR applications is collaborative learning. Collaborative learning is a pedagogical method in which individuals interact in order to solve problems together [25]. Gokhale [26] is considered to have introduced the concept of collaborative learning in university-level teaching and outlined the advantages of collaborative learning over individual learning.
Virtual collaborative environments are systems that connect multiple users who interact in real-time in a virtual space via an internet connection and can serve as a platform for collaborative learning. Virtual collaborative environments that allow specialists from different disciplines to solve complex problems together are promising venues for research and for the improvement of work and productivity [27]. Activities taking place in virtual collaborative environments may be valuable for enhancing STEM education. Saorín et al. [28] Appl. Sci. 2023, 13, 8426 2 of 17 concluded that it is possible to meet the future engineering industry demands through training programs involving collaborative 3D modeling environments. Furthermore, according to Dominguez et al. [29], VR can be a powerful tool for training spatial abilities. While virtual collaborative environments are promising for STEM education, applications utilizing this technology have not yet been made in the area of spatial skills training. Spatial skills training is a popular area of study in STEM education because researchers believe that improving spatial skills can help students to succeed in STEM [30]. Thus, it would be valuable to explore the application of collaborative virtual environments to spatial training.
In the present work, we add to the research on virtual collaborative environments for educational settings by describing the design and validation of an immersive VR application used for a collaborative exercise for the construction of 3D models. We tested the application with 29 students with the goal of finding out whether the students' spatial skills were improved through using the application, and whether the students found the application to have good usability and functionality. The main contribution of this work is to present evidence that performing a group modeling exercise in a virtual collaborative environment can improve performance on spatial tests. This document is structured such that, following a review of the most relevant literature, we describe the materials and methods of this study, present the results, and finally discuss the results and provide our conclusions.

Spatial Skills and STEM Education
Spatial ability is generally considered to be the ability to visualize and mentally manipulate relationships between objects and space. Spatial tests were originally used in the study of psychology to predict which individuals would be successful in mechanical vocations [31]. Today, spatial testing and spatial training remain very popular as many researchers believe that improving spatial skills can help students to succeed in STEM [30]. A meta-analysis of training studies concluded that including spatial training in education could lead to increased participation in STEM [32]. While spatial training often leads to improved performance on spatial tests, the evidence that spatial training enhances longterm STEM achievement remains outstanding. A review concluded that spatial training alone is unlikely to raise STEM scores substantially, and rigorous studies are still needed in order to establish a causal effect between spatial training and STEM outcomes [33].
The most common way to assess spatial ability is by using psychometric tests, which are considered to measure "fundamental" spatial skills [34,35]. Two spatial tests are used in this study and are discussed in more detail in Section 3.2 of this paper. Existing spatial tests have many limitations. Multiple researchers have pointed out that spatial skill is not a singular skill [31,36]; thus, no single test can be considered to measure spatial skill on its own. Spatial ability is really an umbrella term that describes many different possibly related but distinct skills, and different tests may assess different specific spatial skills.
Spatial tests typically portray 3D shapes as 2D black and white line drawings, and the way some of the shapes are drawn may lead to confusion for test takers and introduce additional difficulty into spatial tests that is not related to the 3D mental manipulation of shapes [37]. For this reason, Bartlett and Camba advocated for using more advanced visualization technologies to present spatial problems [38]. This recommendation is relevant not only for spatial testing, but also for spatial training applications such as the one described in this paper.

VR and Spatial Skills Training
Virtual reality is an increasingly popular medium for research on training spatial ability [39]. In addition to the visual problem of presenting a 3D task using 2D media, spatial tasks may face issues of engagement and motivation. Using media like VR to present spatial training tasks may help with student motivation, as traditional paper and pencil tasks may be unappealing to students, who might find them boring or unmotivating [40].
In recent years, numerous researchers have been exploring new media like VR and augmented reality (AR) for spatial training tasks. One approach is to train mental rotation skills by having users rotate shapes in a virtual environment [41][42][43]. In another example, visual cues were added to the virtual environment with the goal of training mental rotation skills [44]. In addition to training mental rotation skills specifically, others have attempted to train spatial skills by having students view their own 3D models in VR [45], training the spatial skill of perspective-taking [46], and training general spatial ability [47]. Rather than immersive VR, other researchers have explored using AR to train spatial ability [48,49].
Virtual Reality-based spatial training programs have been demonstrated to be effective, as shown in the following examples. In one study, researchers used exercises in an AR book in an attempt to train spatial relations and spatial visualization skills and used a VR orienteering game to train the skill of spatial orientation [40]. This study used a control group who did not complete the training and found that the gains in performance on spatial tests at the end of the study were greater for the experimental group [40]. In another study, researchers developed a novel VR spatial training application in which students manipulated shapes in a virtual environment by moving, rotating, and scaling the shapes. To validate the tool, the researchers conducted an experiment in which one group of students used a VR headset to participate in the activity, and another group performed the same activity on a computer screen. Spatial test scores improved more for the virtual reality group than for the computer screen group [50]. These studies suggest that VR spatial training is effective, and potentially more so than computer-based spatial training media.
Following the study published by Bekele and Campion [51], it is important to differentiate between VR observed on a screen and immersive VR [52], in which the user is completely visually immersed in virtual space. According to Loeffer and Anderson [53], immersive VR allows the user to interact in a 3D computer-generated environment, and immersive VR is what has drawn the most interest of educators and educational researchers [54,55]. Oh and Nussli [56] argued that the potential of this technology for the classroom seems limitless. The works presented by Witmer et al. [57], Bliss et al. [58], and Koh et al. [59] conclude that immersive virtual environments are as effective as real-world environments for training spatial ability. Levels of immersion may be especially relevant to training spatial skills, as a study found that different levels of complexity of spatial learning may be better learned at different levels of immersion [60].
Other researchers have assessed VR-based spatial training instruments on the basis of usability and enjoyability. One study found that the right combination of levels of immersion and controls could lead to the most positive experience for the user in a VRbased mental rotation training task [43]. Another study assessed a VR-based training application for mental rotation. In this case, the researchers compared conditions in which the users could actively manipulate the objects versus passively observing the motion. Users rated the active condition higher rated for playfulness, ease of use, and use-intention score [39]. Thus, the design of spatial training applications is important not only for user comfort and enjoyment, but potentially also for motivation and training effectiveness.

Multi-User Collaborative Applications
In our review of the literature, the earliest platform we found for the development of multi-user VR applications was described by Christer [61]. The DIVE platform permits a high number of users to participate and interact in the same virtual environment. Since then, the development of scalable architectures that can be adapted to virtual environments has been a great challenge in spite of the advances in computing [62]. However, such applications are now being developed. For example, Rai et al. [63] presented a prototype of a networked virtual environment that centered on the combination of a Minority Game [64] and an Online Induction Algorithm with lossless data transfer functions. Furthermore, many popular games on consumer VR headsets, such as the Meta Quest 2, allow users to play together remotely over the internet with their avatars inhabiting the same virtual Beyond entertainment, virtual collaborative environments have the potential to enhance skills training and productivity [27]. User experience is also highly important in virtual collaborative environments [65]. Although other researchers have created spatial skills training applications using immersive VR, as described in Section 2.2, we were not able to find any examples of such applications which leveraged a multi-user collaborative environment for this purpose. Thus, we consider our application to be novel in the sense that it leverages a virtual collaborative environment for a spatial skills training activity.

Materials and Methods
Our investigation centers around an experimental analysis that compares students' scores on spatial tests before and after performing an activity in which they construct geometric solids in a virtual immersive collaborative environment. We worked with an initial group of 41 student volunteers from a first-year industrial engineering course. As these students had completed technical coursework in high school, they were expected to have acquired some skills in spatial visualization through their past training in technical drawing.
The students' baseline spatial skills were measured with two well-known tests, which were administered online on the students' own time. Twelve students were removed from the study at this point since they did not complete the tests. The twenty-nine remaining students were divided randomly into nine groups of three students, plus one group of two students, for the completion of the collaborative virtual activity. Twelve of the twenty-nine participants were women, and seventeen were men.
The virtual activity was carried out with a software application we developed specifically for this study, which centered on the creation of a geometric model constructed from primitive solids. The students participated simultaneously with their group members in the virtual space and interacted together to build the model. Since this was a new software application and the students did not know how to use it, before each group of students began the modeling process, they first built a model in a shared virtual space with an instructor. The instructor helped the students to understand the functionality of the software and the modeling method.
At the end of the modeling activity, the students again took the two spatial tests under the same conditions as they had conducted previously. By comparing their performance on the pre-and post-tests, we draw some conclusions about the activity's ability to improve spatial skills. In order to assess the design of the novel software we created for this study, the students also completed two questionnaires about the usability and functionality of the software after they finished the modeling activity.

Virtual Reality Experience
The application used for the virtual experience was built on a HP laptop running 64-bit Windows with an Intel ® Core™ i7-8750H processor running at 2 GHz with 8GB RAM, and an Nvidia GeForce GTX 1080 Ti graphics card. The devices used for the VR experience were HP Reverb head-mounted displays with a screen resolution of 2160 × 2160 and two handheld controllers for user interaction.

Software
The virtual experience we created for this study is a collaborative VR application based on Unity. The central activity within the experience is the construction of a model from geometric solids. The virtual environment is shown in Figure 1.
were HP Reverb head-mounted displays with a screen resolution of 2160 × 2160 and two handheld controllers for user interaction.

Software
The virtual experience we created for this study is a collaborative VR application based on Unity. The central activity within the experience is the construction of a model from geometric solids. The virtual environment is shown in Figure 1.  In addition to the creation of primitive shapes, the menu also allows the user to carry out two other operations: delete a previously created primitive shape and select the direction in which the shapes can be scaled.
The remaining menu options allow for the configuration of the environment. The "open" button allows an image file to be uploaded and displayed on the screen. The buttons "up" and "down" adjust the camera view within the environment to be better suited for the user's height. The "freeze menu" button allows the user to anchor the menu in one spot, since, by default, the menu moves with the user's point of view. were HP Reverb head-mounted displays with a screen resolution of 2160 × 2160 and two handheld controllers for user interaction.

Software
The virtual experience we created for this study is a collaborative VR application based on Unity. The central activity within the experience is the construction of a model from geometric solids. The virtual environment is shown in Figure 1.  In addition to the creation of primitive shapes, the menu also allows the user to carry out two other operations: delete a previously created primitive shape and select the direction in which the shapes can be scaled.
The remaining menu options allow for the configuration of the environment. The "open" button allows an image file to be uploaded and displayed on the screen. The buttons "up" and "down" adjust the camera view within the environment to be better suited for the user's height. The "freeze menu" button allows the user to anchor the menu in one spot, since, by default, the menu moves with the user's point of view. In addition to the creation of primitive shapes, the menu also allows the user to carry out two other operations: delete a previously created primitive shape and select the direction in which the shapes can be scaled.
The remaining menu options allow for the configuration of the environment. The "open" button allows an image file to be uploaded and displayed on the screen. The buttons "up" and "down" adjust the camera view within the environment to be better suited for the user's height. The "freeze menu" button allows the user to anchor the menu in one spot, since, by default, the menu moves with the user's point of view.
The created primitives can be displaced, rotated, and scaled by the user through natural gestures performed on the controllers. Figure 3 shows the operation in which the user grabs hold of the primitive with a controller by positioning the curse of a controller over the primitive and pressing the trigger button, then displaces or moves the shape with a simple movement of the controller. Releasing the trigger button releases the primitive.
for the user's height. The "freeze menu" button allows the user to anchor the menu in one spot, since, by default, the menu moves with the user's point of view.
The created primitives can be displaced, rotated, and scaled by the user through natural gestures performed on the controllers. Figure 3 shows the operation in which the user grabs hold of the primitive with a controller by positioning the curse of a controller over the primitive and pressing the trigger button, then displaces or moves the shape with a simple movement of the controller. Releasing the trigger button releases the primitive. In order to facilitate the processes of aligning the primitives to construct the larger model, when the face of one primitive is positioned at a close, experimentally determined distance to the face of another primitive that already exists in the model, it will be positioned automatically such that both primitives are perfectly aligned. In a similar manner, to perform the rotation of a primitive, the user can capture that primitive and turn the controller in the desired direction. The primitive continues the rotation that the user performed with the controller until the trigger button is released (see Figure 4). In order to facilitate the processes of aligning the primitives to construct the larger model, when the face of one primitive is positioned at a close, experimentally determined distance to the face of another primitive that already exists in the model, it will be positioned automatically such that both primitives are perfectly aligned. In a similar manner, to perform the rotation of a primitive, the user can capture that primitive and turn the controller in the desired direction. The primitive continues the rotation that the user performed with the controller until the trigger button is released (see Figure 4).  In the same way as in the displacement, to facilitate the precise alignment of primitives, at the end of a rotation movement, if one of the rotations of the primitives differs in the X, Y, or Z rotation by a small, experimentally determined angle from the position of another primitive with said axis, the rotated primitive will automatically reorient so that both primitives are aligned.
The operation of scaling the primitive is initiated by capturing a primitive with both controllers. Once the primitive is captured by both controllers, the scaling up or scaling down of the primitive is controlled by the change in the distance between the two controllers (see Figure 5). In the same way as in the displacement, to facilitate the precise alignment of primitives, at the end of a rotation movement, if one of the rotations of the primitives differs in the X, Y, or Z rotation by a small, experimentally determined angle from the position of another primitive with said axis, the rotated primitive will automatically reorient so that both primitives are aligned.
The operation of scaling the primitive is initiated by capturing a primitive with both controllers. Once the primitive is captured by both controllers, the scaling up or scaling down of the primitive is controlled by the change in the distance between the two controllers (see Figure 5). the X, Y, or Z rotation by a small, experimentally determined angle from the position of another primitive with said axis, the rotated primitive will automatically reorient so that both primitives are aligned.
The operation of scaling the primitive is initiated by capturing a primitive with both controllers. Once the primitive is captured by both controllers, the scaling up or scaling down of the primitive is controlled by the change in the distance between the two controllers (see Figure 5). Just as in the other operations, to facilitate the construction of the model, once the scaling of one primitive is finished, the driving dimension is rounded to the nearest integer, so that the final sizes of each primitive can easily be aligned. The scaling factor will Just as in the other operations, to facilitate the construction of the model, once the scaling of one primitive is finished, the driving dimension is rounded to the nearest integer, so that the final sizes of each primitive can easily be aligned. The scaling factor will be applied to the X, Y, or Z axis depending on which button, "X-axis", "Y-axis", or "Z-axis", was selected from the menu.

Collaborative Environment
The software was designed to be used in a collaborative manner, meaning that multiple users can simultaneously be present in the same virtual environment and participate in the construction of the same model together. Thus, the application allows for the coordination of the actions performed by different users. The users can manipulate any of the primitives at any time when the primitive is not being held by another user. If another user is holding the primitive, the primitive appears transparent to the control cursor of the rest of the users until the user who is holding the primitive lets go.
The users who are in the same virtual environment are visible to the rest of the users in the form of simplified avatars (see Figure 6). The users' names hover above their avatars' heads. During the entire activity, the users can communicate verbally.
Finally, given that a connection problem may occur since the users are connected remotely, the menu shown in Figure 2 initially shows a "without connection" message which is normally hidden but can be shown in the event of a connection problem.

Modeling Activity
The collaborative activity during the VR experience consisted of modeling six different models constructed of primitive solids. The models are shown in Figure 7. Each individual member of the collaborating group was physically occupying a different office while wearing the VR headset, and they were connected to the same virtual environment through a local network. Participants were allotted two-hour morning sessions separated by 20 min breaks. Although there was no time limit to complete the modeling exercises, the majority of the groups completed them in two sessions. Only three of the ten groups required a third evening session. Appl  Finally, given that a connection problem may occur since the users are connected remotely, the menu shown in Figure 2 initially shows a "without connection" message which is normally hidden but can be shown in the event of a connection problem.

Modeling Activity
The collaborative activity during the VR experience consisted of modeling six different models constructed of primitive solids. The models are shown in Figure 7. Each individual member of the collaborating group was physically occupying a different office while wearing the VR headset, and they were connected to the same virtual environment through a local network. Participants were allotted two-hour morning sessions separated by 20 min breaks. Although there was no time limit to complete the modeling exercises, the majority of the groups completed them in two sessions. Only three of the ten groups required a third evening session.   Finally, given that a connection problem may occur since the users are connected remotely, the menu shown in Figure 2 initially shows a "without connection" message which is normally hidden but can be shown in the event of a connection problem.

Modeling Activity
The collaborative activity during the VR experience consisted of modeling six different models constructed of primitive solids. The models are shown in Figure 7. Each individual member of the collaborating group was physically occupying a different office while wearing the VR headset, and they were connected to the same virtual environment through a local network. Participants were allotted two-hour morning sessions separated by 20 min breaks. Although there was no time limit to complete the modeling exercises, the majority of the groups completed them in two sessions. Only three of the ten groups required a third evening session.  Given that none of the students had previous experience using VR, before beginning the activity, they performed a practice exercise modeling the shape shown on the left of Figure 7. This exercise was led by an instructor, who explained to the students the functionality of the software application and how to complete the modeling process. The time it took to model this example with the instructor varied among the groups depending on the questions asked by the students, but it generally lasted 40 to 60 min.

Tests of Spatial Visualization
To assess the students' spatial ability before and after the virtual experience, we used the Mental Rotation Test (MRT) proposed by Vandenberg and Kuse [66] and the Differential Aptitude Test: Spatial Relations (DAT:SR) [67]. The MRT is a test that measures an individual's ability to mentally rotate 3D objects. The test consists of a total of 20 questions divided into two sections of 10 questions each. For each question, one must select two correct answers out of a total of four options. The two correct answers represent different rotated views of the shape in question. Test-takers are allowed six minutes to complete the test, three for each section, with a pause in between. The MRT was scored out of a total of 40 points, one point for each correctly identified shape. An example question from the MRT is shown in Figure 8.

Tests of Spatial Visualization
To assess the students' spatial ability before and after the virtual experience, we used the Mental Rotation Test (MRT) proposed by Vandenberg and Kuse [66] and the Differential Aptitude Test: Spatial Relations (DAT:SR) [67]. The MRT is a test that measures an individual's ability to mentally rotate 3D objects. The test consists of a total of 20 questions divided into two sections of 10 questions each. For each question, one must select two correct answers out of a total of four options. The two correct answers represent different rotated views of the shape in question. Test-takers are allowed six minutes to complete the test, three for each section, with a pause in between. The MRT was scored out of a total of 40 points, one point for each correctly identified shape. An example question from the MRT is shown in Figure 8. Rather than mental rotation, the DAT:SR is considered to assess the skills of spatial visualization and spatial relations [35]. The DAT:SR consists of 50 questions. The left side shows a flattened pattern with some shaded parts which represents the transformed geometry of one of the four 3D shapes shown on the right. The test taker must select the shape from the right that corresponds to the flattened pattern. Twenty minutes are allotted to complete the test, and the test is scored out of 50 points, with one point for each correct answer. An example question from the DAT:SR is shown in Figure 9. To facilitate the administration and scoring of the tests, the tests were administered digitally using Microsoft Forms. Rather than mental rotation, the DAT:SR is considered to assess the skills of spatial visualization and spatial relations [35]. The DAT:SR consists of 50 questions. The left side shows a flattened pattern with some shaded parts which represents the transformed geometry of one of the four 3D shapes shown on the right. The test taker must select the shape from the right that corresponds to the flattened pattern. Twenty minutes are allotted to complete the test, and the test is scored out of 50 points, with one point for each correct answer. An example question from the DAT:SR is shown in Figure 9.

Tests of Spatial Visualization
To assess the students' spatial ability before and after the virtual experience, we used the Mental Rotation Test (MRT) proposed by Vandenberg and Kuse [66] and the Differential Aptitude Test: Spatial Relations (DAT:SR) [67]. The MRT is a test that measures an individual's ability to mentally rotate 3D objects. The test consists of a total of 20 questions divided into two sections of 10 questions each. For each question, one must select two correct answers out of a total of four options. The two correct answers represent different rotated views of the shape in question. Test-takers are allowed six minutes to complete the test, three for each section, with a pause in between. The MRT was scored out of a total of 40 points, one point for each correctly identified shape. An example question from the MRT is shown in Figure 8. Rather than mental rotation, the DAT:SR is considered to assess the skills of spatial visualization and spatial relations [35]. The DAT:SR consists of 50 questions. The left side shows a flattened pattern with some shaded parts which represents the transformed geometry of one of the four 3D shapes shown on the right. The test taker must select the shape from the right that corresponds to the flattened pattern. Twenty minutes are allotted to complete the test, and the test is scored out of 50 points, with one point for each correct answer. An example question from the DAT:SR is shown in Figure 9. To facilitate the administration and scoring of the tests, the tests were administered digitally using Microsoft Forms. To facilitate the administration and scoring of the tests, the tests were administered digitally using Microsoft Forms.

Tests of Usability and Functionality of the VR Application
We used two questionnaires to assess the usability and functionality of the VR application.

Usability Assessment
The usability assessment used in this study is based on the System Usability Scale (SUS) [68], a tool designed to assess the usability of the user experience of any device, application, or product. The instrument involves answering a series of Likert-type questions, based on a 1-5 scale. An answer of 1 indicates that the user strongly disagrees with the statement, and 5 indicates that they strongly agree. The questions used in our usability assessment are shown in Table 1. The tool's functionality is well integrated Q10 I had to learn many things before I could use this tool

Functionality Assessment
Similarly to the usability assessment, the functionality assessment involved answering a series of Likert-type questions based on a 1-5 scale, where 1 indicates that the user strongly disagrees with the statement, and 5 indicates that they strongly agree. The questions used in our assessment are shown in Table 2. Table 2. Questions used in the functionality assessment.

Q1
Navigating the command menus was not a problem for me Q6 Drawing in 3D was not a problem for me Q2 Navigating the scene was not a problem for me Q7 Deleting an object was not a problem for me Q3 Changing the scale of the scene was not a problem for me Q8 Editing an object was not a problem for me Q4 Finding the command I was looking for was not a problem for me Q9 Copying an object was not a problem for me Q5 Creating a 3D primitive was not a problem for me Q10 Undoing and redoing the last command was not a problem for me

Results
This study began with the completion of the MRT and DAT as pre-tests of spatial skills and concluded with a post-test using the same instruments. Table 3 shows the results obtained from the 29 students that participated. The distribution of the MRT pre-test and post-test data was normal and without outliers. The distribution of the DAT pre-test and post-test data was bimodal and without outliers. Following the recommendation of Poncet et al., (2016) [69], we proceeded with the paired t-test analysis for both tests. The mean score on the MRT improved by 34.0%, and the mean score on the DAT:SR improved by 1.6%. Thus, the improvement on the MRT score was higher than the improvement on the DAT:SR score, but performance improved on both tests. A Pearson correlation coefficient was computed to assess the linear relationship between the MRT pre-test score and the DAT:SR pre-test score. There was not a significant correlation between the two scores, r(28) = 0.34, p = 0.700. Figure 10 shows a scatterplot of the pre-test scores on the MRT and DAT:SR, demonstrating visually this lack of significant correlation.
A paired samples t-test was used to compare performance on the MRT pre-test and post-test. There was a significant improvement in the MRT post-test scores (M = 25.79, SD = 7.97) over the MRT pre-test scores (M = 19.24, SD = 9.28); t(28) = 5.94, p < 0.001. The effect size was large, with a Cohen's d of 1.10, 95% CI [0. 63, 1.56].
A paired samples t-test was also used to compare performance on the DAT:SR pretest and post-test. There was a significant improvement in the DAT:SR post-test scores (M = 33.07, SD = 11.32) over the scores on the DAT:SR pre-test (M = 32.52, SD = 10.86); t(28) = 4.33, p < 0.001. The effect size was large, with a Cohen's d of 0.80, 95% CI [0. 37, 1.19].
A Pearson correlation coefficient was computed to assess the linear relationship between the MRT pre-test score and the DAT:SR pre-test score. There was not a significant correlation between the two scores, r(28) = 0.34, p = 0.700. Figure 10 shows a scatterplot of the pre-test scores on the MRT and DAT:SR, demonstrating visually this lack of significant correlation. A Pearson correlation coefficient was also computed to assess the linear relationship between the MRT post-test score and the DAT:SR post-test score. There was a significant positive correlation between the two post-test scores, r(28) = 0.84, p < 0.001. Figure 11 shows a scatterplot of the pre-test scores on the MRT and DAT:SR, demonstrating visually this positive correlation. A Pearson correlation coefficient was also computed to assess the linear relationship between the MRT post-test score and the DAT:SR post-test score. There was a significant positive correlation between the two post-test scores, r(28) = 0.84, p < 0.001. Figure 11 shows a scatterplot of the pre-test scores on the MRT and DAT:SR, demonstrating visually this positive correlation. We will now report the results obtained on the questionnaires used to validate the usability and functionality of the VR application developed for use in this study. Following Lewis (1993) [70], we chose to use the mean to characterize the Likert scale data. The average score for each question on the usability assessment is reported in Table 4. Table 4. Results of usability assessment (1 = strongly disagree, 5 = strongly agree).

Question
Average Score Standard Deviation We will now report the results obtained on the questionnaires used to validate the usability and functionality of the VR application developed for use in this study. Following Lewis (1993) [70], we chose to use the mean to characterize the Likert scale data. The average score for each question on the usability assessment is reported in Table 4. As shown in Table 4, all the positive questions (even-numbered questions) received average scores above 4, and all the negative questions (odd-numbered questions) received scores below 1.5, indicating that the students found the application to have good usability. Table 5 shows the results of the functionality assessment. All users gave the highest possible rating to the questions of ease of navigating in the scene, creating the 3D primitives, drawing in 3D, and deleting the created primitives. There were also two questions rated with the lowest possible score by all users, which referred to the duplication of existing objects (Q9) and undoing or redoing previous actions (Q10). These functionalities were not implemented in the current version of our application, hence the low scores.

Discussion
Student performance on both the MRT and DAT:SR improved significantly from the pre-test to the post-test. This suggests that the 3D modeling activity the students worked on in our application may have had a positive impact on the spatial skills which are measured by these instruments. What is not as clear is why the correlation between scores on the MRT and DAT:SR was only significant in the post-test case. This could be due to the greater overall improvement in MRT scores compared to DAT:SR scores. Others have similarly reported that the DAT:SR yields relatively high pre-test scores, and that post-training gains are not as high on this test as on other spatial tests, possibly due to a ceiling effect [35].
Other possible explanations could be a practice effect that the students better understood the MRT test the second time around, or that the students put more effort into the assessments after having completed the in-person portion of this study. While the MRT is an extremely popular and widely used instrument, it has been subject to criticism regarding its accuracy and fairness [71]. Test-retest reliability is not generally demonstrated with the MRT, and even one additional test session tends to lead to improvements in scores [72]. However, this study is not unique in using the same spatial instruments as pre-and post-tests; this is a common practice across most spatial training studies. More research is needed to validate our 3D modeling activity with more students, but the initial results are promising with regard to its ability to train spatial skills.
The second portion of our study focused on the usability and functionality of our application. The usability was rated consistently high, suggesting that the participants found the application intuitive to use. The functionality was also rated highly, except for the questions of two functions that were not yet implemented into our application. These findings suggest that the interactions proposed in this activity, including the manipulation of 3D primitives to build the model and the collaboration between users, may be useful for future applications involving 3D modeling in a virtual collaborative space.
The virtual collaborative aspects of the application appeared to lead to a positive experience for participants. Gathering together in the virtual space seemed to be a fun novelty for the participants, introducing something more exciting than collaborating in real physical space or working alone in VR. Student enjoyment of our application is quite important, since student interest in VR is viewed as one of the primary drivers of its adoption in education [73]. Since we observed each group's shift in the modeling approach, which started off working one by one in the earlier models and dividing the task, and working simultaneously on different pieces of the model later on, it appears that the opportunity for collaboration influenced people's approach to the modeling task. It is possible that in the beginning, the participants needed to obtain feedback or reassurance from one another regarding how to use the application and how to best construct the model, but as they built confidence, they moved toward working independently.

Limitations and Future Work
One limitation of this study is the small sample size. Conducting this study was quite time-intensive, as the students had to spend many hours participating in the modeling activity, which hindered our ability to obtain a larger number of participants. Another limitation is that the students all had an academic background that is already associated with higher performance on spatial assessments. In the future, this application could be tested with students from other academic backgrounds.
To address the possibility of practice effects, this application could be tested in a context with a control group that either does not participate in any spatial training or who participates in a different type of spatial training activity. Such a study design for a follow-up study could help remove the possibility of improvements in scores being related, in part or in full, to practice effects. Future work could also use a different approach to validate the training, such as by incorporating the pre-and post-spatial testing into the spatial training instrument itself. This may be advantageous, as VR may also be a more accurate medium to use to assess spatial skills in comparison to traditional tests [74].
It appears that the virtual collaborative environment made the modeling task more engaging for the students than it would have been to perform the same task alone in VR. Future studies could use our tool in both conditions, alone or collaboratively, to assess the specific impact of collaboration on the effectiveness of the tool.
Regarding the usability of the application, future work could incorporate an embedded tutorial within the application rather than having a live instruction session at the start of the study. This could reduce the amount of work on the instructor's part and facilitate the use of the tool with more groups of students in the future.

Conclusions
As has been shown in numerous scientific publications, the use of VR in education is becoming more widespread and supported by educators. Within this category, immersive VR appears to be a potential tool that can foster students' development in certain aptitudes and abilities. In this work, we have focused on the development of spatial relation skills and spatial perception skills, which are considered by many to be important for students who want to pursue STEM studies. To find out the degree to which immersive VR can help students develop these skills, we developed an application with a simple navigation interface and a reduced number of commands.
We investigated two aspects of the application. First, we studied the application's potential to improve spatial abilities. The post-test scores on the MRT and DAT:SR showed significant improvement over the pre-test scores, suggesting that the application may have the potential to improve the spatial skills assessed by these two tests. Second, we studied the application's usability and functionality by having the users answer questionnaires. From the results of the evaluations of the usability and functionality of the application, we can conclude that the tool is simple enough and comfortable enough to use so as not to detract from the building of skills. These findings can be helpful for future developers of collaborative VR applications for 3D modeling who may be able to use similar interfaces and interactions to what we presented here.
Our application leverages a collaborative environment that permits users to interact simultaneously with the project and maintain audiovisual communication. While many studies have used VR for spatial skills training, training in a collaborative virtual environment is a more novel approach. Our findings suggest that collaborative virtual environments may be advantageous for the training of spatial skills.