A Comparison Study on the Learning Effectiveness of Construction Training Scenarios in a Virtual Reality Environment

: While VR-based training has been proven to improve learning effectiveness over conventional methods, there is a lack of study on its learning effectiveness due to the implementation of training modes. This study aims to investigate the learning effectiveness of engineering students under different training modes in VR-based construction design training. Three VR scenarios with varying degrees of immersiveness were developed based on Dale’s cone of learning experience, including (1) Audio-visual based training, (2) Interactive-based training, and (3) Contrived hands-on experience training. Sixteen students with varying backgrounds participated in this study. The results posit a positive correlation between learning effectiveness and the degree of immersiveness, with a mean score of 77.33%, 81.33%, and 82.67% in each training scenario, respectively. Participants with lower academic performance tend to perform signiﬁcantly better in audio-visual and interactive-based training. Meanwhile, participants with experience in gaming tend to outperform the latter group. Results also showed that participants with less experience in gaming beneﬁted the most from hands-on VR training. The ﬁndings suggest that the general audience retained the most information via hands-on VR training; however, training scenarios should be contextualized toward the targeted group to maximize learning effectiveness.


Introduction
Construction hazards leading to fatalities have been a major concern in the construction industry for decades, as it remains one of the highest death rates among the occupations in the workforce [1,2]. Several factors are associated with these incidences occurring on the construction site, primarily led by human factors, such as unsafe behaviors, insufficient education requirements, and incompetency among construction personnel [3,4]. Prior studies showed that realistic training scenarios successfully reduced the frequency of construction accidents, delivering safety-related information effectively [5,6]. Nevertheless, exposure to on-site risks and high deliverance costs has hindered the deployment of on-site training methods [7,8]. Furthermore, advanced visualization technologies are bound to revolutionize construction practitioners' methods of information and knowledge exchange and provide an opportunity for trainees to get first-hand experience of a construction site to prepare and equip them for future jobs [9,10]. In this regard, with the rapid advancement of VR technology, the adoption of VR technology in construction training gained popularity, as it similarly presents information to on-site training while eliminating the aforementioned risks [11]. While VR-based construction training proved to be a more effective method in memory retention and recall over conventional training methods, there is a lack of study on the impact of the deliverance method within the virtual environment on the learning effectiveness of users. Different types of VR-based construction training reported substantial variance in the improvement of learning effectiveness, which suggests an underlying factor affecting learning effectiveness due to the development process of VR training [12][13][14][15]. Hence, it is crucial to investigate the drawbacks of VR training scenarios under different constraints regarding memory and knowledge retention. It is essential to address some of the uncertainties present in the knowledge gap of VR training development to maximize its benefits for learners.
This study aims to (1) improve the learning effectiveness of VR-based construction training through passive and active approaches; (2) compare the performance of engineering students of varying demographics and academic performance under different VR-based construction training scenarios; and (3) propose development directions to maximize the benefits of VR-based construction training. To achieve the aim of this study, a research question was formulated as follows: What are the implications of virtual construction training scenarios under varying immersiveness conditions on engineering students of different demographics?
In this study, three different VR training scenarios were developed based on Edgar Dale's cone of experience relating to real-world applications of learning methods. It is hypothesized that as the training approach moves from passive to active methods, the learning effectiveness should improve overall. The first VR scenario is developed to deliver information through an audio-visual-based training approach, which is considered a passive method. Engineering students are taught based on a virtual instructor showing students building components while delivering a speech in the virtual environment. The second VR scenario is conducted through a semi-passive approach, which is through an interactive-based training approach. In this case, students are given opportunities to interact with the building components during the learning process, and animations are simulated in the virtual environment to aid in the visualization process. The third VR scenario utilizes an active approach, which is known as a contrived hands-on experience training. In addition to providing interaction with the building components, students are required to perform hands-on tasks to complete the training. At the end of each scenario, students are required to undergo a quiz to test their knowledge retention and recall. The results are then analyzed and critically discussed in further sections.

Virtual Reality Technology in Construction Engineering Education
The use of VR technology has steadily gained popularity in construction engineering education since decades ago, long before semi or fully immersive virtual techniques were introduced [16][17][18]. The advancement of VR technology has proven its effectiveness in delivering knowledge to learners in the construction industry, primarily due to the realistic reproduction of the construction site in a virtual environment [19][20][21]. VR's ability to simulate the construction site is highly advantageous over conventional training methods as it removes the on-site construction hazards and risks, which would otherwise be exposed to the learners [22,23]. Aside from that, VR-based construction training can mitigate the high costs of real-world training, further incentivizing the deployment of VR-based training [14,24,25].
There is a vast area of VR-based applications in the construction industry, especially in education and training. Building information modeling (BIM) technology is commonly integrated with VR to facilitate information flow between the educational game and building components [21,26]. Various VR-based educational training has been developed over the years, covering different aspects of the construction industry, such as construction safety [27][28][29], operation of construction equipment [30], construction process [31], construction management [32], and so on. The wide application of VR technology in construction education is mainly attributed to the highly interactive learning environment, providing engagement and motivation for learners to focus between training sessions [33,34]. In addition, VR technology also serves as an effective collaboration platform due to its excellence in representing the spatiotemporal dimensions [35,36]. Therefore, there is strong evidence of the benefits of VR in construction education applications [37]. Learners who were trained using the VR platform also generally performed better over conventional methods, as VR simulates hands-on experience [38]. However, some research found that there are no significant differences between VR and conventional methods, despite the strong beliefs of the participants [39]. Besides that, VR provides an improved spatial perception of the environment over conventional and non-immersive VR (e.g., desktop VR) methods [40,41]. Nevertheless, most research was conducted independently on the virtual environment or on a comparison study against conventional training methods [42,43]. Despite research reporting variance in the learning outcomes, there is insufficient knowledge of the factors affecting these results. Hence, this study posits that comparing different developmental approaches toward VR-based construction educational training allows further insights to verify these occurrences and understand their rationale.

Critical Factors Affecting the Learning Effectiveness of VR-Based Educational Training
Language is considered one of the most basic tools in delivering information in both real and virtual environments, whether through text or speech [44]. It acts as a medium for action, which is crucial in the learning process [45]. Therefore, an effective speech system is necessary to achieve high learning effectiveness in a training environment. In the context of VR-based training, this can be achieved by implementing audio speech or in combination with subtitles. Audio and visual stimuli in an immersive environment also act as a cue for an action or reaction, providing engagement with the users [46,47]. Therefore, visual and sound FX is considered an essential aspect of game development as it maintains the game's momentum [48]. Gestures are also deeply integrated with cognitive capabilities, representing emotions in a dialogue [49]. Hence, the implementation of gestures is indispensable in creating an interactive environment, which is achieved by animations in a VR game. In the educational theory on the learning process, it is argued that learning through actions is the basic building block of humanity [50]. It is further expanded that the conduct of acts is a highly effective tool for learning new information [51]. Thus, through the simulation of a hands-on experience in the virtual environment, VR contributes to a natural and effective learning method for the users.

Methodology
The experimental design of this study is presented in this section. The details of the participants, experimental design, and comparative evaluation is discussed in each subsection.

Participants
A total of 32 engineering students from the civil, mechanical, and electrical departments of Monash University Malaysia were selected to participate in this study. All participants were in their final year of study (the age of participants are roughly equal), and no monetary or non-monetary incentives were given out during the study. The participants were split equally into two groups in a random order, whereby 16 students were each separated into the control and test group. The sample size consists of 22 civil, 6 mechanical, and 4 electrical engineering students, whereby the gender distribution of the sample size consists of 24 males and 8 females. A study conducted by Wang and Dunston showed that a sample size of 16 was sufficient to obtain a reliable result [52]. Another VR education system conducted in another study also proved reliable and consistent results using a sample size of 25 [53]. A comparison study related to VR applications in the assembly process also showed reliable results with a sample size of 15 [54]. The control group was used to ensure similar difficulties between three different VR scenarios and was verified through a paper-based study to verify and minimize persisting biases across all the VR scenarios. Meanwhile, the test group underwent all three developed VR scenarios, and the collected data were used to analyze students' learning effectiveness under different VR training approaches.

Experimental Design
In this study, a quantitative approach is designed and applied to evaluate and analyze the results, as suggested by Creswell [55]. All participants from the test group are subject to three VR scenarios consecutively to minimize internal and external factors affecting the results. The VR scenarios are also designed to have the same scoring system to eliminate biases. Besides that, the type, size, and color of the user interface and graphical objects used in the testing process persist across all VR scenarios. The content of each VR scenario also shares a very similar topic and is further adjusted to ensure similar difficulties through the control group. The control group is subject to a paper-based method, whereby the learning process is conducted through a physical-based lecture on the same content as the developed VR scenarios. Then, it is immediately subjected to a multiple-choice quiz containing the same questions as the VR scenarios. Meanwhile, the demographics, assessment scores, and completion time of the participants from the test group are recorded and exported into a Microsoft Excel spreadsheet for further analysis.

Comparative Evaluation
The evaluation measures are designed based on the Charette Test Method, using a scoring and time measurement system [56]. At the end of each VR scenario, a quiz consisting of five questions, each carrying the same weightage, is provided in the virtual environment. Every question comprises three to four multiple choices and does not contain multiple correct answers. During the questioning phase, the timer is paused while the virtual instructor reads out the question. This ensures that the time measurement system is not affected by the length of the questions. Before the start of training, participants are required to fill in a pre-test demographics survey, identifying the participants' engineering disciplines, academic performance, experience in gaming, and gender. An external validation process is adopted in this study based on the charette test method and similar past research, which consists of three criteria: effectiveness, speed, and usability [56,57]. These three criteria are used to establish a relation between the three VR scenarios developed based on Dale's cone of experience [58], as shown below in Figure 1.
The cone of learning as the visual metaphor for types of learning content from the concrete learning experience at the bottom of the cone engaged more senses of the learner such as hearing, seeing, touching, etc. than the abstract experience such as seeing the words or hearing the instructions. The report by Fadel et al., (2008) supported Edgar Dale's Cone of Experience/Learning which represents the tangible reality of learning experience first-hand [59]. It is demonstrated that the least effective methods to learn new tasks are by means of reading a manual or instructions and listening to a presentation. In contrast, the most effective method of learning experiences involves doing a physical task and simulating the real task, which is called learning by doing. In this regard, the construction training scenarios in a VR environment are designed in three different stages according to Edgar Dale's cone of learning.

Design Framework
A design framework was developed for this study to systematically design and verify the VR scenario's integrity, shown below in Figure 2.
The topic selected in this study is the building components of a typical two-story residential townhouse with a Light Steel Framing system. Through VR-based training, learners are expected to gain insights into the key components of the building and its associated functionality from four different aspects, including the structural, mechanical, electrical, and plumbing point of view. After each training scenario, the external validation test was conducted immediately to assess the learners' memory retention and recall time capabilities. The learning topic slightly differs for every scenario to eliminate memory dependency from any previously learned topic.

Game Development
The VR training is developed on Unity ® 2019.3.7f1 version to support the various types of VR devices such as Personal Computer-based VR and stand-alone VR headsets. All BIM-based building components were imported from Autodesk ® Revit ® as FBX files, while the material and texture files were converted using Autodesk ® 3DS Max ® . The building components consist of light steel framing (LSF) system, one of the commonly used industrialized construction systems. The developed VR training scenarios are supported on the SteamVR platform only. The game is tested on a few different head-mounted displays (HMDs), including HTC Vive, Valve Index, and Oculus Quest 2 through Oculus Link. Nevertheless, the actual test is conducted on HTC Vive. Oculus Quest 2 was not used as there is some network latency, which could cause nausea, leading to an external factor affecting the results [40]. Meanwhile, Valve Index was not used during the actual tests as it weighs roughly 50% more than HTC Vive, which may lead to user discomfort. The game supports up to 144Hz to minimize flickering while maximizing user comfort. The computer specifications used in the development and testing stages are Nvidia GeForce RTX™ 3090, AMD Ryzen™ Threadripper™ 3990x, 128GB RAM, and 2TB SSD. The game utilizes a teleport-based movement system, and interaction is performed by grabbing, hovering, or colliding. A scene manager system is developed through visual scripting, handling different events, including audio clips, dialogues, subtitles, animation, visual effects, triggers, and game flow. A time and score manager system is also integrated into the game, which mainly interacts with the scene manager system to record data within the game. The game development times of the audio-visual, interactive, and contrived handson training scenarios were five, three, and three weeks, respectively. The relatively longer development time of the audio-visual training scenario was due to the initial development of movement, interaction, and manager system, which would subsequently be reused in the other training scenarios.

Audio-Visual-Based VR Training
The focus of this training is to deliver information through an audio-visual-based approach. In this scenario, users started outside of the residential house. A virtual instructor begins by introducing the LSF system used in the building, as shown in Figure 3a, including some structural components and their associated function. Afterward, the user is required to enter the building. The virtual instructor follows the user and then begins explaining the wall system used in the building, such as material, thermal and acoustic properties. Throughout the scenario, the user and environment interaction is minimal. The virtual lecturer is fully rigged with animation, and the speech is delivered through both audio recordings and subtitles. Some visual effects are provided to facilitate the explanation process, as shown in Figure 3b. The user is then required to participate in a quiz in the virtual environment. The quiz consists of 5 questions in total, whereby 3 questions are on the topic of the LSF system, and 2 questions are on the topic of a wall system.

Interactive-Based VR Training
The philosophy of this training method is to approach users with a highly interactive environment. The training begins by introducing some key components and the functionality of the electrical system inside the house. Then, the virtual instructor asks the user to proceed to the second floor of the building. The virtual instructor briefly discusses some implementations of smart technologies in a residential house, such as smart light bulbs. Afterward, the user is required to enter a bedroom. The virtual instructor explains the application of the heating, ventilation, and air conditioning (HVAC) systems and the factors that need to be considered during the design stage. An example of an HVAC system used in this training is the chilled water system. Lastly, the virtual instructor presents the piping system to the user, including the main water supply, greywater, and blackwater system. Throughout the scenario, the virtual environment is highly interactive in addition to the basic functionality introduced in the audio-visual-based training. For instance, users are required to open and investigate the electrical system, as shown in Figure 4a. An automatic user detection system activates or deactivates the light bulbs whenever the user enters or leaves the room. The users are also able to visualize the air ventilation in the building, as shown in Figure 4b. Aside from that, users can visualize the supply of clean water and its purpose, as shown in Figure 4c. Besides that, all HVAC and piping systems are fully animated to provide a highly immersive environment, as shown in Figure 4d. After the training, the user is required to answer five questions related to the learned topic, one on the electrical system, two on the HVAC system, and the remaining two on the water piping system.

Contrived Hands-On Experience VR Training
This training is designed to deliver knowledge through performing hands-on tasks. In addition to the functionality of the interactive-based training, hands-on tasks are also introduced in this scenario. The training begins with an introduction to the roofing system used in this building, which is a warm deck-pitched roof. Then, the user is allowed the freedom to obtain further information by interacting with the components freely. Afterward, the user is briefed on the safety measures before performing maintenance tasks. After the user completes the safety tasks, the virtual instructor teaches the user about piping maintenance methods. The user is required to perform hands-on tasks, such as unscrewing bolts, replacing gaskets, and so on. The virtual environment is highly interactive throughout the scenario, and a contrived hands-on experience is introduced whenever possible. For instance, Figure 5a shows the user removing the pipe from the connections, one of the many hands-on tasks required to complete the training. Meanwhile, Figure 5b shows the user performing a contrived hands-on task from a real-world perspective. After completing all hands-on tasks, the user is required to undergo a quiz. The quiz consists of five questions, where three are related to the roofing system, and two to the safety and maintenance works.

Results and Discussion
This section highlights the study's findings and provides an in-depth analysis of the results.

Overall Performance Comparison between All VR Training Scenarios
The comparison of the participants' overall performance between three developed VR scenarios is evaluated based on two criteria: the quiz performance score obtained in each scenario and the time required to complete it. Figure 6 shows the mean score of the participants in all VR training scenarios. From Figure 6, it is seen that participants scored 4% higher in the interactive-based training compared to the audio-visual-based training, which is equivalent to a 5.2% improvement over the audio-visual-based training. Therefore, an inference that a highly interactive virtual environment positively reinforces learning effectiveness can be established from the results. The improvement is likely due to a stronger engagement between the users and the training through a higher degree of immersiveness. Aside from that, the incorporation of contrived hands-on tasks in the virtual environment further led to an improvement of 1.3% in performance scores over interactive-based training. The performance gain from implementing a contrived hands-on experience in the VR training is less significant when compared to the performance improvement upon implementing a highly interactive environment. However, a one-way analysis of variance (ANOVA) test with a 95% confidence level showed that the effect of the training scenarios on the participants' performance score is insignificant, with F = 0.59 (DoF = 2) and p-value = 0.55. Although the statistical test showed that there are no significant differences between the mean performance under different training scenarios, it should be noted that the difficulty of each training scenario was scaled to be roughly equal. Hence, it is expected that the mean performance score should be roughly equal as well. Nevertheless, the implementation of a contrived hands-on experience benefits through memory recall time improvements, as shown in Table 1.  Table 1, it is seen that there is a decreasing trend in the mean completion time as the virtual environment becomes increasingly immersive. This further strengthened the methodology of the developmental approach, strongly relating the phenomenon to the real-world application of learning approaches. The mean completion time decreased by approximately 8% when comparing the interactive-based training against the audio-visualbased training, showing an improvement in memory recall rate. There is also a further improvement of 3% when contrived hands-on experience is exposed to the participants. An observation is made that the standard deviation and the maximum value of the completion time on contrived hands-on experience training are exceptionally high. This is likely due to the complexity of the training scenario, whereby users are required to perform hands-on tasks. Therefore, users who are unfamiliar with the controls may be overwhelmed by the situation, leading to confusion or incapability to retain information properly. A one-way ANOVA test with a 95% confidence level conducted on the completion time between three scenarios showed that F = 3.18 (DoF = 2) and p-value = 0.05.

Demographic Factors Affecting the Performance Score of the VR Training
A few key demographic factors showed varying learning behaviors between the groups. Figure 7 shows the mean performance score of different weighted average mark (WAM) groups in all VR training scenarios, whereby WAM is a scoring system used at Monash University to calculate students' academic performance. The WAM system is calculated based on the following formula: Whereby CP y is the credit points of the units undertaken during their respective years, and P y is the marks obtained for the units undertaken during their respective years. The WAM score of each student ranges from 0-100, whereby the WAM group with a score of 60-69, 70-79, and 80+ falls under the category second class honors division B, second-class honors division A, and first-class honors, respectively. In the sample size of this study, there are no students who held a WAM score of 90 and above.
As can be seen in Figure 7, there is a performance score improvement across all WAM groups when comparing the interactive-based training against the audio-visual-based training. This showed that a highly interactive virtual environment is always preferred to maximize the learning effectiveness of the users. However, there is a mixed response from different WAM groups when subjected to contrived hands-on experience training. Participants from the lowest academic performance group tend to perform worse when a hands-on training approach is utilized during the learning process. A one-way multivariate analysis of variance (MANOVA) test using Pillai's Trace showed that p-value = 0.76, F (12,36) = 0.67, and Pillai's V = 0.55, indicating that there are no significant differences between the performance of participants and each academic group. Nevertheless, participants stated that it was generally more difficult to focus on the auditory (audio recordings) and visual (subtitles) feedback as they were distracted by the given hands-on tasks, which may be affecting the performance score. Meanwhile, participants from the middle academic performance group retained knowledge more efficiently through contrived hands-on experience training. Participants felt that practicing this method is highly effective in maintaining attention, which contributes to higher memory retention capabilities. Meanwhile, a t-test conducted between male and female participants showed a p value of 0.90, indicating that there are no significant differences between the mean performance between the gender groups. Aside from that, users' experience in gaming notably affects the learning effectiveness of VR training in general as well, as shown in Table 2. The obtained results were further categorized using the demographics survey. Participants with a gaming frequency of less than once a week, regardless of the gaming platform, were categorized as "rarely/sometimes". Meanwhile, the latter group with a gaming frequency of more than once a week is considered as "often" regarding the gaming experience. Participants with more experience in gaming completed the training at a faster rate (50 s faster) and scored better (21% higher) at the same time. Participants who are familiar with gaming expressed comfort and ease of navigation in the virtual environment. Among the participants with less experience in gaming, it is generally agreed that clarity and ease of use are very significant in navigating the virtual environment.

System Usability of the VR Training Scenarios
The system usability of the developed VR training scenarios was also evaluated based on the performance score and is tabulated in Table 3. The system usability score is based on a maximum score of 5, which is derived from the evaluation of the quiz performance. From Table 3, it is seen that the system usability score across all VR training scenarios is above 3. Aside from that, the minimum score obtained in all training scenarios is above 3 as well. This indicates that all participants successfully completed each training scenario, which suggested that the developed VR training scenarios were highly usable. Nevertheless, there is a mixed reaction toward different stimuli across participants with different demographics. Although there is a noticeable difference in the usability of the training scenarios with increasing immersion, it should be noted that there are numerous factors that were not considered in this study, such as the impact of spatial perception characteristics on the usability of the training scenarios.

Suggestions for VR Training Development Guidelines
As discussed in the previous subsections, it is evident that the learning effectiveness of VR-based training improves as the user receives a higher degree of immersiveness. Therefore, a VR learning effectiveness pyramid is suggested, as shown in Figure 8. From Figure 8, it is suggested that adding the functionalities in VR-based training results in a higher degree of immersiveness, leading to higher learning effectiveness. Nevertheless, this developmental approach is generalized and should only be viewed as a theory. The findings discussed in the previous subsections also showed that some functionalities adversely affect certain demographic groups. Therefore, it is essential to identify the target audience before developing the VR training. After identifying the background of the targeted group, the VR training should be contextualized and personalized to maximize its learning effectiveness. Nevertheless, the results can only be generalized after repeating the study with a larger sample size.

Conclusions
There is currently insufficient knowledge on how the development process of VRbased training affects the learning effectiveness of users, despite research reporting significant variance in memory retention and recall capabilities in developed VR-based training. Most research tends to focus on comparing paper-based and VR-based methods, which leaves a research gap in the comparison between different VR-based methods. Therefore, this study developed three different VR-based training scenarios, varying from passive to active approaches, to further identify the critical success factors affecting the learning effectiveness of VR training. The training scenarios are developed based on Dale's cone of learning as a guideline, whereby each subsequent scenario contained an increasing degree of immersiveness. The developed training scenarios were comparatively evaluated through performance scores and completion time, whereby participants were required to partake in a virtual quiz at the end of each scenario. The findings showed that there is a positive correlation between the degree of immersiveness and learning effectiveness. However, the demographics of the participants play a significant role in dictating the outcome of the results. For instance, participants with lower academic performance reacted adversely to contrived hands-on experience training. A learning effectiveness pyramid is also proposed in this study as a general guideline in the development of VR-based training. Nevertheless, VR-based training should be contextualized based on the background of the targeted group to maximize its benefits. A limitation of this comparison study includes the limited sample size to draw a definitive conclusion. Besides that, the user cognitive load is not measured, despite the difference in the complexity of workload between the three VR scenarios. In addition, factors affecting user performance such as user spatial perception were not considered in this study. Further studies should be made by incorporating different implementation combinations of learning methods to better understand how developmental approaches affect learning outcomes. Aside from that, future work will study the effects of other factors on user learning performance, particularly on the cognitive load of users when subjected to different stimuli using an electroencephalogram (EEG).  Informed Consent Statement: Informed consent was obtained from all subjects involved in the study.

Data Availability Statement:
The data presented in this study are available on request from the corresponding author. The data are not publicly available due to protecting participants privacy under Monash University Human Ethics with project reference number 23844.