Exploring Simulation-Based Virtual Reality as a Mock-Up Tool to Support the Design of First Responders Training

Abstract

Intervention by First Responders (FRs) is essential in disaster response, and their preparation greatly benefits from continuous updates. However, the design of effective training experiences targeted to FRs can be very demanding from the viewpoint of a Training Provisioner (TP). Virtual Reality (VR) may have a key role to play in enhancing and facilitating this task. In fact, VR technology has already proven to be very helpful in the field of emergency training, as well as its use as a powerful design and mock-up tool in many other contexts. In this work, the application of VR as a mock-up tool supporting TPs in the arrangement and validation of a training experience, either real or virtual, is explored. In particular, a case study is considered concerning the training of an FR for hydro-geological risks. Within this context, the proposed approach is compared against dramaturgy prototyping, a method commonly used for the design of experiential courses. Results indicate that the adoption of a VR-based mock-up tool (VRMT) can provide TPs with good insights on the arrangement of the training and precious indications on how to actually map this information onto real-world exercises.

1. Introduction

Training plays a fundamental role in appropriate disaster response. However, in the context of preparation for high-severity, low-frequency events such as those typically faced by First Responders (FR), creating appropriate training becomes particularly challenging [1,2]. In order to develop effective training for this user group, special attention must be devoted to both designing and conducting relevant experiences [3]. Moreover, since tasks regarding equipment operation, maintenance and decision making are typically involved in these experiences, the necessity to continuously improve the training is additionally introduced [4]. Thus, given also the complexity of the adopted protocols, drafting, updating or even re-designing such experiences can be very demanding from a Training Provisioner (TP) perspective, since multiple factors need to be considered.

In this context, Virtual Reality (VR) could in principle be used as a mock-up tool to help TPs in the design process. In particular, TPs may be provided with an immersive Virtual Environment (VE) representing the training scenario and including the virtual replicas of all the equipment involved in the considered procedure; this environment could be used as a sandbox setting, where it would be easy to test all the actions conceived in the prototyping process and to evaluate, e.g., their timing and their spatial constraints, as well as their order of execution.

Indeed, VR is already used in the design review processes that are adopted in CAD-based product development, [5], in BIM modeling [6] and in several other application domains. For instance, the authors of [7] demonstrated that VR could be profitably used to double-check the risks to which construction workers could be exposed and to leverage this information for optimizing building design in order to minimize these risks. Despite these findings, the use of VR as a tool to support the design of training experiences has gone mostly unexplored.

However, VR technology itself already plays a large role in this context, as simulation-based training is attracting a great deal of interest from both academia and industry [8,9]. In fact, VR training systems are expanding beyond laboratory settings and are increasingly frequently integrated into traditional training programs [10]. This is especially the case for teaching users how to carry out activities that are mainly composed of procedural and practical tasks, including the use of special equipment and machinery, which typically benefit from learning-by-doing and hands-on training approaches [11,12]. For instance, the VR-based framework presented in [13] allowed a crisis management team—including a policeman for traffic control, a fireman for dealing with the emergency and a medical doctor for rescuing injured subjects—to collaborate within a VE representing a critical site. The considered scenario, namely a rescue mission of an injured individual lying on the floor of a subway station platform that was on fire, requested each actor in the simulation to follow different procedures and use specific tools to accomplish their own task: the policeman was in charge of delimiting the dangerous area and putting up a cordon, the fireman had to operate a fire hose to extinguish the fire, and the medical doctor had to measure the individual’s vital signs.

Moreover, the sandbox nature of the medium enables trainees to experience possibly harmful and unusual situations in a safe and repeatable virtual setting [14,15]. VR training systems have been proposed in the literature both for rare and high-risk scenarios [13,16,17], as well as for common and low-risk scenarios [18,19].

Despite the benefits brought by the use of VR technologies for simulation-based training, some aspects regarding the design and validation process of these experiences are still under investigation, as confirmed by the increasing number of studies on this topic. For instance, the authors of [20] described an empirical study on the use of VR to gather requirements in the field of simulation and training. More specifically, a number of stakeholders (navy officials and physicians) and developers collaborated in a joint design process for the development of two training scenarios. The results of the study showed that, compared with traditional requirement elicitation techniques, VR allowed developers to better identify the contextual issues of the simulations and helped stakeholders to better communicate to developers the proper position and scale of objects in the training scenarios. The authors in [21] presented a conceptual framework for building VR training simulators. The key aspect of this framework is represented by the introduction of basic principles for building the architecture of the simulators based on two layers: a user-centered design layer, aimed at modeling the subjects’ orientations, and an object-closed design layer to implement the training scripts, ensuring the specificity of operations to be performed and 3D models used in the simulations.

In this paper, the use of a VR-based mock-up tool (VRMT) to support a TP in the design and validation of a training experience is explored, by applying it as an intermediate step in the transition from a real-world training experience to a fully-fledged VR simulation. The devised methodology is applied to a case study in which Civil Protection trainees are instructed on how to set up and use a High Capacity Pumping (HCP) module. The feedback provided by TPs on the VRMT has been collected and used to compare the proposed approach with a design method that is commonly adopted in the development of experiential courses and that leverages dramaturgical techniques [22].

The activity was performed in collaboration with an Italian regional Civil Protection body (Piedmont Region Civil Protection and Forest Fire Fighting Units [23]) and the Piedmont Region Coordination body of Civil Protection Volunteering [24] in the context of the PITEM RISK [25] project (for which Politecnico di Torino has been appointed as the implementing body for the Piedmont Region Civil Protection Unit). In particular, the design of the VR training system fell within the scope of the RISK FOR [26] sub-project, which aims at improving the training of the many actors involved in the disaster management of the territory between Italy and France, and of the RISK ACT [27] sub-project, whose goal is to exploit the outputs of RISK FOR in real-world use cases such as the situation addressed in this paper.

2. Materials and Methods

2.1. Case Study

As mentioned above, the VRMT-based approach was applied to the design of a VR-oriented training procedure for one of the meteo-hydrological risk divisions of the considered FR body, the Civil Protection HCP module. For the kind of tasks involved in this context, simulation-based training [28] can be easily regarded as the most suitable choice.

The objective of the HCP module is to secure deluged areas by draining water from puddles or flooded buildings to avoid further damages and causalities. In particular, when the module is activated in an area along the sides of a river, the main goal is to eliminate puddles of water that can break the river bank.

A standard set of training guidelines for this procedure was already in place [29], but they were heavily oriented to real-world training. Hence, they were quite limiting in the context of a deployment in a VR scenario. For instance, in real-world simulation scenarios, replicating a particular disaster event (location, weather conditions, etc.) is often unfeasible, but this becomes relatively easy when representing the event in a VE.

The training experience was based on a hands-on exercise held in the city of Ivrea during a day of practical operations, when 20 participants from the Piedmont Region Civil Protection Unit and Coordination body of Civil Protection Volunteering simulated an intervention on a riverbank using actual equipment. Over five hours, a team of four researchers collected information and recorded videos to highlight the basic structure of the HCP procedure. The research team observed different participants, asked questions about the operations and possible variations and tried to use the equipment directly. Four main phases were identified to define the overall structure of the training. Since there were no well-defined roles in the exercise and each participant could perform every step of the procedure, the separation was based solely on time. A series of guidelines to be followed and equipment to be used in each phase was also outlined. The identified phases are reported below.

• Preparation of the operational field: The team must inspect the flooded ground to decide the areas of water aspiration and delivery, as well as the positioning of the pump. The pump (Figure 1d) must be located near the aspiration (suction) area, avoiding if possible elevated regions in order not to affect the pumping capacity. Once the pump is placed, the team must delimit the operational field with poles and signaling tape in order to avoid external interference. If the delivery area is another river, a protective sheet (Figure 1c) must be positioned to avoid additional damage to its bank. During this phase and the following phases, each operator must wear adequate Personal Protective Equipment (PPE), such as gloves, a helmet, safety boots and a life vest;
• Assembly of aspiration and delivery chains: The team assembles the aspiration and delivery chains. For each chain, two operators must work together to handle the transportation of the tubes and operate safely. The aspiration chain is composed of semi-rigid tubes, starting from the pump up to the suction area, and by a final element (the filter, Figure 1a) that must be immersed in the puddle to be drained. The filter must be tied by a rope to be used later for its recovery. The operator that dips the filter must be secured to a second rope held by the other operator. Starting from the pump up to the delivery area, the delivery chain can be composed of rigid or semi-rigid tubes or foldable hoses (Figure 1b). The last element of this chain must be a rigid tube, placed on the protective sheet and secured to the ground using a rope and several pegs. All the tubes and hoses used in this phase have a male and a female endpoint. During the assembly procedure, it is necessary to match a male endpoint with a female one and secure them using hooks and levers (Figure 1e);
• Pump start-up and monitoring: One operator conduces the pump, checking first if the filter and impeller valves are closed and then switching the pump on using the key. It is necessary to close all the pump doors to limit noise. During this phase, the other operators must check the aspiration and delivery chains (Figure 1f) to detect possible leakages. In case of leakages, it is necessary to rapidly stop the operations, switch the pump off and repair or change the damaged tube or hose. Finally, when all the water has been aspirated and discharged, the pump can be switched off;
• Disassembly: With the pump switched off, one operator opens the filter and impeller valves to drain the water left in the pump. Once the water is drained, the filter can be recovered using the appropriate rope. The other operators disassemble the chains. For this last step, it is necessary to start from the delivery chain by first detaching the foldable hoses.

2.2. Dramaturgy-Based Prototyping

A viable design approach which could meet the needs of prototyping a training that entails procedural tasks and equipment management is the so-called dramaturgy-based design [30]. Dramaturgy, as an experiential course design method, is a holistic approach that facilitates the design experience by integrating a wide range of social, cultural, physical, creative, emotional and reflective activities [22]. The advantages of this approach for course design have been widely studied [22,30,31].

Dramaturgy-based prototyping (DP) can be effectively applied to different design phases, such as design ideation, testing and maintenance; in particular, its use is most common in the context of design prototype testing [32]. The training design methodology adopted by the involved FR bodies usually does not make use of dramaturgy-like approaches, as training is generally developed directly in the field. In order to conduct a fair comparison of the proposed VRMT approach with the traditional alternative from the literature, the methodology adopted by Civil Protection was complemented with a dramaturgy-based training prototype testing phase. In this way, the devised methodology was adequate both to fulfil the learning requirements of the case study and to run design sessions also in an indoor environment.

The main sequential phases of DP are defined below.

• A group of experienced trainers meets in a sufficiently large indoor space, with all the material required for prototyping (stationery, blackboards with chalks, etc.);
• After having clearly defined the goal of the DP activity, the group of trainers go through a first brainstorming phase, collecting opinions, suggestions and concerns from everyone. The main tools used in this phase are notes, drawings and reference material (e.g., photos, videos, manuals, etc.);
• The brainstorming results are discussed, rearranged and converted into a peer-drafted textual and visual storyboard, which all the participants have to agree upon;
• At this point, the dramaturgy comes into play. The newly designed training is physically reproduced by the trainers. The aim is to evaluate, and possibly validate, the training prototype before applying it in the field;
• If the training prototype does not satisfy the initial requirements, the group of trainers returns to phase 2 and continues the prototyping process.

In the dramaturgy phase, most of the required actions have to be mimicked or, if feasible, reproduced by exploiting improvised props. This is because most of the HCP equipment cannot be easily moved into an indoor environment. Moreover, most of the procedures are built around a particular disaster-oriented setting (e.g., river overflow, urban flooding). Although these conditions can (and must) be simulated for the actual training, they cannot be easily simulated during the design stages; hence, the trainers have to greatly rely on their imagination. Reference videos from the past can be used to improve the mimicking; for instance, to check timings and the correctness of procedures. If an action is totally impractical for the given context (such as simulating the climbing of a ladder), it will simply be faked.

Although this approach can be easily deployed and is relatively cheap, it heavily relies on subjective aspects such as the ability of the trainer to guess what the effect in real life of the actions which have been merely roughly represented or even completely improvised will be. Most often, the training has to be further validated by applying it in the field, which can be very cost-intensive considering the complexity of the scenarios and equipment involved, especially if the prototype then is found to be flawed and a new DP session is required to improve on the design.

2.3. VRMT Implementation

The VRMT was developed to be used via an immersive VR HMD and its hand controllers. Specifically, the HTC Vive Pro [33] kit and ecosystem were used in the study, together with Valve Index [34] controllers to achieve finer control over the VE and more realistic interactions with the virtual objects. The HMD features a display resolution of $1400\times 1600$ pixels per eye, spanning a horizontal ${110}^{\circ }$ field of view with a 90 Hz refresh rate. Its native positional tracking leverages the infrared lasers emitted by the so-called base stations (built upon Valve’s Lighthouse technology) which, combined with the HMD’s built-in sensors, enable a 6DOF outside-in tracking over an area of up to 10 m × 10 m. The user interacts with the VE using the tracked controllers and their physical buttons.

The VRMT application was implemented using the Unity [35] game engine and the SteamVR framework. All the logic and algorithms were written in the C# programming language. The 3D assets used to create the VE (in particular, to recreate a high-fidelity replica of the pump) were modeled with Blender [36]. Appropriate care was taken also to reproduce the audio cues from the real pump and equipment in VR.

All the phases of the HCP training were integrated in the VRMT to offer a realistic replica of the real counterpart. Moreover, the VRMT was endowed with the possibility to let the trainees choose whether to experience the entire training (Figure 2) or just a subset of phases and tasks. The available tasks were pump placement, the delimitation of the operational field using poles and tape, the placement of the protective sheet on the river bank, pump management and the use of the safety or recovery rope.

When experiencing all the devised phases, the trainees had to choose a given role between taking care of the aspiration or the delivery chain (the other role was handled by the VRMT). Appropriate care was devoted to the simulation of tubes and hoses, implementing all the necessary steps to assemble the aspiration and delivery chains, particularly regarding the coupling of the chain elements.

Since most of the tasks involved the collaboration of two different operators, Non-Player Characters (NPCs) were added to the VRMT to assist the trainees during the whole experience; e.g., carrying the tubes (Figure 2e), helping during the assembly operations (Figure 2f,h or warning the user in case of leakages. Furthermore, the NPCs assembled the chain that was not chosen by the trainees. All the NPCs were managed using a common event-driven approach through finite-state machines.

With the aim of minimizing the discrepancies between the real and the simulated VE, accessory visual feedback (e.g., highlighting) was implemented to support the interaction with virtual objects, whereas voiceovers were added to NPCs to describe the whole procedure and give a general context to the trainees.

The implementation was validated by two officers of the Piedmont Region Civil Protection and Forest Fire Fighting Units. Overall, the validation had a positive outcome, and the officers highlighted how the experience was more “real” and engaging than initially conceived. It was also emphasized that the VRMT seemed truly capable of accurately representing the most practical aspects of the HCP procedure.

2.4. Experiment Design

To evaluate the use of the devised VRMT as a support tool in training design from a TP perspective, a user study was arranged as follows. The two most experienced trainers of the Piedmont Region Coordination body of Civil Protection Volunteering, with a background in real-world exercise and training preparation for the HCP procedure, took on the roles of TPs. Firstly, they were asked to participate as a team in a DP session (following the steps described in Section 2.2; in the session, they were requested to provide a revised training procedure as an outcome. Subsequently, three trainees from the participating Civil Protection bodies with different levels of expertise in the HCP procedure were invited to engage in the VRMT experience (one at a time), whereas TPs were asked to analyze their behavior. TPs were allowed to take notes and talk to each other as well as to the trainees while observing the HMD’s first-person perspective through a projected screen ($110″$ diagonal, 6 m distant). In the following, the two TPs will be referred to as TP 1 and TP 2.

Feedback from the TPs was collected through a structured interview by means of a questionnaire arranged in three sections. The first section included items adapted from the Instructional Materials Motivation Survey (IMMS) [37] and was devoted to the evaluation of the trainees’ performance from the TPs’s viewpoint based on the expected learning objectives. In the second section, items were designed to gather general feedback on the suitability of the VRMT as a mock-up tool. The aspects investigated included, for instance, the capability to identify training flaws, the easiness to spot potential equipment misuse and risk-exposure, the foreseeable real-world deployability, etc. Statements belonging to both the first and second section had to be rated on a scale ranging from “totally disagree” (1) to “totally agree” (5). Finally, in the third section, TPs were asked to express their preference by scoring in favor of the DP (1) or the VRMT (5) in seven dimensions.

3. Results

In this section, the results of the structured interview are reported and discussed. Raw data of the collected feedback from the TPs are integrated by the Cronbach’s $\alpha$ for each relative questionnaire section.

All of the trainees were able to complete the training in the VRMT without interruption, and no participant suffered from cybersickness due to the virtual experience.

3.1. Observed Trainees’ Performance

The trainees’ performance was evaluated as fairly good by the TPs ($M=4.17$, $SD=0.37$, $\alpha =0.86$), meaning both that the training experienced throughout the VRMT was effective in terms of learning effectiveness and, more importantly, that the TPs were able to discern the aspects concerning the correctness of the procedure execution. In more detail (

Table 1. Results of the first section of the TP questionnaire on trainees’ performance ($\alpha =0.86$). Statements with negative phrasing are marked with ${}^{\circ }$.

#	Statement/Item	TP 1	TP 2
1	Trainees made severe errors${}^{\circ }$	2	1
2	Trainees were unaware of the errors made${}^{\circ }$	2	1
3	I think the trainees acquired the knowledge and skills required to autonomously operate in a real situation	4	3
4	Overall, I am satisfied with the arranged training experience	5	4
5	I am satisfied with the competences acquired by the trainees	4	4
6	I am satisfied with the way the trainees executed the HCP procedure	4	4

), they stated that it was possible to note that the trainees made few severe mistakes and were eventually aware of these (items #1 and #2). As could be anticipated, the TPs were more or less undecided about the expected learning transfer; in other words, if the trainees would be able to perform autonomously and successfully in the field without any further real-world training (item #3). Nonetheless, they were satisfied with both the trainees’ performance and the defined training (items #4–6).

3.2. Suitability of VR as a Mock-Up Tool

An even more positive feedback was gathered in the second section of the questionnaire (

Table 2. Results of the second section of the TP questionnaire on the suitability of VR as a mock-up tool ($\alpha =0.71$).

#	Statement/Item	TP 1	TP 2
1	From the provided point of view, it was easy to get all the operations performed by the trainees (execution order, equipment interaction, coordination, etc.)	5	3
2	Observing the trainees operate in VR allowed me to analyze potential issues of the arranged training	4	4
3	I think it is foreseeable to deploy the arranged training in a real-world exercise	5	5
4	I think observing the trainees executing the arranged training in VR made me a better trainer	4	4
5	Observing trainees operate in VR enabled me to meditate on how to arrange and explain specific parts of the training	5	4
6	I could identify which parts of the arranged training should be reworked and/or revised	4	4
7	I think that VR is a great tool to prototype a FR training exercise	5	4

), as TPs definitely judged the use of the VRMT as supportive for the operations of interest ($M=4.29$, $SD=0.36$, $\alpha =0.71$). This entailed relevant aspects such as the prospect of identifying possible didactic flaws in the training procedure (item #2) and gathering insights on how to further improve and develop specific parts of the training (items #5 and #6). Generally speaking, they found VRMT to be a very good tool to prototype real-world training (item #7), as confirmed also by the outstanding score obtained for item #3 regarding the deployability of the designed training “as-is” from VR to a real-world setting. A controversial result was obtained regarding the easiness of observing the trainees’ operation and interaction from the provided first-person view (item #1); hence, in the future, other approaches should be considered to improve this aspect.

3.3. VRMT vs. DP

Less definite were the results coming from the comparative section of the questionnaire. Overall, an inconspicuous preference was assigned to VRMT over DP ($M=3.15$, $SD=0.91$, $\alpha =0.84$). Quite a mixture of preferences are observable in Figure 3. The strengths of the VRMT emerged in terms of the capability to validate in advance the organization and structure of the training (items #1 and #2). VRMT was also found to be moderately advantageous as a tool to identify possible flaws and setbacks (item #3). Although some dimensions were expected to be in favor of DP, such as that it allows faster iteration on training design (item #6), other results were quite surprising (items #4, #5 and #7). A possible interpretation of the two ties and the slight preference in favor of the DP (as being better for enabling the identification of possible equipment misuse) is that, from the perspective of the TPs, the DP involves an more active process than watching a trainee performing the training on a screen. This interpretation is in line with item #1 of Table 2, as discussed aboev (Section 3.1). Thus, in the future, a possible countermeasure could be to allow the TPs to join the VRMT as a shared virtual space and let them freely observe and interact with the simulated environment and the trainees.

3.4. Discussion

After the experience, the two TPs were asked to provide their open feedback in order to collect opinions, concerns and suggestions.

Both the TPs admitted being not very familiar with the dramaturgy part of the DP, since they usually design training experiences which are easily verifiable by putting them into practice. TP 1 claimed to be very expert in the considered HCP procedure, whereas TP 2 claimed to be more specialized in other procedures. This is probably the reason why TP 2 seemed to identify a lower amount of errors made by the observed trainees in the VRMT.

Regarding the comments made by TP 1, he affirmed that DP and VRTM can be seen as complementary. He explained this comment by pointing out that, despite the VRTM effectively surpassing DP from many points of view, it has at least one weakness compared with DP. Specifically, he found it very difficult to identify incorrect actions and behaviors while observing the trainees within the VRMT. To cope with this issue, he suggested that different points of view other than the first-person view could be provided. These comments are in line with the scores provided in the direct comparison section of the questionnaire, where TP 1 was more inclined towards DP than his colleague.

TP 2 praised the remarkable ease of use of the VRTM and the fact that it could significantly simplify the organization of training. In particular, he mentioned that in-field tests are normally quite difficult to arrange, and the organization becomes even more complex in the case of exceptional events such as the current pandemic. Especially in these cases, the advantages of VRTM surpass those of DP, since a VRTM can be designed as an on-line multi-user experience or at least streamed over a network, thus reducing the need for in-person, physical interactions. In the same way as TP 1, TP 2 also noticed the difficulties in spotting safety-critical incorrect actions made by the trainees within the VRMT.

3.5. Trainees Survey

Aside from the main user study, the opinion of the three trainees was also collected by means of a dedicated questionnaire, with the aim of evaluating their experience in the VRMT. In the following, the trainees will be referred to as TR 1, TR 2 and TR 3.

The trainee questionnaire was organized in two parts. The first part included the Attractiveness (ATT) and Hedonic Quality Stimulation (HQ-S) dimensions from the AttrakDiff questionnaire [38], similarly to the approach in [39]. This part presented a set of opposite terms: for each pair, a value in a scale ranging from 1 (the positive term) to 7 (the negative term) had to be assigned to describe the experience in the VRMT. The results, as reported in Figure 4, show that the provided judgements were either positive or neutral for each considered dimension.

The second section included custom statements to investigate aspects such as the trainees’ satisfaction with the arranged training, the cooperation with the NPCs and the interaction with the TPs; open feedback was collected as well (

Table 3. Results of the custom section of the trainee questionnaire ($\alpha =0.84$). Statements with negative phrasing are marked with ${}^{\circ }$.

#	Statement/Item	TR 1	TR 2	TR 3
1	Being observed by a trainer negatively influenced my performance${}^{\circ }$	2	1	1
2	I think I handled the (simulated) equipment appropriately	4	3	3
3	Overall, I am satisfied with this experience	4	4	5
4	I found that the Non-Player Characters (NPCs) simulation was well designed	3	4	5
5	I would have preferred that the other operators were human-controlled (instead of NPCs)	5	5	3

).

The presence of the TPs apparently did not negatively affect the trainees’ behavior (item #1). Trainees evaluated the experience as satisfactory (item #3). For instance, TR 3 commented that the devised experience was capable of letting him acquire the intended competences more easily than he expected beforehand.

However, trainees were only moderately confident about the handling of simulated equipment (item #2), as corroborated also by the feedback provided by TR 2 and TR 3, who stated that it was quite awkward for them not to feel the weights of the tubes when carrying them to build the chains.

Overall, trainees were satisfied with the NPCs’ behavior. However, TR 1 would have preferred in some circumstances to see the NPCs walking around to double-check the equipment instead of waiting for the progression of the procedure. Having the NPCs controlled by a human (or by improved state machines) was suggested as a compelling option to further improve the multi-user fidelity of the simulation, and this is worthwhile to consider in the future.

4. Conclusions

In this paper, the use of VR technology as a mock-up tool for designing FR training procedures was investigated. A VRMT scenario was developed and used to investigate and enhance an existing FR training procedure regarding the setup and use of the HCP module to be taught to Civil Protection trainees. Starting from a well-defined training protocol, the aim was to expand the training in order to take advantage of the full potential of VR technology, with the future goal of making it ready to be integrated in a VR-based training scenario.

The proposed approach was then compared with a traditional training design methodology, namely DP. Two expert TPs from the involved FR bodies were asked to test the two approaches and provide their judgements in different dimensions.

The VRMT emerged as a valid approach to support the training design from the TPs perspective, including regarding particularly relevant aspects such as the ability to analyze the trainees’ performance, to identify flaws and gather insights on how to improve specific parts of the training, as well as the expected deployability in a real-world training scenario. The side-by-side comparison with the DP approach highlighted that the VRMT emerged as a clear winner in terms of ease of validation for both the organization and structure of training. The main limitation that was found for the VRMT was that it is more laborious to iterate on the training design than the DP; hence, in the future, efforts should be devoted to improving this aspect by considering also, e.g., no-code programming [40] approaches to facilitate unskilled TPs in directly modifying the mock-up. Finally, the TPs found that with the VRMT was more challenging than with the DP to identify possible equipment misuse, probably due to the provided first-person perspective view of the trainees. Thus, in the future, it is advisable to allow the TPs to join the VRMT as in a shared virtual space and to let them freely observe and interact with the simulated environment and the trainees.