Remote Virtual Simulation for Incident Commanders—Cognitive Aspects

Abstract

Due to the COVID-19 restrictions, on-site Incident Commander (IC) practical training and examinations in Sweden were canceled as of March 2020. The graduation of one IC class was, however, conducted through Remote Virtual Simulation (RVS), the first such examination to our current knowledge. This paper presents the necessary enablers for setting up RVS and its influence on cognitive aspects of assessing practical competences. Data were gathered through observations, questionnaires, and interviews from students and instructors, using action-case research methodology. The results show the potential of RVS for supporting higher cognitive processes, such as recognition, comprehension, problem solving, decision making, and allowed students to demonstrate whether they had achieved the required learning objectives. Other reported benefits were the value of not gathering people (imposed by the pandemic), experiencing new, challenging incident scenarios, increased motivation for applying RVS based training both for students and instructors, and reduced traveling (corresponding to 15,400 km for a class). While further research is needed for defining how to integrate RVS in practical training and assessment for IC education and for increased generalizability, this research pinpoints current benefits and limitations, in relation to the cognitive aspects and in comparison, to previous examination formats.

1. Introduction

Fire and Rescue Service (FRS) personnel respond to a wide range of emergencies affecting the civil society. The Incident Commander (IC) on the first (lowest) level in the command chain (IC-1) is often the first officer arriving at the incident scene, and thereby responsible for the initial assessment, decisions on the initial actions, and for providing accurate and informative reports to higher officers and/or the command center.

Incident commanders are devoted firefighters who have acquired additional competence for leading responses. An IC at the first level of command (IC-1) will usually lead four or five firefighters’ actions with relevant equipment (a firetruck and a water truck) during handling routine incidents (for the emergency services) and the initial phase of more serious incidents, until an IC trained at a higher level arrives at the scene. There are several levels of command and related training courses, in many European Countries four or five levels [1,2,3] where the levels reflect the extent and severity of the incidents one may take the command over the response. The number of persons with higher qualification is lower for each level. The total force of the Swedish Fire Service consists of 12,500 responders (of which 2/3 are employed part-time, i.e., have other regular jobs as their main occupation). Sweden has about 2500 responders qualified as IC-1.

The education to become a firefighter includes practical training to acquire technical skills (handling equipment and performing operations according to procedures). The additional education to become IC-1 focuses on improving “non-technical skills” [4] as situational awareness, decision making, communication, and leadership. For firefighters, education is offered in classroom sessions and practical training at the training field. When practical training is scenario-based, it is called Live Simulation (LS). The scenarios unfold in a physical, controlled environment, using real fire (burning wood or gas), smoke, vehicles, and people acting according to a predefined setup for an arranged incident scenario. The physical objects and environment in LS are considered to allow naturalistic experiences, and thereby trigger cognitive processes in a similar way as real incidents do. Since the training of firefighter students is coordinated with the training of IC-students, the latter can command a student-firefighter team, thus practicing communication and leadership. However, LS has limitations as a method for training ICs, since the training facilities (involving a limited number of steel-and-concrete buildings, which have already sustained numerous fires) may not provide adequate variation and detailed cues to train situational awareness and decision making.

During the last decade, Virtual Simulation (VS) has become a mature method for practice-based training, implemented by several organizations.

There are contradictory opinions among stakeholders regarding the effectiveness of VS for training from the different educational fields, such as medicine, nursing, architecture, and management. However, also critical voices recognize the possible complementary value of VS training as a supplement to LS training [5].

Since COVID-19 hindered many training possibilities, especially for groups in laboratories or training grounds, the focus on allowing remote training (in the format of Remote Virtual Simulation, RVS), supporting the targeted cognitive processes has increased. Training and learning interventions are often discussed from the angle of cognitive science [6], due to the influence of cognitive load [7] for understanding and solving tasks. Additionally, higher cognitive processes such as recognition, decision making, and problem solving are essential during an emergency response [8].

In March 2020, the Swedish Civil Contingencies Agency (MSB), responsible for firefighter and IC-education in Sweden, stopped all on-site training to avoid gathering of people. By that time, one class (22 students) only lacked the final LS examination to qualify. At the same time, several of the FRSs needed the qualification, to increase their resilience during the pandemic. The MSB, having experience with VS on-site for basic IC training, decided to conduct the final exams using RVS for this class, based on the successful results of a pilot-test. This was the first IC-examination in remote virtual environments, to our current knowledge. Based on the action case research approach suggested by Braa and Vidgen [9], and the theoretical framework of cognitive science [10,11,12] this case report investigates the implementation of the practical part of the final examination for one class of IC-students, using RVS, at MSB, Sweden, through the following hypothesis:

Hypothesis 1 (H1).

RVS supports cognitive aspects of recognition, decision making, and problem solving adequately to allow students to demonstrate IC-skills.

Hypothesis 2 (H2).

Through RVS examination, the instructors can assess the student’s skills as ICs.

Since RVS had not been used before, the first question was whether it could be used at all, if it would be accepted, and if so, how it related to earlier practices. Conducting an RVS pilot test was a necessary step prior to the final examination for one class in the RVS format. The results of the present action case may inform FRS professionals’ educators how RVS can be used for training and assessment. The post-exam evaluation from instructors (performed through interviews) and students (performed through questionnaires) aims at answering the above hypotheses. The results also suggest that considering higher cognitive processes for evaluation of (R)VS tools may be a viable method for comparing and improving such tools and implementing them in future education.

The results are based on the implementation of RVS for the final examination of one (the first) IC-class (22 students). We acknowledge the low number of students as a limitation for the study. The scenarios used (described later) are according to the curriculum for IC-1, involving straightforward responses, without conflicting goals or high emotional pressure. Therefore, the results, may not be generalizable to training of higher levels of command. The conclusions are made in the Swedish context, with the specific resources, technical setting, educational structure, economic and organizational structure.

2. Theoretical Background

2.1. Cognitive Science

Cognitive Science represents a multidisciplinary approach to the human mind, often focusing on its problem-solving capabilities, as the differences between novices and experts [10,11,13]. Research on education often relies on contributions from Cognitive Science, especially for defining aspects improving learning [14] and tries to understand how the human brain functions. The Layered Reference Model of the Brain [12] decomposes cognitive processes into six layers (subconscious: Sensation, memory, perception, and action; and conscious: Meta cognitive processes and higher cognitive processes) encapsulating a total of 37 elemental cognitive processes. Among the elements of Layer 6 (higher cognitive processes), we find the cognitive elements of recognition, learning, decision making, and problem solving, which can be considered essential for training, contra not directly assessable subconscious elements [15].

VS for learning involves purposeful, computer generated graphical environments where the user can interact with the environment and representations of objects and humans, and based on specific rules, experience the effects of the interaction. This requires creating relevant scenarios unfolding in simulated real-life settings [16] with the potential to reveal cognitive learning processes through behavioral indicators [17]. RVS is conceptually not different from VS (such as to the pedagogical and cognitive aspects). However, it is different from VS regarding the technology and conduction, and this may influence the experience of both instructors and students in unknown ways.

To investigate H1 and H2, whether (R)VS supports the students’ cognitive aspects to perform and demonstrate knowledge as IC, and the possibility of the instructors to assess it, we considered how virtual training and assessment interacts with the 16 higher cognitive processes of the brain (Layer 6) [12]. The exclusion process for some of the cognitive processes, is described below. Recognition (6.1) is crucial for performing as IC, also in the virtual environment. The used questionnaire includes questions about the perceived realism of the virtual space, as to buildings, vehicles, avatars, flame, and smoke, thus addressing the Cognitive Function, Recognition (6.1). Subtle cues can also be included in the scenarios by the instructors, to train and assess the student’s recognition. Imagery (6.2) addresses the cognitive process of abstractly seeing visual images stored in the brain, without any sensory input. This cognitive process is not directly assessable and could only have been revealed by asking each student in interviews, which stored images they recalled, which was not done. Comprehension (6.3) is the action or capability of understanding, thus constructing a representation of the incident site. The cognitive aspect of comprehension draws parallels to the concept of situational awareness, as defined by Endsley [18,19,20] (involving recognition and interpretation of relevant cues as well as projection of the perceived situation in the time relevant for operation) and acknowledged by Flin et al. [4] as a very important non-technical skill. Comprehension can be assessed in VS by triggering events, effects or visualize cues, letting the instructors use firefighter avatars and asking the IC-students questions about various elements at the virtual incident site. This was actively used by the instructors during RVS examination. Learning (6.4) is about gaining knowledge or skill in some action or practice. Detailed learning objectives for IC-1 students involve procedural knowledge of the duties included in the role. These are thoroughly assessed. Reasoning, including Deduction and Induction (6.5, 6.6, and 6.7) are not stressed in the objectives of IC-1. Decision Making (6.8) is the process of choosing a course of action, among a set of alternatives. The human decision-making process is heavily affected by time constrains and level of risk. While analytical processes are used when time is ample, rule-based or intuitive processes are used when time is limited, as in incident command [4]. The Recognition-Primed Decision (RPD) model of rapid decision making [21,22] is often used when studying decisions at the incident site [23,24]. The IC-student must decide which actions shall be implemented (by the avatars) to resolve the incident and is thereby assessable. Problem Solving (6.9) is the way to the goal, mitigating consequences to the lowest possible level of damage using the available resources. If appropriate decisions were taken (and implemented through understandable orders to the avatars) the situation in the virtual environment will improve. Otherwise, adversity will increase, thereby making the cognitive process assessable. The cognitive processes of Explanation (6.10), Analysis (6.11), and Synthesis (6.12) may take place in the aftermaths of the active training or examination session and are thereby not assessable in the VS, but in the reflective feedback afterwards. Creation (6.13) is not expected to occur while training/assessing IC-1 students. Analogy (6.14) is a process in which a person understands a situation in terms of another situation. It may have links to the model of Recognition Primed Decision [21]. However, this model is associated with experience, which IC-1 students still do not have much of in the new role but may have from the role as a firefighter. This cognitive process has been considered not assessable. Planning (6.15) finds differences between the current and desired situation and governs decisions and actions. Planning involves also “instant pre-play” a cognitive process involving Imagery (6.2) to assess whether a choice of action is believed to give a favorable outcome for the affected people. Not directly assessable, but through 6.8 and 6.9 (Decision Making and Problem Solving). The last cognitive process, Quantification (6.16) has been considered less relevant for the job of the IC-1.

2.2. Simulation Training

Since cognitive aspects are extensively studied in healthcare, education or technology development, many influencing studies come from these areas. Virtual reality applications are highly domain specific, thus restricting generalizability, at least at the present state of maturity of the research area. Many influencing studies come from health care. Among others, research has been conducted on patient simulators [25,26,27], simulation for the operating room [28,29], prehospital care [30,31], pain [32], psychotherapy and cognitive support [33]. Other articles explore the qualities of simulators needed to engage health care practitioners [34,35]. Education expands in many different other domains using virtual reality technologies [36,37,38] or serious games in general for simulating the “work environment”, e.g., Labster for Biotech subjects [39] or for a wide range of courses in virtual worlds such as Sloodle [40] or MaxWhere [41]. Other articles present technological advancements influencing cognitive aspects while using these new technologies, e.g., support of attention [42], navigation, and orientation (the different technologies require different support for navigation) [43,44,45] handling empathy [46] or emotion [47]. However, determining the added value of the different technologies for the various domains is demanding. It would be essential to know if more immersiveness contributes to more effective work [48] or learning [49] or how it is related to the design of the used environments [50]. There are questions about realism in virtual reality or serious games, and how the simulation fidelity is influenced by a buy-in effect of simulation technologies [51]. Often, simple analyses, e.g., a SWOT analysis [52] or ROI [53] may give a better insight into the added values. However, it is difficult to compare technologies in various domains and usage conditions.

2.3. Learning Approach

To take the role as an IC for the first level of command in a Swedish FRS, the person is required by law to have an “IC-1 course diploma” from the MSB College [54]. The pedagogical basis of IC education is based on reaching the third stage of Blooms taxonomy, i.e., remembering, understanding, and acting (through simulation training) [55]. Learning activities are organized accordingly in different learning spaces, as described in chapter 4. Procedures and actions during incident command are described in textbooks [56] and curriculums of fire academies [57]. The most important phases, and actions (the IC is expected to perform), visualised by Wijkmark and Heldal [58], are presented in Figure 1.

The student will go through all the phases of the response and thereby train and show the observable behaviors corresponding to the course sub-objectives, see Figure 1; confirm call; prepare team; initial orders; risk analysis; window report (by radio, describing what object is affected, what is the damage, and the current threat); stop the traffic; gather information (by talking to people on-site, perform reconnaissance); decision on actions (tactics, risk, make the optimal use of resources at hand); communication (team, higher command level in the FRS); collaborate with police and ambulance services; follow-up with a situation report (by radio); including the object, damage, threats, goal, actions that have been taken, and estimation of time; evaluate the effects of actions taken; end the incident operation. This is the IC-1 part of the seven-step model, the procedure, and command support tool applied in incidents in Sweden [56].

The focus of the IC-1 course is “routine incidents”, i.e., house fires and car crashes, however, the content is not static. As the number of electric drive and hybrid electric vehicles has increased, this issue is now addressed in the education [59]. Moreover, overturn accidents with hazardous cargo are included, as Figure 2 shows. Increased consumption of different chemicals, generating increased transport [60], is a motivation for including these issues in the education.

3. Method

Cooperation between practitioners and scientists is often initiated by (at least) part of the organization who wish to explore a new technology for training purposes, thus making the collaboration process an intervention to achieve a desirable change. This points towards action research [61]. However, the aim of understanding the rich context of the domain, as well as the needs of the users (instructors and students) are also important for the scientists. This points towards interpretation and case study as a research method [62]. Braa and Vidgen [9] recognized the dilemmas often involved in “in-context” research and suggested an action case as a research methodology, often suitable when conducting information system research in the organizational context (such as the present study, conducted at MSB). The method recognizes the importance (and necessity) of balancing action (towards desirable change) with obtaining understanding, through interpretation. Studying the implementation of new techniques (without collaborating in the design of the technique), is suggested by Braa and Vidgen [9] to be a typical action case, if the participation of the organization in testing is adequate. The present work has the characteristics necessary to be typified as an action case. It is common that projects consist of several testing stages, i.e., in our case a pilot-test and a final examination conducted as RVS. These were compared to previous experiences in the training field through LS.

For the pilot test, a group of four instructors designed five scenarios, representing the challenges IC-1 students must resolve to qualify. These scenarios were prepared in the software XVR On-Scene, used by MSB, (XVR-sim, Delft, The Netherlands), and the remote technical setup was developed. Eight experienced ICs from different Swedish FRSs were invited to participate as “students”. The instructors (i1–i4) and the pilot-test-participants were interviewed after the pilot-test. The results of the pilot-test were used as the foundation to the next step, the RVS examination. Of the 22 students participating in the RVS examination, 20 chose to answer the pre-exam and post-exam questionnaires.

The study was based on the battery of questions developed by Schroeder et al. [48] with added questions regarding the current incidents and to relate the experiences in the RVS to the LS environments. These added questions were inspired by the cognitive aspects presented in the “Layered Cognitive Model of the Brain” [12]. The pre-exam questionnaire covered background information of the participants (six questions) addressing the experience as firefighters, gaming experience, and familiarity to virtual simulation, followed by a post-exam questionnaire addressing the RVS examination experiences (25 questions). During the RVS, data were also collected in observations, and afterwards the instructors were interviewed. The instructor group was strengthened with one more instructor for the exam (i5), to provide redundancy in the case of illness. The five instructors/assessors were interviewed, after the exam was completed.

4. Learning Spaces

IC education often involves three different learning spaces: The Classroom Setting (CS), the Live Simulation (LS), and the Virtual Simulation (VS).

4.1. Classroom Setting (CS)

In the CS, natural-science-based lectures aim at creating a theoretical understanding of the potential hazards, combined with knowledge on command principles and legal aspects. This is often performed using cases and scenarios illustrated in PowerPoint slides [63], videos, pictures with added animation of fire and smoke, used in discussions or table-top training using models of cities, as shown in Figure 2. In CS, the focus is on discussion-based learning, reaching the first two steps of Bloom’s taxonomy [55], remembering and understanding. CS can be at a fire academy campus or performed via a distance learning system, allowing the students to participate from their home or fire station.

4.2. Live Simulation (LS)

Live Simulation (LS) is included in practice-based training of firefighters and ICs worldwide. Fire academies have training facilities (buildings, fire trucks, and equipment), allowing for the simulation of several scenarios in a physical and the same geographical space. LS is used to allow firefighter students to train technical skills, and IC-students to practice decision-making competences, in a controlled environment. In simulation training, the IC-student will step into the role of the IC and lead a team of firefighter students in a simulated scenario. The IC-student must perform in the simulated incident, not just discuss or describe what she/he would have done in the situation. This is an important learning step, taking the student to the third step of Bloom’s taxonomy, i.e., not simply remembering, and understanding, but also acting [55].

The steel and concrete buildings available in the fire colleges world-wide are built to withstand fire and water several times a day, for years. They must also represent different types of real-life buildings. As an example, the building shown in Figure 3 would represent an apartment building in one scenario and a mechanical workshop in another. Thus, they cannot look like any real-life buildings. Due to environmental and safety precautions, quantities and types of fuel are regulated, resulting in a controlled fire development. This limits the possible development of the fire, the cues, events, and consequences of the decision making and actions.

The final examination of IC-students at MSB, have been performed in LS at the training ground, while the IC-candidate resolves the incident by following the steps described above. The instructors/assessors stand aside in the training field, observing and listening to the IC-1 student communicating with firefighters, bystanders on-site and via radio to the higher command level or dispatch center.

4.3. Virtual Simulation (VS)

On-site VS, using computer-simulated scenarios in 3D environments, has been used internationally and by MSB during recent years [64,65]. In VS, a student can act in the role of an IC in front of a large screen, move around in the virtual environment using a gamepad, talk to avatars (e.g., firefighters or bystanders) and make decisions on actions that are carried out in the simulated environment. In Figure 4 is an example of an apartment fire, that has spread to the roof (left) and a garage fire, while the affected family stands outside their home (to be compared to the LS settings in the previous chapter). The counterplay, i.e., the response by avatars and radio communication, is played by instructors, either in live role-play (by approaching the student) or through a speaker. Radios are used for communication as in a real incident. The setup in the room is schematically described in Figure 5a and an actual picture is shown in Figure 5b.

Based on the learning objectives, the instructor can build the scenario in a VS software tool, with prepared events and triggers, depending on the scenario and expected actions that the IC-student will take. During the training session, the scenarios are instructor controlled, giving the instructors the possibility to change the situation, and to “effectuate” the student’s orders and act through various avatars. The IC-student acts in the incident, thus reaching the third step in Bloom’s [55] taxonomy. At MSB, VS had been used for onsite basic IC-training, and not for examination, until March 2020.

5. Enablers for RVS Examination

5.1. Experiences of VS Training On-Site at MSB before the COVID-19 Pandemic

VS training had been used in the IC-1 ordinary training schedule, i.e., covering 2 days, for all students at one of the two MSB Colleges (Sandø) since 2018. During the VS training from January to September 2019, the experiences of 90 students (35% of all IC-1 students in MSB in 2019) were studied and analyzed [66]. An excerpt of the data providing evidence for the viability of VS as a training format (performed at MSB Sandø) is presented in

Table 1. Students’ response (n = 90) to VS training on-site, excerpt from [66]. Answers in Likert scale, 1 (low) to 5 (very high).

Questions	% Answers Likert 4 or 5	Average (Scale 1–5)	Standard Deviation
Experienced presence, compared to previous very high?	72%	3.90	0.83
Experienced presence in the simulated environment?	59%	3.63	0.86
Experienced being in the same env. as the “persons” you met?	68%	3.81	0.99
How easy was it to understand the training objectives?	80%	4.16	0.73
Would you like to perform similar training at your fire station?	100%	4.86	0.35
Would you like to perform similar training in your spare time?	80%	4.28	0.95

.

The results from this study provided instructors with extensive experience in developing and adjusting virtual scenarios, conducting VS training, and adjusting technological solutions. The data in Table 1 show the student information available to MSB when the COVID-19 pandemic struck. The number of students who answered the questionnaires were sufficiently large to provide internal validity. The acceptance and experience of VS by involved instructors were the foundation to support further action. Enabled by this experience, the instructors at MSB managed to adjust scenarios and develop the technical setup for the remote format of the examination and perform a pilot study, only days after the COVID-19 closure in March 2020.

5.2. The Pilot Test, before Deciding upon Remote Virtual Simulation Examination

In the few reported cases of using VS for assessment [1,67], the sessions have been held on-site, not remotely, which was a pressing need. However, the learning objectives, reflecting the necessary competencies for safe and effective incident command are the same. In LS, students and instructors are in the same physical space. The instructors/assessors can watch the student move, observe (see and hear) when he/she talks, and then observe the actions of the persons the student talked to and to observe the actions taken, e.g., if the leadership and the communication is satisfactory in relation to the assessment. Testing how the instructors/assessors would manage this, to reliably assess the students’ performance in RVS, needed to be developed. This motivated the Pilot test, to explore the feasibility as well as test modes for transmitting to the instructors/assessors the necessary information to reliably assess the students.

Five scenarios were designed and built for RVS examination, see

Table 2. The scenarios used during the RSV examination.

Nr	Scenario Description	Learning Points Observed and Assessed
S1	Road traffic collision. A farmer has ended up in the ditch while attempting to avoid a collision with a deer. The farmer is not injured. On the pickup he has an IBC (Intermediate Bulk Container) with an unknown chemical. The tank faucet had been damaged and there was a leak. The chemical is Roundup, a herbicide that cannot be found in the decision support tool used by ICs in Sweden.	GENERAL LEARNING POINTS as presented in Figure 1. SPECIFIC FOR THIS SCNENARIO: - Gather information about the chemical and the tank. - If the chemical is unknown to the student, ask for support from the command center. - Decide on how to handle the chemical, and the leak. - Make sure the animal is handled.
S2	A garage attached to a Villa is on fire. The fire has started in a pile of junk in the garage attached to a villa. The family is safe outside.	GENERAL LEARNING POINTS as in Figure 1. SPECIFIC FOR THIS SCENARIO - Make sure no one is inside the villa - Gather information on what is in the garage and make correct decisions accordingly
S3	A fire in an apartment on the third floor. It is uncertain if anyone is inside the apartment initially. After a while, the friend of the owner of the apartment approaches the IC and explains that the owner is abroad, but her cat is in the apartment.	GENERAL LEARNING POINTS as in Figure 1. SPECIFIC FOR THIS SCENARIO - Make a suitable decision on tactics. - Gather information about the apartment and if someone is inside. - Inform the owner of the building about the end of the operation.
S4	Road traffic collision including three vehicles under an overpass. The collision is caused by timber on the road, that have come loose from a timber truck.	GENERAL LEARNING POINTS as in Figure 1. SPECIFIC FOR THIS SCENARIO - During reconnaissance, discover the timber and thereby the complexity of the incident. - Risk analysis and restrictions on where the firefighters can work. - Divide the incident into sectors and prepare orders for arriving firetrucks.
S5	Fire in a warehouse. Some youngsters have broken into the warehouse and started two fires before they left. There are caravans and vehicles, welding gas, etc. inside.	GENERAL LEARNING POINTS as in Figure 1. SPECIFIC FOR THIS SCENARIO - During reconnaissance, discover the other fire and thereby see the complexity of the incident. - Risk analysis and restrictions on where the firefighters can work. - Divide the incident into sectors and prepare orders for the arriving firetrucks. - Participate in a command meeting when the next level commander arrives, report on the actions taken and the plan.

for a brief overview. The technical setup, where the assessor could see the student’s face at all time, hear everything said, and see what the student was looking at in the virtual environment, was developed, to provide the necessary assessment conditions as in LS (where the instructor can see the student at all time), see Figure 6. The audio and radio solution were setup using mobile phones and the standard digital communication tool in Sweden, RAKEL (RAdioKommunikation för Effektiv Ledning). The instructors could act as any of the persons involved in the incidents, e.g., another firefighter or a bystander, by choosing a corresponding avatar. The objectives of the Pilot test were to check the technology setup, the required bandwidth, the ease-of-use of technology mainly at the student site (which could be any fire station or the student’s home), and to validate the scenarios and the assessment conditions.

Before the test, several of the eight expert ICs described their moderate expectations towards RVS, including concern of technical problems and difficulty in believing that RVS could be a satisfactory replacement for LS. The objectives of the pilot test were to check the technology setup, the required bandwidth, the ease-of-use of technology mainly at the student site (which could be any fire station or the student’s home). They also had to give their comments on the scenarios, as well as the instructors’ and assessors’ role for running the scenarios. The instructors and assessors performed the counterplay and assessed these “expert-students” remotely. Valuable opinions regarding the setup as to what the assessor must see and hear to provide evidence-based assessment were expressed.

The evaluation of the pilot test showed a positive turn in attitude towards RVS. All the participants agreed that the scenarios were designed to match the learning goals IC-1s need to achieve at a suitable level and corresponded to previous IC-1 LS examinations. They also agreed on the ease of use of technology at the student site. Using their computer keyboard, they could move in the environment on the incident scene, which was projected on a larger screen to allow the sense of higher presence [68]. No lagging was detected, and the communication via radio was working as within real incidents and previous LS examinations.

One pilot test participant, stated with a quite surprised tone, after the test: “This was really great. Why haven’t you done this before? Everything you need [to perform in the role as an IC on the scene] is there”. The instructors conducting the pilot test also expressed their experience as a positive surprise.

One of the researchers observed the instructors/assessors while “assessing the pilot-test students”. It was noticed that the display showing the student’s face was nearly not used, while the display showing what the student was looking at was in use most of the time. This means that the instructors extract useful information about what the student focuses attention on, and whether that is conscious “reading” of cues. This means that the assessors can follow the cognitive processes of the student, such as Recognition (6.1) and Comprehension (6.3) [12].

The pilot test compensated for the lack of experience in performing VS examination at MSB and was valuable for checking technical issues to perform the examination remotely. After the evaluation of the pilot test had been presented to the MSB management, it took only 15 days until the formal decision to perform an RVS examination was taken. The RVS examination was conducted during the period of 27 April–19 May 2020.

6. Results: RVS Examination

6.1. The RVS Examination—The Students’ Experiences

The information gathered through pre-exam questionnaires revealed that all the students were men, with an average age of 40 (span between 32 and 56 years of age). The average number of years as a firefighter was 12, ranging from 3 to 31 years. Seven were part-time firefighters in rural areas, while 13 were full-time firefighters in cities. The experience of real fires among the IC-students varied from no real compartment fire (building fire) experience, to above 100 real fires.

Regarding familiarity with computer or mobile phone games, 70% never played computer games, and 60% never played mobile phone games. Only 15% stated that they played mobile phone games more than a few times per month, and no one played computer games more than a few times per month. Their previous knowledge or familiarity of VS for FRSs was limited. Three had prior knowledge or experience of the software used, gained from participation in VS projects where their FRS collaborated with MSB, and two had previous experience with the Response Simulator (https://www.vstepsimulation.com/response-simulator/rs-creator, accessed 17 February 2021).

Oral spontaneous comments from the students (documented by notes and video recordings) after completing the five scenarios, and/or free text comments written in the post-exam questionnaires are presented in

Table 3. Students’ experience of the RVS examination, oral comments (documented in notes and video recordings), and/or written free text comments on post-exam questionnaire.

Student	Comment
S1	I think it worked out well. Thanks to you [instructors], and it must be more of this in the course, especially remotely. It was gold [great], as close to real as it can get. And I did not have to drive 2000 km to the College [for the examination].
S3	This was great, it works great remotely
S4	I had a hard time interpreting a realistic picture of all impressions. It was hard to get the real feeling. Felt like I was talking all the time, and it was hard to feel the connection to the staff [firefighters].
S5	This is beyond my expectation. Interesting scenarios, the environment you built, giving orders works great [the firefighter avatars carry out the orders], and it feels like you are at the incident scene. This is the best substitute for being on-site.
S9	I was not comfortable in the situation. It is a good supplement, but I would have needed more real training [in LS before]. The scenarios were good, and I would have liked to train more times without the pressure of examination.
S14	This is more realistic than other methods for exercises.
S16	Overall, a great surprise. You do not have to pretend; all you see is what it is. Not like in the training ground.

.

One of the students, s4, comments on the cognitive difficulty of perceiving the represented situation. The comment “I had a hard time interpreting a realistic picture of all impressions” points towards experiencing a cognitive overload. However, this is the only negative comment on the cognitive part of the arranged examination. The second comment of the same student, “It was hard to feel the connection to the staff”, addresses interpersonal aspects of simulation training, which was not in focus in the present article. (One of the instructors, has also commented that “Leadership”, which is an interpersonal non-technical skill, is better taught in the training field, with real people).

After the final assessments of the five scenarios, a total of 4 h including breaks and feedback, the session was closed, and the students were asked to fill in the post-RVS questionnaire. The results are presented in

Table 4. Students’ response (n = 20) to the RVS examination. Answers in Likert scale, 1 (low) to 5 (very high).

Questions	% Answers Likert 4 or 5	Average (Scale 1–5)	Standard Deviation
Experienced presence, compared to previous very high?	70%	3.95	0.89
Experienced presence in the simulated environment?	75%	3.85	0.75
Experienced being in the same env. as the “persons” you met?	65%	3.60	0.99
How easy was it to understand the training objectives?	60%	3.80	0.89
How easy was it to communicate with others?	60%	3.80	1.01
Would you like to perform a similar training at your fire station?	90%	4.50	0.69
Would you like to perform a similar training in your spare time?	80%	4.28	0.95
To what extent do you consider RVS as a method for IC training?	75%	4.30	0.86

. The results show that 90% of the students (18 of 20) would like (Likert 4 or 5) to perform a similar RVS training again, at their fire station, while 10% of the students (two persons) responded: “Neither or” (Likert 3). In addition, 75% perceived RVS as a suitable form (Likert 4 or 5) for IC-training.

In previous VS sessions, the students used a gamepad to move in the virtual environment, while in the RVS, the arrow keys were used to move around. No student stated any obstacles related to use of the keyboard for movements. For the question “How easy was it to move in the environment?”, one student (5%) stated hard (Likert 2) and all the others stated no problems (Likert 3–5). In the previous VS study performed in 2019, 15% of the students expressed an unfamiliarity with the gamepad and considered it as an obstacle [66]. Consequently, using the keyboard was an improvement for some students.

Regarding the students’ experience of approaching and communicating with the firefighters (the instructors-controlled avatars), 60% stated that it was easy or very easy (Likert 4–5) and 10% stated hard (Likert 2). This suggests that future research should address the avatars’ lay-out and the communication between the IC-student and the avatars to a greater extent.

On the question: Please describe aspects that you found pleasant in the task, 50% answered that they appreciated the good counterplay, the voice acting done by instructors, which enhanced the sense of realism of the situations.

6.2. The RVS Examination—The Instructors’ Experiences

All five instructors conducting the RVS examination were convinced that the students were presented with similar challenges and performed similarly as in LS examinations. They also perceived the students’ movement in the virtual environment and their communication with the avatars as easy and unproblematic.

The instructors stated that they could trustfully assess the students based on the learning objectives. One instructor explained the values of the virtual environment as “Everything that relates to the situation assessment, the development of the incident, like the spread of the fire and the extent of the damage, is possible to include in the virtual environment, which makes it extremely effective for assessment” (i1). Only one answer was given to the question regarding whether there are course objectives that cannot be assessed in RVS. This instructor (i2) commented: “…leadership might not be optimal here [in RVS], you need to train [to assess] this with real people, physically so to speak, to be able to train the basics”.

A new possibility appreciated by the instructors was seeing the students’ faces and reactions through their facial expressions and always seeing what the students were looking at. This cannot be achieved in LS, where the instructor cannot be sure about what the student is looking at. An instructor explains this in the following way: “I see and hear the student all the time. I can more easily assess communication and the orders given. I can see the exact picture of what he is looking at... It can sometimes be difficult to determine what the student is focusing on in a live exercise in the field” (i2).

The advantage of playing roles through avatars for instructors is explained by one instructor in the following way: “To have the opportunity [as instructor or assessor] to play the IC-student’s personnel [firefighters] makes it possible to ask questions if orders are unclear. Also, later during the scenario, one can [with the firefighter avatar] walk up to the IC and ask a related question to assess to what extent he or she understands the situation at hand” (i1).

6.3. Cognitive Aspects in Simulation Training

The different simulation formats, LS, and (R)VS are compared in

Table 5. Higher cognitive processes are supported in LS and (R)VS.

Higher Cognitive Processes	LS	(R)VS
6.1 Recognition Here, focused on perceived visual realism of the incident site, as to buildings, vehicles, involved participants, flames, and smoke.	Buildings, built to stand several fires per day and to represent different real-world objects. Real firetrucks and equipment are used. Old cars are used to represent cars in accidents. Involved participants are real people, often students or retired people hired as actors. Fire, smoke, evolvement, cues, and risks and cues are limited, due to safety and environmental regulations. Changes in the situation are not supported. Recognition is partly supported based on the above representations.	Buildings, vehicles, involved participants, flames, and smoke are chosen from a database. Events to trigger or change fire and smoke behavior illustrate cues and risks that are preprogrammed or changed during the training session. Changes in the situation are supported. Recognition is supported based on the above representations.
6.2 Imagery	The perceived realism of the incident scene is based a lot on imagery. This is very much dependent on the instructor’s ability to describe the situation using the available method for training LS/(R)VS and individual experiences of the students. As we know the support for imagery is not included in training and assessment.
6.3 Comprehension The action or capability of understanding. Involves constructing and internal representations based on existing knowledge. IC-students do not have experiences from the IC-perspective in an incident, although they have experienced from incident scenes as firefighters.	Existing knowledge related to the scenario may be affected by the fact that the LS objects are used for several scenarios and are familiar to all IC-students who were previously firefighter students. Therefore, it can be based on the previous experience of training at the LS training ground (i.e., where the fire can/cannot be placed, what are the possible scenarios), and by the additional information provided verbally by instructors. The team of firefighter-students are familiar with the training ground, and may “help” the IC-student by not asking when orders are unclear or safety measures do not meet the scenario. Few instructors live-play the police, ambulance, or bystanders, all looking the same. Comprehension partly supported.	The virtual environments and object, buildings, and avatars are all new to the IC-students. The instructors play the firefighter, police, and bystanders, all with different avatars. This makes it possible to use avatars to ask questions or react if the IC-student gives an unclear order. Comprehension can be supported over a wider specter.
6.4 Learning Learning acquisition of knowledge and skills resulting in a upgrade of the cognitive model. Confirmation of existing knowledge or deeper understanding are also recognized as learning [69].	Active experimentation [70] is not supported since the situation cannot evolve dynamically and one has very few tries in the training ground. Initial scenario design must be followed. Procedural learning is supported. Learning cannot be supported for all learning objectives.	Active experimentation [70] is supported, since the situation can evolve, the scenario can be changed, and more scenarios can be played. This will enhance learning. Procedural learning is supported. Learning can be supported for several learning objectives.
6.8 Decision making the process of choosing a course of action based on the current situation and the available resources. Especially for ICs the decision making is based on the above-mentioned aspects of cognition	Decision making is supported by the available stimuli of LS and the above-mentioned aspects.	Decision making is supported based on a wider specter of stimuli and the above-mentioned aspects.

, out of the model of higher cognitive processes. The compared formats support different higher cognitive processes to a different extent. This highlights the complementarity of the methods, and may enhance the knowledge about benefits and limitations of each. Table 5 summarizes the findings in the study with the focus on how the higher cognitive processes are supported in LS and (R)VS.

The research hypothesis “RVS supports cognitive aspects of recognition, learning, decision making and problem solving for examining practical skills” has been confirmed. In the present study of the RVS examination, the students reported (Table 4) similar experiences as in Table 1 [66], showing results after introducing VS training on-site.

The second research hypothesis “Through RVS Examination, the instructors can assess student’s practical skills as ICs” was confirmed, as well. The pilot study (with eight highly experienced ICs from different FRSs acting as IC-students) and the following real RVS examination of 22 students, were evaluated positively by all five instructors, the eight experienced ICs and 18, out of 20 students, who participated in the research. The instructors/accessors commented on the performance of the students to be “average”, compared to the earlier LS examinations. Two of the students failed, which is “typical” for classes of this size. The instructors/assessors also assigned graded marks to the students (for the possible benefit of the research project, while the students received a Pass/No Pass result). However, this has not been done before in LS, so a detailed comparison of the students’ performance was not possible. Both students and instructors seem to agree that the cognitive aspects for training and assessing ICs are supported well by (R)VS, while interpersonal skills are better supported by the live settings.

7. Discussion

Successfully performing an RVS examination (during the COVID-19 pandemic) may trigger more RVS training and examination also after the pandemic. The RVS examination proved to be technically feasible in Sweden, with the lowest bandwidth of 30/30 Mbps [71]. The impact of training in virtual environments, and transfer to real settings is a research objective. Since we always offer the best training available to every responder, it is impossible to conduct research involving a “non-treatment” control group. Research during the police-student education, with the possibility of offering training to the control group after the research was completed [72,73], shows that student-groups who trained on the communication procedure with a helicopter (one group LS, one group VS, control-group only read manual with procedures) performed similarly independent of the simulation format, and better than the control group. Similar results arose also from a study of procedural learning on tank-maintenance procedures [74]. The two simulation formats gave similar results upon assessing the students in the physical realm, and better than no-simulation training. Hall [75] studied the effect of VS training on fire ground ICs decision making, out of their self-evaluation and perceived confidence, and Gillespie [76] studied the transfer of virtual knowledge to the physical environment, connected to the acceptance of the virtual training. The LS physical examination format is very well established. Some virtual (on-site) assessments have been reported [1,67], while the present study is, to our knowledge, the first remote IC-1 examination mentioned in the research literature.

Psychological and social variables, which may have affected the students and enhanced their positive attitude towards RVS (for example, a wish to comply with the researchers) [77,78], are considered less prevalent in the remote setting, compared to VS training on-site (which was evaluated equally positively in 2019—see Table 1). Additionally, in an examination setting, the students are focused on their own performance, since “it counts” to pass the exam. The seriousness of the situation was likely to provoke honest reactions on behalf of the students, as to the perceived quality of the arrangement.

The study demonstrates the necessary, likely minimum, steps of familiarization and technology implementation in emergency response training for successful implementation of RVS examinations.

• The technology had previously been used for VS training on-site. Thereby, existing technical, scenario design, and conduction competence saved time and guaranteed usability.
• It was possible to perform a pilot test with experienced ICs. The positive evaluation motivated the final decision to use the RVS examination.
• A key component was the competence and interest of one champion and support from experienced VS instructors who were assigned time to participate.

The experience of successful RVS examinations has motivated a broader implementation of RVS training and examination at MSB, and this can inspire other fire academies in taking similar steps. This may provide further opportunities to study the implementation process as a future contribution to the (R)VS literature. However, it takes time to develop skills to design, build, and run RVS exercises with high quality. There is a risk that the organizations do not understand the competence needed and therefore do not allocate enough resources for the instructors to deliver scenarios of sufficient quality, which could result in less acceptance of RVS. While a stricter investigation of possibilities and challenges regarding the potential value for remote examinations may need further investigation, indeed the present study demonstrated added benefits for remote training as a solution to be included in future education. We believe that the instructors/assessors experienced “being closer to the student” (despite the physical distances) since they could continuously see what the student was looking at and hear what the student said.

Sweden is a long country (1572 km), with several sparsely populated regions. The Swedish fire and rescue services personnel therefore consist of 67% part-time firefighters and ICs, i.e., with other regular jobs. MSB has only two colleges offering an IC education, which makes the student travel costs high and the time away from the regular job (for part-time IC-students) and family (for all students) long. Enhancing and developing distance education by performing RVS training and examination may therefore also represent societal, human, and environmental benefits. Performing the exam in the RVS format on average saved each student a round trip of 9 h by car, based on 768 km distance on average, i.e., a total of 15,400 km or 38% of the Earth’s circumference.

8. Conclusions

This study is an action case where the researchers participated in, and at the same time studied, the implementation of the VS training method and technology at the MSB, to the final step of conducting an RVS examination for IC-1 students. The cognitive aspects of recognition, learning, decision making, and problem solving were studied through questionnaires which the students filled post-training. The results indicate that the RVS, as implemented in the analyzed training and examination, adequately supported the above-mentioned cognitive aspects.

The existing VS implementation experiences at the MSB and corresponding studies of the students’ cognitive benefits were the enablers, building competence in the organization, and thus making the COVID-19-forced RVS examination possible within a short preparation time.. This study demonstrated a proof of concept developed under time pressure, and with the precondition that students should be able to use standard PC equipment to perform their IC-1 final examination remotely. It demonstrates the possibilities and current challenges of RVS examination in the Swedish IC education. The RVS examination was performed satisfactorily and experienced positively by all involved parties. The main values of RVS for the students was that they could in fact graduate and they saved the travelling time and time away from home and regular jobs. The RVS was recognized by IC-1 students, instructors, assessors, and the MSB management, as high-quality training and examination methods, that have recently been implemented in the education of IC commanders at all levels.