Arching from Function to Form—Important Design Elements of Simulation Exercises (SimEx) in Emergency Response and Disaster Risk Management

Abstract

With Rasmussen’s abstraction hierarchy as starting point, the present article focuses on understanding some of the aspects guiding the development of a simulation exercise (SimEx) from a multi-faceted perspective, based on interviews and post-exercise evaluations conducted with both exercise designers and participants. The results show that, in order to achieve its overarching objective, an exercise must fulfill a wide range of “functions”, which in turn can take various kinds of “forms” or actual representations in the physical world. The paper discusses a number of identified required functions of a SimEx, sometimes labeled as design elements, and furthermore elaborates on differences and specific requirements at form level, e.g., virtual vs. physical exercises.

1. Introduction

Simulation exercises (SimEx) are carried out in a diverse range of settings throughout the world, in order to support the development, readiness, and strengthening of disaster and emergency response systems and operational networks. A simulation can be defined as an “imitation of the operation of a real-world process or system over time” [1] (p. 3) while an exercise can be described as “a form of practice, training, monitoring or evaluation of capabilities, involving the description or simulation of an emergency, to which a described or simulated response is made” [2] (p. 4).

In order to facilitate the discussion in the current paper, the authors use the framework put forward by [3] Rasmussen (1985), in which he describes systems (or designed artefacts, such as a simulation exercise) in relation to a hierarchy of functional abstraction. In essence, any artefact can be described by its “overarching purpose or objective”, via a number of “functions” that the artefact must be able to perform in order to fulfill the overarching objective, to “form”, i.e., the actual representation of the artefact in the physical world. Different solutions at form level may have different pros and cons with regards to the required functions.

Deriving from Rasmussen’s hierarchy, the first criterion that one system/artefact must fulfill is to have a functional purpose, namely a purpose which it is being designed for. In the present case, simulation can have different purposes, depending on the level at which they are implemented; if the SimEx is aimed at individual level, its main purposes are “improving participants’ knowledge of emergency activities, policies and procedures and improving overall competence and confidence” [4] (p. 274), while at organizational level, the exercises are directed towards “safely test(ing) new plans and procedures before they are officially adopted, assess(ing) overall emergency response capability, enabl(ing) a process of continuous learning, if lessons are collected and linked to action plans” [5] (p. 4) (see more in the next section).

An important point of departure for the present paper is that conducting/implementing simulation exercises, regardless of context or purpose, should be viewed as a means to an end, in that they are aimed at supporting the overarching objectives of learning, identifying those gaps in preparedness which might have negative impacts on the organization, those systems which function well in order to take the lessons learnt on board for future development, as well as contributing to capacity building for the participants. However, the question is whether the improvements expected in terms of capacity development as a consequence of simulation exercises “persist overtime and translate into improved emergency response capabilities” [4] (p. 274). In this paper, the authors will identify and discuss a set of important design elements to be considered when designing and conducting simulation exercises.

The paper is seeking to contribute to the discussion on the aspects to be considered in the development of simulation exercises. The discussion is framed within the overall Disaster Risk Management (DRM) work and draws on recent experiences both before and during the COVID-19 pandemic, where remote exercises have been used to support operational readiness, capacity building, and system strengthening.

2. Rasmussen’s Abstraction Hierarchy in the Present Context

In the case of a SimEx, the purpose and objectives must always be clear from the start of the designing phase, in order for the design to accommodate all the needs and for the exercise to take the form which might bring the most useful results for the participants. The overarching objective of SimEX for emergency preparedness, response, and disaster management is viewed as enhancing the capacities and capabilities of mandated response agencies and networks, by ensuring improved coordination, cooperation, information sharing, and decision making. This includes compatibility and interoperability between functions and teams across different scales of operations. This type of exercise also tests and reflects the knowledge and experience gained from participation in related training and education programs, such as the European Union Civil Protection Mechanism (EUCPM) training course program or the United Nations Disaster Assessment and Coordination (UNDAC) mechanism trainings, but also training courses of local and/or national Emergency Management Authorities (LEMA/NEMA), as planned and conducted by Prepared International (PPI). In addition, these exercises can also be used for providing students and future practitioners with practical learning of the theoretical aspects of disaster risk management, as is the case at both Lund University (LU) and United Nations University Institute for Environment and Human Security (UNU-EHS) based in Bonn. When the exercises are aimed at students, their main aim is to deepen understanding of and skills in utilizing major preparedness tools such as contingency planning and simulation exercises.

In Rasmussen’s hierarchy, the abstract function is defined as “the intended causal structure of the process in terms of mass, energy, information, or value flows” [5] (p. 592). In the case of a SimEx, this refers to the process of transferring information to and building capacities of the participants, in order to develop their ability to respond to real-life situations mirrored in the simulation exercises. The difference between abstract and generalized functions lies in the fact that for the latter, one must look into the processes that ensure the functionality of the system, in this case, that the objectives of the SimEx are achieved. Some of the processes that can be included in the current discussion are setting the exercise objectives, identifying the players, choosing the type of exercise, defining the scenario, deciding upon the required logistics, and determining the evaluation and monitoring processes [6].

When it comes to a SimEx’s physical function(s), these can also be referred to as “design elements”, since they focus on its connection to reality and real-life situations, team-building aspects, and other elements which are necessary to fulfill the exercise’s overarching objective. These elements act as guidance for developing and planning the exercise. Normally, this type of exercise is organized based on a specific pre-identified need. Determining the exercise purpose and specific objectives is the step that guides all further planning steps of the exercise [6].

Last but not least, the physical form of the exercise is entirely based on the design elements agreed upon and derives from the spatial and environmental conditions which must be tested for their accurate mirroring of the real-life situation. The most important aspect to consider in deciding on the form of the exercise is that this simulates the actual disaster management conditions. Once the design elements are agreed upon, the “form” of the SimEx has to be discussed. This entails the exercise type (e.g., table-top, drill, functional, or full-scale), the exercise form (physical vs. virtual), and various learning approaches. Each of these aspects will guide the broader discussion on positive and negative aspects of each potential “form” solution with respect to the previously agreed design elements, “functions”.

The outbreak of the global COVID-19 pandemic, including the implementation of public health and social restrictive measures, along with severely limiting travel and quarantine restrictions, emphasized the need for flexibility and adaptability of SimEx and a request for a paradigm shift to virtual, digitally supported simulation exercises. This paradigm shift required new didactic learning approaches based on sophisticated, interactive information technologies, including enhanced concepts for creating and integrating audio-visual components, that ultimately provide participants with a variety of translocal learning spaces that together create a compelling training environment. In addition, the learning rhythm must be adapted to the mental receptive capacities in purely virtual simulation exercises. At the very moment, the number of COVID 19-cases is getting lower in certain geographic areas, and thus the authors have started seeing an increase in the number of so-called “hybrid exercises”, in which virtual and face-to-face components are mixed. Such exercises have the advantage that they mirror the real deployment situation today, where key parts of the response are, in fact, also in real cases online-based, such as coordination, information sharing, management decisions, and other tasks.

3. Previous Research

A literature review conducted by Beerens and Tehler (2016) [7] showed that there is a lack of academic interest in the field of disaster exercise evaluation even though its importance for disaster risk management is recognized and heavily emphasized by both academics and practitioners. Considering that the present paper understands conducting/implementing SimEx as a means to an end, the authors have tried to identify previous research on the overarching objectives of learning and capacity building of participants in these types of exercises. The framework which was used as the basis for the present literature review built upon one of the principles of learning presented by Corpuz and Salandanan [8]—learning seen as behavioral change is a consequence of experience. Thus, the authors of the current paper focused on understanding how simulation exercises are looked upon by research from a perspective of them being used as tools in building capacity (changing behaviors and informing disaster management abilities).

The authors have identified that most publications on the topic of exercises and risk management can be found in the medical sciences field, focusing on medical management evaluation, medical performance, and simulation exercises in health care for effective responses [9,10,11,12,13,14,15,16,17]. The same aspect has been underlined by the aforementioned literature review in which 132 publications out of 307 analyzed were classified under the subject areas “health”, “medicine”, and “nursing” [7].

However, the authors were able to identify some publications relevant to their research, related to the engagement of practitioners [18] and students [19,20] in disaster risk management and hazard perceptions. In addition, in the last couple of years, more research has been published on the impact of educational methods for knowledge transfer in disaster risk reduction, building either on the grounded theory approach [21] or the experiential learning theory [19,20,21,22]. Experiential learning is what simulation exercises focus on and as research has shown, these types of activities support students [23] and increase their attention by allowing them “to see a phenomena unfold…[be] able to personally manipulate and practice using that phenomena in a first-hand environment” [24]. Thus, for the methods involving students, the authors focused on practically (experientially) testing their ability to apply the theoretical knowledge gained during their studies and allowed them to evaluate the learning curves.

Knowledge building is also one of the main focuses of the Sendai Framework for Disaster Risk Reduction 2015–2030, being essential for understanding disaster risk and can be achieved by “sharing experiences, lessons learned, good practices and training and education on disaster risk reduction” [25] (p. 15). Thus, the present paper aims at understanding how these aspects can come together in different settings and look upon simulation exercises in emergency response and disaster risk management as key to achieve disaster education, understood as “a crosscutting issue to achieve the four priorities” [26] of the aforementioned framework.

Regardless of the publications identified, the need for greater clarity in supporting evidence for the usefulness of specific methods identified by Beerens and Tehler, 2016, is the aspect that the present paper aims to tackle by describing the most relevant design elements for a SimEx to be able to bring together a wide range of stakeholders who have different backgrounds, organizational frameworks, and experience levels. The discussion on design of exercises can also be found in the literature on crew management training in which researchers underline the need for a systemic view [27,28,29,30,31] and the impact of exercise design for long-term retention [32].

Furthermore, the importance of evaluating these exercises, their impact, and the functions required for them to reach their objectives and be shaped to fit the overall purpose is underlined with the hope of providing exercise designers with input and solutions at form level.

Study Limitations

This study has some potential limitations in regard to the sample size of the cases, their diversity, and the data collection methodology used. As it will be shown in the next section, the methodology is a mixture of behavioral and attitudinal research, aimed at describing the most important aspects to be included in a simulation exercise, regardless of whether it is a virtual or physical one, based on the participants and organizers’ experiences. Thus, the results of the study can be seen as limited in relation to the sample chosen and the interpretation of the data acquired. The authors believe that the conclusions might be applicable to other similar cases and for similar scopes as for the exercises evaluated, but they do not assume that the results are generally valid for all types of SimEx.

4. Materials and Methods

The study is based on recent experiences in developing, conducting, and participating in emergency preparedness and response simulation exercises in a number of different settings. With some exemptions, see case 3 below, the main input to the study is the experience of the authors in contributing to the design, execution, and evaluation of the presented cases.

According to Pahl and Wyles [33], “human perception and behaviour can be subject to systematic and rigorous scientific study, using theory-based hypothesis testing” (2017, p. 1405). Building upon the researchers’ idea that social and psychological concepts such as attitudes, responsibility, norms, and awareness of consequences are key factors in supporting information and knowledge provision, the authors of the present paper aimed at understanding how both the users’ (students’ and practitioners’) behaviors and perceptions influenced their ability to respond to the cases presented during the simulation exercises. These have been analyzed through informal post-exercise evaluations (for the students at Lund University), formal post-exercise evaluations (for the exercises conducted by Prepared International), and exercise observations during the play-out.

For the exercise conducted at Lund University, the behavioral experience was also complemented by attitudinal research when some of the students and facilitators were interviewed post-exercise (see more details below) in order to better understand their challenges and be provided with recommendations on improving the SimEx based on the experiences and perceptions of the participants.

The analysis of the information retrieved from the three cases is conducted by combining the results per most important aspects in the pre-, during, and post-exercise phases, depending on the information retrieved from each of the cases and those aspects that they could be analyzed for. The common patterns of the three cases are the basis of the Results section’s structure and they are aimed at highlighting the main results of the evaluations conducted for each of the three cases.

The cases from which experiences were drawn are:

• United Nations University Institute for Environment and Human Security (UNU-EHS), based on the:
  Annual scenario-based Simulation Exercise on emergency preparedness and crisis response, usually conducted at the German Federal Academy for Civil Protection and Civil Defense (BABZ), former Academy for Crisis Prevention, Emergency Planning, and Civil Protection (AKNZ) of the Federal Office of Civil Protection and Disaster Assistance (BBK). Due to the prevailing COVID pandemic at the end of 2020, this SimEx was translated into a digital version based on a Zoom virtual environment. The SimEx offers students a unique opportunity to test their capacities to respond as a team to a simulated disaster event. The exercise tests multiple functions of the group which is usually structured into different teams with varying responsibilities. It is a coordinated response to a situation in a time-pressured, realistic simulation that involves several agencies. The enhanced tabletop exercise focuses on the coordination, integration, and interaction of the group of students including its policies, procedures, roles, and responsibilities. Communication and coordination between the student teams and all external agencies, represented by members of the exercise control (ExCon) group, is of great significance. The simulation exercise takes place within a multi-hazard scenario of a large-scale (L3) disaster that heavily affected extensive areas in western and central Europe and;
  Official evaluation of EU Modules Table Top exercises, the dModTTX, that enabled the continuation of EU exercises even when the specific circumstances of the COVID pandemic hindered the physical presence of participants. The dModTTX is also intended as a complementary alternative to physical exercises, thus creating an added benefit. While the physical exercise framework for Module TTX is well-established and tested throughout a considerable number of prior exercises, significant additional assistance was sought to develop a corresponding virtual infrastructure and environment. The pioneering Digital Modules Table Top Exercises were funded by the European Commission and organized by a consortium formed by the German Federal Agency for Technical Relief (THW), the Ministry of the Interior of the Republic of Croatia (MUP), the Federal Public Service Home Affairs—Directorate General for Civil Protection, Belgium (FPS-DGCP), the Administration of the Republic of Slovenia for Civil Protection and Disaster Relief (URSZR), and the Danish Emergency Management Agency (DEMA).
• Prepared International (PPI)
  In order to discuss the aspects which can guide the development of a simulation exercise from the designer perspective, the authors have chosen to present two exercises designed, planned for, and conducted by PPI in the last two years. One of them is the 2019 Caribbean Coast Exercise, which was organized by the International Disaster Management Exercises (IDMEX) Consortium and included two Plugin-exercises (PIX) conducted in the Caribbean with the Royal Dutch Navy as main contributor. The other exercise focused on increasing the Preparedness for Emergency Response in the western region of Nepal, thus being a national simulation exercise. These exercises were chosen because of the diversity of scenarios they focused on—the Caribbean exercise focusing on hurricane and volcanic eruption, while the Nepali exercise was built up on an earthquake scenario. The two exercises referred to from PPI’s perspective were designed to test the general preparedness and response coordination mechanisms among all levels of governance—from local to national level—and stakeholders. In addition, they were aimed at testing, improving, and/or establishing the procedures, standard operation procedures (SOPs), and agreements necessary to incorporate capacity and activation response.
• Lund University, Division of Risk Management and Societal Safety, Master Programme in Disaster Risk Management and Climate Change Adaptation (LU-DRMCCA)
  This case is based on a course module on simulation exercises. In this module, students at the Master of Science program in Disaster Risk Management and Climate Change Adaptation are guided through the process of designing, running, and evaluating simulation exercises. The exercises are built around contingency plans previously developed by the students and are traditionally run with physical presence of all participants. The last two years, the module has been taught entirely online. The students get to experience both developing/facilitating exercises and participating in exercises.
  In addition to experiences of two of the authors as designers of the course module, qualitative interviews were held with four designers and participants (students take on the role of both designers and participants in the course module) of simulation exercises in the case above. The interviews were of a semi-structured nature with rather open questions with the objective of getting the informants’ views of what the most central design elements are when planning and conducting exercises. The interviews were all held online and lasted for about 45 min each. The interview notes were then reviewed and analyzed to identify key themes. Further on, the interview data were consolidated under themes and key messages finally incorporated into this paper. The main themes included in the analysis were design criteria—setup and schedule, facilitation and roles, technology and platforms for interaction, safe learning environment, and team work.

5. Results

Analyzing the notes from the interviews conducted at Lund University, the reports from the two exercises organized by PPI and the results of the scenario-based simulation exercises conducted at UNU-EHS, as well as the evaluation of the Digital Modules Table-Top Exercise (dModTTX) Pilot, interesting patterns were identified in regard to all stages of a simulation exercise. The findings are clustered based on Rasmussen’s hierarchy, while also taking into consideration the three main phases of a simulation exercise: pre-exercise, exercise implementation, and post-exercise, as it will be shown in the sections below.

5.1. Functional Purpose

In order to make sure that the purpose of the exercise is aligned with the participants’ expectations and behaviors, thus ensuring a common ground for setting up the exercise on, ground rules are considered to be required (Figure 1). They allow the participants to understand their roles within the teams and stimulate engagement of their peers during the actual exercise. In addition, according to the interviewees from Lund University, the initial discussion on ground rules has been seen as supporting the “small world” of the simulation exercises, by setting clear boundaries, communication processes, and common values that support the building of relationships and trust among the members of the facilitation team, as well as the exercise participant group. These ground rules create a social contract, building social cohesion and providing physiological safety, thus enhancing coordination, communication, and collaboration during the exercise.

However, the interviewees named essential differences between the ground rules necessary for on-site versus online exercises. Some interviewees highlighted the need for several breaks, more focus on members’ engagement, and group dynamics for the latter. In terms of the breaks, it was identified that scheduling more frequent exercise breaks is essential in keeping people engaged and energized, much more in the online versions of exercises than in the on-campus ones due to “Zoom fatigue” (see [35] for more information on the concept). The Zoom fatigue, inter alia, creates a barrier for the trainer(s) and exercise facilitators, who are often unable to assess the mood of the participants through the screen. In addition, UNU students felt that in virtual simulations, one may exhibit fatigue that may not relate to the exercise itself but to the extended screen time. This virtual miscommunication can create ambivalence among the group members and hamper the group dynamics. Due to limited opportunities to express themselves, non-native participants may feel insecure about their level of language. Some participants may also suffer from “video anxiety” triggered by the fear of speaking in front of the camera. Additionally, while exercise breaks in physical simulations refer to connecting over lunch or coffee breaks, in virtual simulations, breaks refer to the avoidance of the computer screen, thus limiting valuable exchanges.

In general, UNU students reported that sitting in the comfort zone of your familiar surrounding at home and attending a virtual emergency response simulation exercise is different and most likely more difficult than participating in a real physical simulation exercise. However, the training effect of both simulation types might be very similar. Advanced virtual SimEx, generating a real-world scenario based on sophisticated, interactive, information technologies and an appropriate combination of injects, role players, and a sophisticated overall setting provides participants with the same learning experience they would otherwise perceive in a physical SimEx. Additionally, the constant pressure put on participants to complete their tasks in due time and exposing participants to sudden, unexpected elements corresponds to the pressure that one would expect in a real field situation.

In addition, in order to make sure the participants stay engaged and collaborate towards reaching a common objective (the functional purpose described by Rasmussen), it is essential to focus on building group dynamics, ensuring that everyone feels included and can actively participate. This was a recurrent topic during the interviews conducted at Lund University, with the interviewees identifying the importance of the facilitation team monitoring this aspect, making sure the conversation is flowing, the teams are assisted if they experience problems, and ensuring that there is space for everyone to contribute to the work. During the feedback session of the UNU-EHS exercises, the issue of inclusion during virtual exercises emerged, being emphasized that differences in time zones may hinder the effectiveness and ability to participate of the international participants. This aspect has also been underlined by the interviewees from Lund University who identified that team building processes were much easier in face-to-face situations, while the online exercises hindered the capacity of the facilitation team to maintain everyone’s attention and engage all participants in the exercise play-out.

Based on the information shared by the interviewees, it seems to be essential to follow the steps below in order to facilitate the inclusion and participation of the target groups of the exercises:

• Share expectations and concerns at the very beginning—discuss them openly in the facilitation team, with the participants, and external observers (if any).
• If possible and appropriate for the training, rotate people in the key roles during the exercise sessions, e.g., team leader, communications officer, etc.
• Make sure a dialogue is possible within the facilitation team in order to overcome conflicting ideas and develop fruitful debates, which are aimed at improving the capacities of the participants and result in lessons learnt for the facilitation team.
• Ensure that the training language is adapted to the context and participants.

5.2. Abstract Function

This part of the abstraction hierarchy has been understood, in the present context, as referring to the processes of information-sharing and capacity-building, both aimed at building capacities of the participants to be able to react to real-life situations which are mirrored in the simulations. In order for these simulations to be as close to reality as possible, the inject plan and storyline (scenario) must be inspired from past events and based on both the existent capacities of the emergency responders tested, as well as the needs that normally emerge in similar situations. Risk scenarios aim at answering the question “What can happen?” and focus on determining “a chain of events that may lead to something that is undesirable according to the values/objectives that are used as the basis for the analysis” [36] (p. 8). Sometimes, if the planning of the exercise is done by external actors, they might choose to develop an initial draft of the scenario before departing to the exercise location, but finalize it and the injects on the spot, supported by observations of the locations where the exercise will be played out and conversations with the relevant stakeholders (e.g., PPI in Nepal—observation in Dang district and conversations with Red Cross representatives).

In order to ensure that the capacity-building process and the information shared and/or accumulated is emerging from a common ground of the participants, two essential elements of a simulation exercise come into play—briefing (Figure 2) and debriefing (Figure 3).

Regardless of if the simulation exercise takes place on-site or virtually, both the participants and facilitators the authors interviewed agreed that a clear briefing and introduction is crucial for a successful exercise. The main aspects which must be included in a briefing session are the presentation of a clear aim and objectives of the exercise, the day plan and schedule for the exercise, the areas and spaces which will be used during the exercise (if on-site), the ground rules to be followed by all participants, and an introductory round of the people in the room (either on-site or digitally) and their roles. The aim of this briefing is to effectively manage participant expectations, set clear boundaries, and ensure everyone knows how to engage with the exercise environment.

If the exercise is time-constrained, an exercise briefing note or video can be sent out to participants beforehand, where the background and procedures of the exercise are introduced step-by-step. Even though this infringes on the possibility of participants to ask questions and raise concerns, these can be addressed at the beginning of the exercise, when all participants and members of the facilitation team meet in plenary. This can also be the opportunity to allow the participants to familiarize with the virtual environment and the main functions embedded within the technology platform(s) to be used during the simulation. When it comes to students, the briefing session is also the best opportunity to link theory to practice (the exercise), by the means of a session describing which theories or knowledge previously studied are to be tested during the exercise. This will be recapped in the debrief session.

At the other end of the spectrum, there is the debriefing session which tends to be one of the most overlooked sessions in a simulation exercise with participants often choosing to skip it. However, it is of utmost importance and a critical part of a simulation exercise. The debrief aims to ensure lessons are identified, challenges and achievements are reflected upon, and recommendations linked to system strengthening, capacity development, or continuous learning come together. The debrief often reconnects participants with the planned “red thread” to reinforce the link between theory and practice. In the debrief session, the theory discussed in the beginning will be recapitulated and analyzed based on the recently conducted exercise perspective.

However, all interviewees agreed that conducting the debrief session online is much more difficult than for on-site exercises because it becomes even more complicated to engage participants in the discussions through a screen. In addition, the discussions do not flow as they might do on-site because you have to respect all the online environment etiquette, such as the “raise hand” rule. In addition, due to Zoom fatigue and a sense of physical disconnection from the others online, participants in virtual simulations sometimes struggle to engage in the new teams established, especially where relationships have not been established and no before or during the exercise sessions/activities were planned for in order to help the new team form.

From a facilitator’s perspective, it becomes extremely difficult to “read the room” in online settings and know how to adjust the session to the participants’ needs. This can become even more problematic if the facilitators are not well adapted to the technology and thus, might result in their work being restricted to previous and potentially limited knowledge and capacities. Regardless, it is essential to remember that the exercise is not over until the debrief is over. Just as any training, simulation exercises are part of a continuous capacity development and/or strengthening process, where the evaluation of the previous exercise and current training status feeds into the design and development of the exercise program.

5.3. Generalized Functions

As a complement to the abstract function, Rasmussen understands the generalized functions as the processes that ensure the functionality of the system. In the present case, this implies those aspects that contribute to reaching the purpose of the exercise. The authors consider that the most essential element in guaranteeing that the exercise fulfills its objectives is creating a safe learning environment. This is the physical or virtual space that allows for building knowledge and asking questions without fear of rejection or embarrassment. This is often referred to as psychological safety, which can be defined as a shared belief that the team is safe for interpersonal risk taking [38]. Creating a safe space for people and teams also means providing an environment to learn from their mistakes in a constructive way.

In looking at how to best create a safe learning environment, the authors have been able to identify a couple of aspects that are crucial in both online and on-site versions of exercises. These aspects refer to, inter alia, the need of the participants to be seen and heard, the need to identify a red thread—a continuous learning cycle, allocating a mentor or coach to participant groups in order to make sure questions do not remain unanswered, and turning technology from a barrier into a tool for communication.

For the latter, the UNU-EHS exercises showed that innovative information technologies, i.e., integrating audio-visual components, green screens authentic broadcasts are essential in creating very realistic training environments. However, as experience has shown, since virtual simulations rely heavily on technology, communication, electricity access, and Internet connectivity can affect the exercise play-out significantly. Failure in one or more of these components, or a poor Internet connection with corresponding transmission errors, is likely to discourage participants from contributing to their group work that would consequently demotivate other group members as well.

When it comes to the need for a continuous learning cycle, it is required that the structure of the exercise allows for debriefs, information-sharing, task performance feedback, reflections, or additional instructions to clarify learning outcomes. A safe learning environment also provides participants the opportunity to exercise not only those skills, functions, processes, tasks, or knowledge with which one is comfortable, confident, or highly competent in, but it is especially beneficial for those which one is unsure of and which the practical exercising helps to strengthen one’s personal capacity and make sure that they are equipped both physically and mentally for when and/or if the situation occurs in real life.

Repetition! Repetition! Repetition! We cannot overemphasize how important this aspect is! Repetition is one of the key benefits of simulation exercises, regardless of whether thinking about repeating tasks or testing new plans, procedures, or building skills and confidence.

There is another relevant aspect which the authors have identified as falling under the generalized functions category, as described by Rasmussen, and that is the evaluation phase, based on the improvement methodology. In large-scale exercises, the Quality Management and Evaluation Cell is the one supporting the ExCon in guaranteeing a high quality in the implementation of the exercise by, inter alia, providing feedback on possible needed adjustments to other cells. This cell is also responsible for the self-evaluation process.

In the case of PPI’s exercises, the evaluation phase consisted of a formal exercise evaluation, an integrated analysis, and an evaluation report that identified strengths and areas for improvement in the participant’s preparedness, as observed during the exercise. The underlying analytical framework incorporates key performance indicators including interoperability, coordination, response activity, SOPs, and/or communication and reporting. Based on the evaluation results, recommendations related to areas for improvement were identified to help develop corrective actions to be tracked throughout the improvement planning phase. This also includes the opportunity for participants to provide dynamic feedback on the exercise. An evaluation process should be focused on key performance indicators, which can include interoperability of teams, coordination and cooperation among relevant stakeholders, response activity, communication and reporting among actors, safety and security, self-sufficiency, application of SOPs into practice, etc.

Any exercise evaluation will have two main components: the evaluation of the exercise, the process of designing the exercise, and integration needs and training objectives of every stakeholder, the execution, and lessons to be learned in the process, but also the actual performance of the participants. Therefore, the focus is both on the staff organizing the exercise project as well as on the participants themselves. It is highly recommended to use both internal and external evaluators [7].

To train all facets of the disaster risk management cycle and strengthen operational, managerial, personal, and professional skills, a variety of SimEX forms will be needed. Currently, a core problem is the fact that civil protection and disaster management organizations are missing not only the ability to exercise in different forms but also to evaluate exercises in a way that would inform the next exercise cycle so that it leads to an enhancement of capacity and fills the existent gaps. Generally, each simulation exercise provides room for mutual learning through individual practical performances and joint debriefing procedures. In both stages, mentors and evaluators play a vital role. Unlike physical simulations, virtual environments allow for numerous evaluators, mentors, or observers to be chosen for assessing and providing feedback regarding the performances of participants and the effectiveness of the simulation exercise. Teams can also compare their performances amongst each other.

The evaluation phase (Figure 4) informs the successful achievement of the exercise purpose and objectives, often leading to an action plan or to the development of future exercises or next steps in the exercise program. It is also the part of the SimEx where it can be decided whether the exercise could engage the participants in an appropriately challenging exercise environment, built upon real life situations, scenarios, and existent and desired capabilities.

5.4. Physical Functions

The design elements are extremely important in preparing a SimEx, but they could not be identified without a clear exercise purpose and specific objectives, since the two are the elements guiding the planning of the exercise. Thus, a comprehensive scoping process must be conducted, aimed at identifying and agreeing on the purpose, objectives, outcomes, audience, and exercise context. When this process is finished, some of the physical functions can be derived from its results. They are extremely diverse and can range from establishing clear communication lines within the facilitation team, but also among the facilitation team and the participants, the story-telling process, including role-players, etc. When it comes to the communication lines, they are controlled by the Exercise Control (ExCon) cell (Figure 5), which can be smaller or larger depending on the size of the exercise. This cell includes individuals who are assigned administrative and logistical support tasks during the exercise and are overseeing the implementation of the exercise, and in practitioners’ exercises, dealing with media and media requests, supporting the handling of VIP visits, and are responsible for the safety and security of all participants and staff.

In large-scale exercises, the ExCon is supported by the Exercise Organization and Situation Cell where those overseeing the injects and role-player cells come together to ensure a smooth cooperation between the two. It is the cell that supports the ExCon in keeping abreast of the real-time situation development.

In addition to the two cells, the communication between the ExCon and the participants is facilitated by the Main Trainer, who is the one taking training-related decisions directly or in coordination with the ExCon and strongly interacts with the Exercise Organization and Situation Cell and the Quality Management and Evaluation Cell.

Last, but not least, the Facilitation Cell is the one setting up and operating the exercise site, as well as managing exercise play. The facilitators are the ones providing key data to the participants and role-players, as well as those behind prompting and/or initiating certain participant actions to ensure exercise continuity. Under this cell, in large exercises, an Inject Cell might be included to guarantee the timely flow of injects with the right kind of media and/or channel used, based on a strong cooperation with the Main Trainer and ExCon to identify the exact timings needed.

However, this structure is mainly used in functional exercises, while in exercises prepared and played-out by students (as it is the case at Lund University), even though an ExCon exists, the team works closely together and is not split into cells but rather allows for the members to take on different functions during the exercise, e.g., controlling the exercise, delivering injects, managing the communication link (via phone, social media, etc.), observing the participants and taking on different characters as role players.

Regardless of its setup, each simulation exercise requires the development of a structured timeline pre-exercise, in order to support the exercise activities and make sure injects and interventions are delivered. However, both on-site and virtual exercises require flexible deliverable injects and availability of additional tasks or narrative elements, which can be used in case Plan A does not work as desired. It is not only the technicalities of the preparations and delivery that are important when it comes to the play-out of the exercise, but also the social cohesion of participants and organizing team. When the exercises take place on-site, social events can be arranged to make sure the members get to know each other and an atmosphere of trusting exchange is created. Social momentum and the bonding effect of non-virtual environments cannot be replaced by any virtual solution. However, solutions can be found to ensure a social cohesion is created, as the UNU-EHS exercises have shown where packages with materials required for the joint social events were sent to the participants and facilitation team members. Experiences have shown that this fosters cooperation and understanding between all individuals of the exercise.

For the injects’ delivery and ad hoc injects creation, the role players come into the picture. These are the skilled and well-prepared persons that act a specific role during the exercise and thus, they must receive clear instructions on what is expected from them and in large exercises, they can be supported by the Role Player Cell.

It is extremely important to take the time to introduce all the participants, roles, and characters at the beginning of the exercise (exceptions make those roles and characters who are to be discovered throughout the exercise by the means of injects and similar). During the feedback session of the UNU-EHS exercises, it has been concluded that role playing was one of the steering elements of the exercise and that the virtual tools enabled the involvement of role players from a broader geographic context, thus enhancing the feelings of participation and inclusion.

5.5. Physical Form

Last but not least, according to Rasmussen (1985), one must consider the physical form that can fulfill all the functions previously described. In the present case, aside from deciding on which type of simulation exercise to focus on, one must also look into the realism of the exercised scenario. Most of the interviewees identified the realism of the tools, interactions, and injects as crucial in supporting the learning process and very useful for the practitioners to be able to understand and identify risk pre-development. This can be due to the fact that participants identify how and where standards and codes can have a real impact on the risk environment, and thus manage to adapt their strategies accordingly. Interviewees also emphasized the importance of replication of real-world systems such as multiple communication channels, in order to enhance the level of realism and complexity.

The level of realism often helps participants “buy into” the exercise, thus allowing them to suspend reality and fully engage in the play-out. It is recommended that props and artefacts for role-players are used both for the on-site and online versions of the simulation exercises to manage to bring the roles to life. However, while on-site simulations scenarios bring participants into a joint formal setting, without external disturbances to replicate realistic disaster conditions, this is much more difficult in a virtual SimEx. During the UNU-EHS exercises, it was noticed that for people participating virtually from their homes, a sense of “forced realism” might be induced by disturbances such as household members at home, lack of dedicated workspace, care of children, noisy environment, etc. In contrast, physical simulation exercises are usually conducted in separate locations, with the requirement that participants remain there for the duration of the exercise to avoid distractions from returning back to familiar (home) conditions.

However, it is important to remember that for the scenario itself, it should always be the function and specific exercise objectives guiding it. This means that realistic scenarios tend to often be used by organizations in order to avoid overwhelming themselves or placing themselves in situations which they cannot deal with. It is thus encouraged to focus on capacity development as the guiding feature in developing the exercise scenarios, rather than the realistic aspect of it.

5.6. Reflections

In order to understand the behaviors and attitudes of the participants, but also to be able to build their capacities to respond to disasters in real-life situations, it is essential to establish ground rules of the SimEx which, regardless of the form of the exercise—virtual or physical—are focused on building trust relationships between the participants and ensuring that all of them start with the same mindset in approaching the scenario, as well as feel included and that they can contribute and fully participate to the exercise. Thus, it is required that expectations and concerns are put forward through open dialogue and in student exercises, key roles are attributed on a temporary basis, allowing the participants to test different positions in the response teams. The briefing is key in linking theory to practice, bringing all participants in line, and managing expectations and boundaries.

In order to analyze people’s behaviors, build their capacities, and allow them to manifest naturally, one must create a safe learning environment, regardless if the exercise takes place online or in-person. For this, a continuous learning cycle must be ensured, by allowing open dialogue, no questions going unanswered, and making sure aspects learned and skills developed are regularly repeated. By the means of psychological safety created by the learning environment, participants can develop their skills in an environment that mirrors the real-life situation while at the same time focusing on training and education. It is crucial for the success of an exercise to make sure the exercised scenario is as realistic as possible, with injects and role players included throughout the entire play-out. In addition, the communication lines must be functional and picture the existent organizational structures.

Since the present paper views SimEx as means to an end, it is essential to have a start (briefing) and an end (debriefing) of the procedure itself. The latter is too often forgotten or rushed because participants are exhausted and/or not willing to share their experiences and challenges in plenary. However, this is the only chance for the organizers to directly retrieve lessons learned and receive unfiltered recommendations for system strengthening and capacity building. While the evaluation will come later in either a qualitative or quantitative research form, informing both about the exercise itself and the participants’ performance, the hot wash-up right after the exercise can then no longer be captured, as in the debrief itself. Therefore, in order to ensure a continuous learning cycle, all parts mentioned in the three sections: pre-, during, and post-exercise are key and should not be left out, regardless of the exercise format and platforms used for performring it.

6. Discussion

Based on the results, the authors have tried to identify what the future development of simulation exercises could be. The authors have taken on board some of the lessons learned from the exercises we conducted both before and during the pandemic and have identified a couple of emergent aspects, as presented in the following sections.

The world is currently facing an unprecedented level of extreme events impacting people and nature, clearly evident in the ever-increasing frequency of severe weather events, epidemics, and human-made disasters. In 2020/2021, the world witnessed a number of record-breaking disasters demonstrating the underlying interconnectedness and complexity [39]. The increasing uncertainty with regard to emerging multi-hazard events, compound and cascading disasters, or complex emergencies requires well-suited, adequate preparedness and response structures able to anticipate the complexities of indistinct future risks. This situation also encourages additional momentum for convergence and consistency. In this respect, scenario-based simulation exercises possess a central position to strategically strengthen “collective swarm intelligence” amongst responders. Especially large-scale, fully fledged simulation exercises such as TRIPLEX 2016, clearly demonstrated the purpose and potential of disaster management exercises to substantially improve existing weaknesses and enhance gaps within international preparedness and response structures.

Experiences of the annual UNU-EHS scenario-based simulation exercise on emergency preparedness and crisis response, conducted at the German Federal Academy for Civil Protection and Civil Defense, underline the high value of experiential learning for participants, while at the same time illustrating unexploited testing and training opportunities regarding the interoperability of multilateral responders. In addition, the joint training experience supports trust building and interpersonal relationships between participants, which are most valuable assets during subsequent real mission deployments into the chaotic disaster environment. In general, intensive networking taking place during such simulation exercises has the capacity to reduce inter-agency barriers. However, more scientific research is needed to capture the concrete extent to which social network contacts built during simulation exercises among participants can improve implementation challenges and humanitarian coordination during international disaster relief operations.

The last year and a half has taught us that virtual environments will become more and more recurrent in the future and virtual simulation exercises stand no exception from this. This means that we will need to become better at building and running virtual exercises, while at the same time understanding that they will not entirely replace on-site exercises and their efficiency.

The future development of virtual exercises requires much more focus on and time for team building of the facilitation team, as well as familiarization with the digital tools. In addition, a larger team might need to be considered in order to ensure coordination and the smooth play-out of the exercise. The future of simulation exercises will also require individually focused exercises which focus on the participants’ own pace of participation and the possibility to re-run or/and pause different aspects of the exercise, based on the needs and capacities of the participant in question. While the exercises should be participant-focused and -led, the scenarios should be balanced in terms of innovation and realism, as well as context-based, in order to ensure that they are suitable for the certification/scope/aim/objectives that the exercises are designed for.

Recent experience of the interviewees and practitioners who contributed to this article showed that more than one facilitator is needed to engage, support, and manage a simulation exercise. However, this aspect becomes even more important in the case of virtual exercises when facilitators might not be in the same location to coordinate adjustments to the exercise on the run. The role of the facilitator has been emphasized in relation to ensuring that the exercises’ form meets the function. This includes delivering on the aims and objectives, and adjusting the simulation setup, timing, and injects to the participants during the implementation phase.

In the case of the Lund University’s exercises, the students are trained to maintain a strong overview of the exercise, ensure continuous feedback, and review the planned injects and timing, based on their fellow classmates’ engagement and response to the exercise environment. In terms of the skills required and/or developed by wearing this adaptable “hat”, it is worth mentioning that the students emphasized the importance of communication and observation skills as well as the ability to adapt, e.g., the exercise plan, on the run to any changes that might emerge in the environment and/or based on participants’ reactions.

Therefore, it is extremely important to organize introductory sessions to the exercise, especially when it is conducted online on how to use the platform and technology at hand, including, e.g., shared documents, group work, shared virtual white-board function, etc. The participants argued that it is very important to be able to use virtual tools for SimEx during the pandemic, thus not having to interrupt the learning experience. The online exercises require, as previously mentioned, that more “health” breaks are planned for (away from the screen), but also that each task is given a longer time to be performed than in on-site simulations, because people might need to move between virtual spaces, change roles, exchange information, or similar, and these activities require more time than in on-site simulations.

Even though it is clear that virtual simulations will not be able to entirely replace the on-site exercises, they have a lot of benefits which can be integrated in the so-called “hybrid” exercises, which might become the most suitable alternative to future simulation exercises, guaranteeing broader participation and inclusion. Among the advantages identified by our interviewees and the experiences the authors have gathered from the exercises, the creative aspect of the virtual exercises was heavily emphasized upon. This refers to the fact that the technologies and creative solutions used in virtual SimEx, e.g., 3D animations, visual and audio aids, etc.m can add realism and enhance the exercise experience in disaster response training.

As a result of the remoteness of the virtual exercises, they appear to be more inclusive, allowing international and interagency participants to join, thus fostering a larger diversity of ideas and knowledge exchange.

The main goal of the EU Pilot dModTTX SimEx, as funded by the European Commission, is to enhance the coordination of civil protection assistance interventions by ensuring improved compatibility and interoperability between the intervention teams and other intervention support as well as by developing the competence of the experts involved. Such virtual exercises also test and reflect the knowledge and experience gained from participation in the Union Civil Protection Mechanism physical training course program. The exercises are participant-focused and -led, developed from a number of innovative but realistic scenarios of international deployments suitable for exercising in a safe environment, suitable for undertaking certification or re-classification of registered civil protection modules/TAST, other capacities of the voluntary pool, and other capacities registered in CECIS and the EUCPT Mechanism.

Lessons from the pilot dModTTX SimEx clearly showed a very successful transition from physical to virtual exercising, incorporating new didactic learning approaches based on sophisticated, interactive information technologies, including enhanced concepts for creating and integrating audio-visual components that ultimately provide participants with a realistic training environment. Moreover, the learning rhythm has been adapted to the mental receptive capacities in purely virtual simulation exercises. An ultimate proof of concept was delivered by the fact that this dModTTX was the first digital Tabletop Exercise that allowed for the official certification of one of the participating modules in frame of the European Union Civil Protection Mechanism. Similar virtual exercises have a lot of potential, both as development tools, but also as a format that might result in both environmental and fiscal savings.

This was also evident in the experiences had by LU, where the in-class simulation facilitator training was transitioned to a remote online learning course, with a practical virtual exercise component. The students adapted well to the new setup and many of the lessons mentioned above (learnt from one batch’s feedback) were piloted and incorporated into the new workshop design and practical training sessions (applied to the next generation/batch of students).

Regardless of whether the SimEx takes place virtually or on-site, it should be seen as a shared learning tool, aimed at providing more inclusion and recognizing the diversity of participants, in terms of both needs and capabilities.

In addition, it is essential to decide upon the focus area and the target persons to be trained. The thematic focus guides key questions around the form of the exercise, since each thematic area entails specific ways to be trained and influences decisions regarding the way the exercises are to be developed, designed, and prepared for.

An example of the importance of choosing the right thematic area for the best outcomes of the exercise can be seen in the differences of skills and capacities tested during Host Nation Support (HNS) training versus Urban Search and Rescue (USAR) exercises. This is due to, inter alia, the fact that they have different objectives—HNS entails operational preparedness to request, receive, and coordinate international response actions, most often including or centered around civil military cooperation, while USAR focuses on the localization, retrieval, and stabilization of people spilled and trapped in urban areas. Thus, the capacities required and the knowledge that should be built, largely differ according to the training scope.

Furthermore, the level of capabilities and skills of the persons or staff to be exercised (operational, leadership, support, etc.), the organizational structures and practices (peer to peer leadership, operational teams deployed, etc.), the organizational culture, and the different personalities of the participants must be taken into consideration to ensure the best fit between design and purpose. The organizational culture aspect can expand to previous methods of training used in the organization and the ways in which capacity development processes have been conducted, since these are extremely relevant in designing the exercise.

7. Conclusions

Simulation exercises continue to evolve, the emphasis currently being on understanding, recognizing, and incorporating the softer skills—team building, creating continuous feedback, and learning. In light of most current events and developments, where, for instance, in 2020 two major planetary emergencies collided: climate change/global warming and the COVID-19 pandemic, the recent devastating floods across several countries in Europe in 2021, cascading earthquake, volcano, and hurricanes disasters in the Caribbean, it is expected that stronger focus will be given to risk assessment, preventive management, and preparation. This includes the use of simulations for responders’ training, system strengthening, and lessons learned in implementation mechanisms.

However, exercises are just a part of the overall disaster risk management cycle. They need to be shaped to fit the overall purpose. A considerable investment in recovery and system strengthening is required to ensure the way forward, with a strong emphasis on adequate and advanced capacity development. This includes but is not limited to the use of simulations and debriefing techniques in a large range of contexts, furthering the development of facilitators, regardless of if it is within governments, private sectors, communities, and/or other societal resilience stakeholders. In order to make sure a long-lasting improvement is achieved and that simulations become part of the training culture, simulation and exercise skills must be embedded within national agencies, response teams, and future practitioners (e.g., students). This will support the creation of a sustainable exercise culture, which no longer relies entirely on external experts and consultants.

Exercises are designed to consider the participation and contribution of all relevant stakeholders, to be inclusive, and to address vulnerable groups’ needs. They can be aimed at, inter alia, testing the general preparedness and response coordination mechanisms among all levels of governance—from local to national level—and stakeholders; testing, improving, and/or establishing SOPs and agreements necessary to incorporate capacity and activation response.

On-site exercises have, by no means, reached their end. The evolution and preponderance of virtual and hybrid exercises should be seen as a useful addition to the toolbox for capacity development. Insights gained during the COVID-19 pandemic should be seen as a means of transforming an exclusive, seldom used way of training and building capacity into a widely available digital blueprint to support the DRM sector. This means that it is now more important than ever to clearly understand the design principles needed to implement a successful simulation exercise, according to the “Form following Function” principle.

In this paper, the authors have highlighted a number of identified functions a SimEx needs to be able to perform in order to reach its objectives, and furthermore discussed advantages and challenges of various solutions at form level, e.g., between physical and virtual exercises. It is our hope that this presentation might serve as input to designers of exercises in real-world, virtual, and hybrid domains in the future.