A Virtual Reality Game-Based Intervention to Enhance Stress Mindset and Performance among Firefighting Trainees from the Singapore Civil Defence Force (SCDF)

Abstract

This research paper investigates the effectiveness of a virtual reality (VR) game-based intervention using real-time biofeedback for stress management and performance among fire-fighting trainees from the Singapore Civil Defence Force (SCDF). Forty-seven trainees were enrolled in this study and randomly assigned into three groups: control, placebo, and intervention. The participants’ physiological responses, psychological responses, and training performances were evaluated during specific times over the standard 22-week training regimen. Participants from the control and placebo groups showed a similar overall perceived stress profile, with an initial increase in the early stages that was subsequently maintained over the remaining training period. Participants from the intervention group had a significantly lower level of perceived stress compared to the control and placebo groups, and their stress-is-enhancing mindset was significantly increased before the game in week 12 compared to week 3. Cortisol levels remained comparable between pre-game and post-game for the placebo group at week 12, but there was a significant reduction in cortisol levels post-game in comparison to pre-game for the intervention group. The biofeedback data as a measurement of root mean square of successive differences (RMSSD) during the gameplay were also significantly increased at week 12 when compared to week 3. Notably, the intervention group had a significant improvement in the final exercise assessment when compared to the control based on the participants’ role as duty officers. In conclusion, a VR game-based intervention with real-time biofeedback shows promise as an engaging and effective way of training firefighting trainees to enhance their stress mindset and reduce their perceived stress, which may enable them to perform better in the daily emergencies that they respond to.

1. Introduction

Stress is an unavoidable aspect of any high-stress job, and the ability to manage it effectively is crucial for the well-being and performance of individuals in such roles. Stress management interventions have proven to be effective in mitigating the negative effects of stress [1]. Different stress management techniques have been used to tackle stress in the roles of first responders to emergencies, which include police, firefighters, and paramedics. This has contributed to evolving research into structured interventions that aim to improve cognition and stress reduction for first responders [2,3,4,5]. Relaxation techniques, such as progressive muscle relaxation [6], breathing techniques [7], psycho-education, mental imaging [8], programs to increase resilience [9,10], and training in social skills, have been used and have had notable success in coping with the adverse repercussions of stress associated with the job [11]. Moreover, mindfulness training for first response officers has been found to be associated with a reduction in organizational and operational stress levels, anger, fatigue, and sleep disorders, as well as an improvement in self-confidence, resilience, and general mental and physical health [12,13]. Nevertheless, a meta-review [14] of 29 studies also concluded that firefighters experienced several psychosocial stressors that were associated with a variety of health-related outcomes, including depression or mental health-related problems, burnout, sleep quality, and alcohol abuse. Besides firefighters, a meta-analysis [15] of 12 stress management interventions in police forces underlined that current interventions were not sufficiently effective, suggesting the need for rigorous studies that focus on the specific stressors that officers confront.

Training approaches often overlook the stress inherent in crisis situations. Preparing first responders emotionally for difficult situations could improve both short-term coping and long-term well-being. The Singapore Civil Defence Force (SCDF) plays a crucial role in delivering emergency response and public safety services throughout Singapore. SCDF personnel frequently encounter emergency situations that may evoke intense negative emotions, which are associated with stress [16,17]. Therefore, there is a pronounced need for effective methods that focus on emotion-regulation skills [18]. While various stress management interventions, such as training programs and counseling, have been implemented to support SCDF officers in coping with stress, there was a growing recognition of the need to explore novel and more quantitatively effective prescribed stress management training. Teaching psychological coping skills via lectures and role plays may not be adequate to lead to the effective use of techniques, especially in stressful situations [19,20,21]. Practicing emotion management skills requires inducing stress, which may be challenging to recreate in didactic settings or role plays in a briefing room.

This paper is dedicated to a specific approach to stress management involving the use of serious games. Serious games are defined as interactive applications with a designated purpose that leverages motivation, engagement, and enjoyment to facilitate knowledge gain, skills acquisition, or attitude change [21,22]. Virtual environments are increasingly employed to train professionals in performance and decision-making under realistic and stressful conditions. About 33% of firefighter injuries arose from exposure to fire, suggesting that these injuries could be mitigated by training firefighters to make better decisions, especially under stress [19]. Keech et al. applied Bayesian path analysis with informative priors and revealed indirect effects of stress mindsets on psychological well-being and perceived stress through proactive coping behaviors and perceived somatic symptoms [20]. In virtual reality (VR) serious games, participants can experience crisis situations without harm to themselves or others, yet in a highly realistic manner that may induce stress levels comparable to real crises. One example was MediSim, which was created to provide training in emergency triage for personnel on the battlefield [23].

Kana VR (previously known as “Stressjam”) is a serious game with VR designed to train participants to improve their stress mindset through real-time biofeedback collected while playing the game [24]. The tasks in the game are specifically designed to challenge participants to manage their stress levels, as indicated by the biofeedback sensor attached to their chest [24]. Jerčić and Sundstedt highlighted the successful performance of individuals who improved their emotional regulation through biofeedback in serious games [24]. Specifically, they reported that individuals who were aware of emotions and increased their skill of emotional regulation were successful in their ability to control arousal and showed better performances, reaction time, and attention in decision-making tasks in the serious game. In a study by Maarsingh et al., the positive effects and attractiveness of Kana VR for improving a stress mindset were shown [25]. The combination of biofeedback with immersive digital games offers many advantages [25,26]. First, in a situation of stress, a person’s attention is rarely focused on the physiological and psychological effects of stress. Trainees would benefit from being continuously informed of their level of arousal while playing a highly captivating game so they could learn to better detect signs of stress and use the appropriate coping strategy. Second, biofeedback in the game may increase engagement and perceived self-efficacy when stress management skills are mastered [26].

Our proof-of-concept study explored the use of a VR serious game with biofeedback such as Kana VR to enhance the stress mindset, which may potentially lead to enhanced training performance. The concept of stress mindset played a pivotal role in understanding how one’s attitude towards stress could impact their stress response [27]. According to the stress mindset measure, there are two types of stress mindsets: a stress-is-enhancing mindset and a stress-is-debilitating mindset. Individuals with a stress-is-enhancing mindset believe that stress can be beneficial and can lead to personal growth and development. They tend to view stress as a challenge that can be overcome, rather than a threat that must be avoided. On the other hand, individuals with a stress-is-debilitating mindset tend to view stress as a negative experience that is harmful to their health and well-being. They may feel overwhelmed and helpless when faced with stressful situations. This concept led to the development and validation of the stress mindset measure (SMM); a tool used to assess stress mindset as a distinct variable. The SMM has been shown to provide an additional perspective and method for understanding individuals’ stress responses. Individuals with a stress-is-enhancing mindset have been found to have lower levels of cortisol, lower levels of inflammation, and better cardiovascular health than individuals with a stress-is-debilitating mindset. Interventions that aim to change an individual’s stress mindset have been found to be effective in reducing stress and improving health outcomes [28].

Therefore, this study aims to provide an understanding of stress management among SCDF firefighting trainees and the potential impact of VR-based serious games intervention. Specifically, we utilized the SMM tool to evaluate stress mindset among SCDF trainees pre- and post-intervention, in addition to in-game performance, perceived stress, real-time biofeedback, and cortisol levels [29,30]. We hypothesized that personnel who have received this training will be actively engaged in the training delivered to them as part of their regular training program, and they would be better able to control their physiological stress responses as measured by heart rate variability and cortisol levels. If successful, this training method would be an engaging and effective way of training firefighting trainees to self-regulate their management of stress in a safe environment, which could potentially allow them to perform better in the daily emergencies to which they are expected to respond.

2. Material and Methods

2.1. IRB Approval

This study was conducted in accordance with the ethical approval HBR-21-00438: A Preliminary Evaluation Using a Novel Virtual Reality Game with Biofeedback for Training the Stress Mindset from the Singapore University of Technology and Design (SUTD).

2.2. Experimental Design

We recruited 63 healthy SCDF trainees undergoing the 22-week Section Commander Course (SCC) and assigned them randomly into three groups: control, placebo, and intervention. Sixteen participants were eventually excluded from the study due to incomplete responses and/or not completing the course.

Table 1. Summary of study groups and their respective psychological, physiological, and training performance.

Group	VR Game-Based Training	Psychological Response Measurements	Physiological Response Measurements	Training Performance Measurements
Control	No	PSS	NIL	BAPT, Final EX
Placebo	Yes (without biofeedback)	PSS	Cortisol levels	BAPT, Final EX
Intervention	Yes (with biofeedback)	PSS, SMM	HRV, Cortisol levels	BAPT, Final EX

summarizes the composition of each of the study groups, while Figure 1a illustrates the experimental design with the timeline of various measurements. The remaining 47 participants completed the same SCC training and assessment. The control group received no game-based training, the placebo group played Kana VR but without real-time biofeedback, and the intervention group played Kana VR with real-time biofeedback. Examples of the participants engaging in the VR gameplay and screenshots of the VR gameplay are shown in Figure 1b–d. The duration of the Kana VR gameplay is about 40 min, and it is administered at week 3 and week 12 of the 22-week SCC. Physiological responses were measured via heart rate variability (HRV) using sensors worn over the participants’ chests and cortisol levels from saliva samples. Psychological response measurements were recorded using the Perceived Stress Scale (PSS) and Stress Mindset Measure (SMM). The participants’ training performance was evaluated using the Breathing Apparatus Proficiency Test (BAPT) and Final Exercise (Final EX) assessment.

2.3. Psychological Response Measurements

2.3.1. Perceived Stress Scale (PSS)

Throughout the 22-week training regimen, participants were given a survey form to assess PSS, which is a 10-item questionnaire designed to measure an individual’s current levels of experienced stress over the past month [31]. Each item in the PSS is scored on a Likert scale (0 = Never, 4 = very often) (

Table 2. Example of 10-item survey questions on perceived stress scale, which is administered monthly to the participants during the course.

10-Item Questions	Likert Scale (0 = Never, 4 = Very Often)
In the last month, how often have you been upset because of something that happened unexpectedly?
In the last month, how often have you felt that you were unable to control the important things in your life?
In the last month, how often have you felt nervous and “stressed”?
In the last month, how often have you felt confident about your ability to handle your personal problems?
In the last month, how often have you felt that things were going your way?
In the last month, how often have you found that you could not cope with all the things that you had to do?
In the last month, how often have you been able to control irritations in your life?
In the last month, how often have you felt that you were on top of things?
In the last month, how often have you been angered because of things that were outside of your control?
In the last month, how often have you felt difficulties were piling up so high that you could not overcome them?

). Item responses are reverse-coded (if required) and summed to produce an overall score, ranging from 0 to a maximum of 40. As levels of perceived stress can be influenced by the participants’ day-to-day activities and other life events, the survey was administered monthly to monitor the participants’ perceived stress throughout the course.

2.3.2. Stress Mindset Measure (SMM)

In addition to PSS, SMM was also utilized as an additional perspective in understanding participants’ stress response [27]. Participants from the intervention group were asked to complete the 8 items of the SMM following a Likert scale of 0 (strongly disagree)–4 (strongly agree) immediately before and after gameplay (

Table 3. Example of 8-item survey questions on stress mindset measure, which is administered before and after gameplay.

8-Item Questions	Likert Scale (0 = Strongly Disagree, 4 = Strongly Agree)
The effects of stress are negative and should be avoided
Experiencing stress facilitates my learning and growth
Experiencing stress depletes my health and vitality
Experiencing stress enhances my performance and productivity
Experiencing stress inhibits my learning and growth
Experiencing stress improves my health and vitality
Experiencing stress debilitates my performance and productivity
The effects of stress are positive and should be utilized

). To obtain the final SMM score, we reverse-scored the odd-numbered items, then added up all the points for the 8 questions and divided the total by 8.

2.4. Physiological Response Measurements

2.4.1. Heart Rate Variability (HRV)

To determine physiological response, the Kana VR sensor is attached to their chest for capturing HRV throughout the VR gameplay. In this study, only the participants from the intervention group used the Kana VR sensor; hence, HRV was only measured in this group. HRV was then utilized to calculate the root mean square of successive differences (RMSSD), which measured the variability in heart rate intervals. Depending on the individual’s lifestyle, a healthy range for adults is typically between 15 and 95 [32]. Certain tasks in the game required the player to maintain a specific RMSSD value, which necessitated making a conscious effort to control physiological and stress responses. Participants monitored the real-time HRV, which was displayed on the screen as a litmus color scale. They learned ways to manage their stress levels by closing their eyes, looking away, or taking deep breaths.

2.4.2. Saliva Collection and Cortisol Levels

During the VR gameplay sessions at week 3 and week 12, participants were provided with a 50 mL tube for saliva collection. Before and after VR gameplay, about 2–3 mL of saliva from the participants in intervention and placebo groups were collected and subsequently stored at −20 °C for further analysis. Saliva samples were chosen for measuring cortisol levels as the method is non-invasive, simple to use, and reliable [33]. The concentrations of cortisol in saliva samples were measured in duplicate using the cortisol ELISA kits from Cayman Chemical (Ann Arbor, MI, USA) as described in the manufacturer’s procedure booklet. The saliva samples used were not diluted but used directly for the measurement of cortisol levels.

2.5. Training Performance Measurements

2.5.1. Breathing Apparatus Proficiency Test (BAPT)

The Breathing Apparatus Proficiency Test (BAPT) is designed to simulate the physical and physiological demands of actual firefighting operations. It comprises the Personal Protective Equipment (PPE) donning, four static stations (i.e., endless ladder, stair climb, hose carry, casualty drag), and a BA maze. In order to pass, trainees are required to complete the maze within 12 min and consume no more than 240 bars of air in their self-contained breathing apparatus.

2.5.2. Final Exercise Assessment (Final EX)

The Final Exercise Assessment is completed by trainees towards the end of the course. It is a simulated scenario where the trainees will take on the roles of the Duty Officer (DO) who will command the operations and provide instructions to the various sections, Section Commanders (SC) who lead a team of 4 trainees (i.e., firefighters), or as firefighters whose main roles are to execute the firefighting tasks. The instructors will assess the trainees based on their roles either as a DO or SC, out of 100 points.

2.5.3. Statistical Analysis

All data were expressed as fold change relative to means of baseline conditions and presented as mean ± SEM [34]. An outlier is predefined when an individual data point is 2 SDs from the mean, which will be excluded in data analysis and presentation as described earlier [35]. Group mean values were analyzed by unpaired Student’s t-test, one-way ANOVA with post hoc analysis using Dunnett’s test, or two-way ANOVA with post hoc analysis using Sidak’s test, as appropriate (GraphPad, San Diego, CA, USA) [36]. Statistical significance was considered when the p-value was <0.05.

3. Results

3.1. Psychological Stress Response Measurements

Over the 22 weeks of the training regimen, all participants were administered the perceived stress survey at various time points during the training period as measurements of psychological stress. Participants from the control and placebo groups had similar overall PSS profiles, where there was an initial increase in perceived stress in the early stages and it was subsequently maintained over the remaining training period. Notably, participants from the intervention group had an overall significantly lower (p < 0.05) level of perceived stress compared to the control and placebo groups (Figure 2a). Within the intervention group, SMM was administered before the game and after the game at weeks 3 and 12 of the training regimen. While there were no observable differences in stress mindset between pre- and post-gameplay at their respective weeks, it was noted that stress mindset was significantly increased (p < 0.05) at pre-game of week 12 when compared to the baseline response at week 3 pre-game. This suggests that the effects on stress mindset acquired from the gameplay may take time to develop (Figure 2b).

3.2. Physiological Stress Response Measurements

Saliva samples collected from the participants pre-game and post-game at weeks 3 and 12 were used as one of the measurements for physiological stress response (Figure 3). There was no significant change in cortisol levels between pre-game and post-game for both the placebo and intervention groups at week 3, even though cortisol levels tended to be higher post-game only in the placebo group (Figure 3a). At week 12, cortisol levels remained comparable between pre-game and post-game for the placebo group. Notably, there was a significant (p < 0.05) reduction in cortisol levels post-game in comparison to pre-game for the intervention group (Figure 3b), suggesting that there was a reduced level of stress after playing the VR game. Consistent with the cortisol results, the biofeedback data as a measurement of HRV (RMSSD) during the gameplay were also significantly increased (p < 0.05) at week 12 when compared to week 3 (Figure 3c), validating that the participants improved their ability to cope physiologically with stress during the second VR intervention.

3.3. Training Performance Measurements

To determine the effectiveness of stress management on training performance, the participants’ performance in the BAPT and Final Ex assessments was compared between the control, placebo, and intervention groups. Both activities were performance indicators as a firefighting trainee in the SCDF. Over the 22 weeks of training regimen, there were two sessions (week 7 and week 16) of BAPT assessments, where the participants were required to wear their personal protective equipment for firefighting, complete static tasks, and transverse through a BA maze within a given time. Hence, the time taken to complete the maze and oxygen consumption throughout the BAPT were important performance indicators for the BAPT assessment (Figure 4). For the BAPT assessment, there were no significant differences in time taken to complete the BA maze or oxygen consumption between control, placebo, or intervention groups at week 7 (Figure 4a,b) or week 16 (Figure 4c,d).

The other training performance is the Final EX assessment at week 22, which involves the participants performing the roles of duty officers or section commanders under a simulated environment of a firefighting operation. Specifically, the role of the duty officer was the ground commander who commanded the entire firefighting operation and gave instructions to the various section commanders. Hence, the role of the duty officer was associated with a certain level of stress as it involved critical thinking and decision-making skills under stressful conditions. Conversely, the role of the section commanders involved leading a team of firefighters to carry out the firefighting operation as instructed by the duty officer. Based on the participants’ role as duty officers, the intervention group had significantly higher scores (p < 0.05) (~15%) in the Final EX assessment when compared to the control group. This was not observed in the placebo group (Figure 5a). However, the final exercise assessments were comparable between all groups when the participants carried out the role of the section commanders (Figure 5b).

4. Discussion

The results obtained from the psychological stress response measurements are consistent with the current literature on the effects of serious games on stress [37,38,39,40,41]. According to previous studies, serious games can enhance positive emotions, motivation, and engagement, which can reduce the negative impact of stress on well-being and performance [38,40]. As shown in Figure 2, the participants who received the serious game intervention reported lower levels of perceived stress than the participants who received the placebo or no intervention. These findings suggest that the serious game intervention was effective in improving the perception of psychological stress of the SCDF trainees. Furthermore, the intervention group showed an increase in stress mindset from week 3 to week 12, indicating that they developed a more adaptive view of stress as a challenge rather than a threat [27]. Overall, the results of this study suggested the potential benefits of VR serious games as a stress management tool for SCDF trainees.

The stress response can be observed by the relationship of two adrenal hormones, cortisol as well as dehydroepiandrosterone (DHEA), which are useful biomarkers for stress measurements [29,30]. They showed that the emergence of a state of anxiety is first observed in an event of stress, followed by the observation of cortisol and DHEA. DHEA was observed to take on an “anti-cortisol” role where it opposed the actions of cortisol. Increased cortisol levels were also stated to be associated with the loss of control in participants within the study. Hormonal reactions to stressful conditions were investigated to explore physical thriving at the micro level [42]. It was suggested that physical flourishing happens when there are more growth-promoting or anabolic hormones than catabolic hormones. For example, subjects who have experienced trauma and evolved psychologically have speedier cortisol acclimatization to other stressors. One possible indicator of resilient psychological and physical functioning may be the adaptation of cortisol to stress. In our study, we focused on stresses that were faced by SCDF trainees and their responses to them to evaluate if stress mindset training is possible through VR gameplay. While the stress mindset training was shown to be effective with VR gameplay in our study, it is important to note that it is not possible to attribute this positive effect to gameplay alone as the stress mindset effect could also be influenced by other factors such as the SCC training itself. This is possibly regarded as one of the limitations of the study design as stress mindset was administered to the intervention group only; hence, it is not possible to compare this effect across the placebo or control groups. Thus, it is important to highlight that the observed positive results in this pilot study remain preliminary, and further research to include more participants to directly compare the effects of Kana VR across the placebo or control groups would provide more conclusive outcomes.

The physiological stress response measurements were in agreement with the current literature that showed the benefits of VR interventions on stress reduction [37,39,41]. As shown in Figure 3, saliva samples collected from the participants pre-game and post-game at week 3 and week 12 revealed no significant change in cortisol levels between pre-game and post-game for both placebo and intervention groups, except for a reduction in cortisol levels post-game in comparison to pre-game for the intervention group at week 12. This suggests that the VR game lowered the stress levels of the participants after they played it in week 12. In addition, the biofeedback data as a measurement of HRV (RMSSD) during the gameplay demonstrated a significant increase at week 12 when compared to week 3 (Figure 3c), indicating that the participants enhanced their ability to cope with stress during the second VR intervention at week 12. These results were supported by previous studies that found positive effects of VR interventions on stress reduction and coping skills [37,39,41].

The results obtained from the BAPT and final exercise assessments indicate that stress management did not have a significant effect on the training performance of the participants. This finding is consistent with some of the current literature on stress management and training performance [43,44], which suggests that stress management training may not directly improve the physical outcomes of leaders or athletes, but rather enhance their psychological well-being, coping skills, and organizational performance. However, other studies have reported positive effects of stress management training on various aspects of performance, such as productivity, cost-effectiveness, and health outcomes [45,46]. Interestingly, the findings of this study indicated that the intervention group, who participated in the VR game with biofeedback, performed better as duty officers than the control and placebo groups in the final exercise assessment. This may suggest that the intervention may have a greater impact on participants who were required to perform more complex and challenging tasks (i.e., as a duty officer) as compared to participants who were required to perform more routine taskings. Therefore, more research is needed to explore the potential benefits and limitations of stress management training for different types of performance indicators and contexts.

5. Conclusions

Our study explored the potential of a VR serious game as a digital tool to train the stress mindset in SCDF trainees. Our results proposed that participants who received the intervention reported lower perceived stress and an increased stress-is-enhancing mindset. Physiological stress markers cortisol levels and heart rate variability were also positively influenced by the intervention. The study observed mixed results on the impact on training performance. While the improved stress management did not relate to better training outcomes, we found that it had an impact on training outcomes when the tasks assessed were of higher complexity (i.e., when taking on the role of duty officers). This aligns with existing literature on the complex relationship between stress management and performance outcomes. The study calls for further research to explore the nuanced impact of VR-based interventions on diverse performance indicators, contexts, and user perceptions of the tool [47,48,49,50].