Next Article in Journal
Beacon-Based Remote Measurement of Social Behavior in ASD Clinical Trials: A Technical Feasibility Assessment
Next Article in Special Issue
A Qualitative Approach to Help Adjust the Design of Management Subjects in ICT Engineering Undergraduate Programs through User Experience in a Smart Classroom Context
Previous Article in Journal
Fully Learnable Model for Task-Driven Image Compressed Sensing
Previous Article in Special Issue
Measuring User Experience, Usability and Interactivity of a Personalized Mobile Augmented Reality Training System
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:

Gamification and Hazard Communication in Virtual Reality: A Qualitative Study

Institute for Research and Innovation in Bioengineering (I3B), Universitat Politècnica de València, Camino de Vera s/n, 46022 Valencia, Spain
Architecture Department, La Salle Campus Barcelona, Ramon Llull University, 2 Quatre Camins St., 08022 Barcelona, Spain
Author to whom correspondence should be addressed.
Sensors 2021, 21(14), 4663;
Submission received: 8 June 2021 / Revised: 2 July 2021 / Accepted: 2 July 2021 / Published: 7 July 2021
(This article belongs to the Collection Human-Computer Interaction in Pervasive Computing Environments)


An effective warning attracts attention, elicits knowledge, and enables compliance behavior. Game mechanics, which are directly linked to human desires, stand out as training, evaluation, and improvement tools. Immersive virtual reality (VR) facilitates training without risk to participants, evaluates the impact of an incorrect action/decision, and creates a smart training environment. The present study analyzes the user experience in a gamified virtual environment of risks using the HTC Vive head-mounted display. The game was developed in the Unreal game engine and consisted of a walk-through maze composed of evident dangers and different signaling variables while user action data were recorded. To demonstrate which aspects provide better interaction, experience, perception and memory, three different warning configurations (dynamic, static and smart) and two different levels of danger (low and high) were presented. To properly assess the impact of the experience, we conducted a survey about personality and knowledge before and after using the game. We proceeded with the qualitative approach by using questions in a bipolar laddering assessment that was compared with the recorded data during the game. The findings indicate that when users are engaged in VR, they tend to test the consequences of their actions rather than maintaining safety. The results also reveal that textual signal variables are not accessed when users are faced with the stress factor of time. Progress is needed in implementing new technologies for warnings and advance notifications to improve the evaluation of human behavior in virtual environments of high-risk surroundings.

1. Introduction

Warnings are tools that are frequently used to convey information about hazards when they cannot be designed out or guarded against [1]. Therefore, warnings need to influence people to act in a way to avoid personal injury and property damage. For this reason, their use is more necessary in a complex, hostile environment with communication blocks or interference and/or a stressful environment [2]. Effective warnings should rapidly attract attention, elicit knowledge, and enable compliance behavior (i.e., lead to appropriate decisions regarding performance execution). Previous studies have found that familiarity, stress, time pressure, or the presence of other mental activities can interfere with and reduce warning compliance rates [3,4]. Nevertheless, warning systems usually comprise passive elements that are always visible or, sometimes, dynamic lighting [2]. Advances in technology have produced a range of tools that can be used in this area to increase performance, evaluate user experience, learning and training, and control sensors and smart systems [5,6,7,8].
New technologies have also brought advances in the field of virtual reality (VR) by rapidly breaking spatial-temporal barriers and expanding its use and scope. The costs of VR consistently decrease as a consequence of the development of low-cost headsets [9], for example. The use of mobile technology to stimulate emotion, training, education, and health and to evaluate decision-making [10,11,12,13,14,15] has been increasing rapidly. Its use is especially highlighted when the physical presence offers some kind [16] of risk or extra cost. Environmental simulations make it possible to study these scenarios under controlled laboratory conditions [17]. Additionally, VR has become an important tool for participatory design, facilitating the evaluation of a product and the involvement of users in different development stages [18,19,20]. We believe that, when used with a qualitative methodology, VR becomes particularly relevant because it allows individuals to reflect and to expose their opinions. It is therefore possible to obtain detailed information with high reliability about the product or technology [21], making it easy to understand user behavior.
Researchers have recently paid attention to the playability and appeal of gamification techniques for human behavior assessments [22,23,24]. Coming from the video games and games, but differing of them by their non-entertainment purpose [24], these techniques are also becoming an important tool for training because they can be closely aligned with the design of good educational experiences [25,26]. In addition, they allow players to naturally produce rich sequences of actions while performing complex tasks by drawing on their competencies [27]. Gamification techniques are interactive [28], which is a key for motivation. Indeed, it is possible to say that game mechanics are linked to human desires, challenges, rewards, status, competition, success, and self-expression.
On the other hand, the user approach to user research techniques is mainly focused on the study of behavioral goals in work settings. For this reason, the task becomes the pivotal point of user-centered analysis and evaluation techniques [29]. The term user experience (UX), popularized by Don Norman [30], includes the feelings and meaningful aspects of user interaction with machines and services. Qualitative techniques enable the assessment of users’ degree of satisfaction, motivation, and interest with the item investigated. Currently, UX methods do not necessarily include the participation of the end user in the creative process of the product. We believe it is essential to define the causes of individuals’ behaviors and decisions in tasks. The Socratic model of postmodern psychology is valuable because it relates to data and addresses complex information about the product, experience, or technology studied [31]. The user is asked to reflect on a specific topic through a dialog between the user and the interviewer.
Originating from a constructivist paradigm, the qualitative method of bipolar laddering (BLA) acquires information from users themselves rather than only from observation of their behavior. Defined as a psychological exploration technique, BLA allows for the identification of key experience factors and how they are linked in a person’s mind [32]. Thus, BLA aims to identify the critical factors of any user experience. It operates using open-ended answer classifiers with positive/negative poles to define the strengths and weakness of systems [33]. This makes participative product design possible and promotes user participation in the testing of product design. Users and facilitators work together to define important aspects through strategic conversations. In this work, we adopted this technique to uncover how the qualities of the system, the implications of use, and personal values are connected in the minds of individuals [31].
In this paper, we present an immersive game (Game for Safety) that consists of a virtual environment endowed with signage and risks in which it is possible to extract user information and improve learning. This work born as an adaptation of the application created for the Game4City 3.0 project, which is being developed (2016–2021) by the Barcelona School of Architecture (Catalonia Polytechnic University, ETSAB-UPC) and the School of Architecture of La Salle—Ramon Llull University (ETSALS-URL) [34,35,36]. Game4City applies VR strategies to design 3D indoor and outdoor spaces for educational purposes [31,36,37,38,39,40]. Its aim is to show how the implementation of “gamified” virtual strategies in architecture, facilities, construction, and urban design can increase students’ spatial comprehension as well as other characters as citizens or professionals, increasing their interest in the collaborative/gamified/interactive design of spaces.
Inspired on Game4City, Game for Safety implements sensors in signaling systems with the aim of increasing obedience, perception and, consequently, efficiency. Therefore, this paper aimed to establish a UX assessment and answer the following research questions:
  • RQ1: Are there differences in the perception, behavior and usability in users with regard to different variable signs as a function of the level of danger?
  • RQ2: What elements could improve the design of a virtual environment for the analysis of user behavior in a gamified risk context?
  • RQ3: Which components of an immersive serious risk game can impact the user?
For this purpose, we propose some new configurations and variables for warnings that make them dynamic and smart to improve human interaction. By making use of a serious game in which the user interacts with the proposed signaling system and distinct dangers, we evaluate signals and game usability. Users execute a VR task while their performance is recorded using their navigation data and later exported into a video. This video is stored for further analysis. Using BLA methods, we explore users’ deep reflections [41], identify perceptions, and compare them with behavioral data to evaluate the user experience [42,43].
This work is organized as follows. Section 2 describes related work. In Section 3, we present the methods, methodology and evaluation procedures. The results of the experiments are presented in Section 4. Section 5 presents our conclusions and future work.

2. Related Work

Several studies have reported the success of games for learning and skill development [26,44,45,46]. The fact that a game environment is full of stimuli makes it an excellent tool to improve learning procedures. Games are flexible and can be consumed in a wide range of locations (e.g., home, school), and they present the opportunity to learn by experimenting and exploring [44]. This is in line with the contemporary context that emphasizes important results during the education process [47,48,49]. Based on this, VR games could provide a comprehensive training environment that brings new opportunities to safety teaching and learning processes [50].
Previous studies have shown that immersive serious games can promote better knowledge retention with the aim of aviation safety [51]. In this study, an immersive serious game is compared with a traditional education method. The serious game presents a better result for learning both immediately after the experience and for a longer period of time afterward. Greater engagement and fear arousal are identified as factors that contribute to explaining the findings.
In a previous study, a 3D game environment for construction engineering safety education greatly improved future construction personnel’s safety competency [52]. The refereed study was divided into three modules. In the first one, named safety knowledge dissemination, the students acquired the knowledge. The second task consisted of reflecting on the knowledge acquired during the first module and identifying the risks in an interactive environment (safety knowledge reflection). Finally, a knowledge assessment was conducted that required participants to resolve the hazards. The technology employed was not immersive.
A non-immersive virtual environment was also the object of study about risk assessment among university students in a chemical plant [53]. The results showed that users of the virtual environment were able to identify more risks in a real environment than nonusers.
Immersive virtual reality is an adequate and important tool that facilitates the points mentioned previously. The use of this type of mechanism for safety training prevents misunderstanding in real situations [54]. A study of emergency exits verified the effect of cause-consequence actions. By training in a virtual environment, users already knew the better action to take in a real hazard environment. However, for the success of a gamified proposal, it is necessary to use a range of variables, such as technological requirements and the design itself [46,55,56], which depends on the dynamic to be gamified, the context or device to be used, and the user´s profile.
In general, static signs made of paper, metal or plastic [57] are the main support system for safety communications. These are symbol-based signs consistent with the International Organization for Standardization (ISO) 3864-1 and American National Standards Institute (ANSI) Z535. Some studies have examined the effect of dynamic features in signs on behavioral compliance during a work-related task and emergency [3,58,59,60]. The findings suggested that dynamic presentations produced higher compliance than static presentations [58,61,62,63,64,65] mainly because of some features that make them more noticeable. This is in line with attention theory [66].
Other new alternatives are smart active systems or smart signals, concepts that have emerged from smart cities [8]. A smart signal is usually a traditional signal to which some smart capabilities are added, enabling an optimal match between occupants´ behavior and characteristics and environmental conditions [2,57]. The smart signal can be dynamic, adaptable and/or interactive. In a dangerous situation, this could be especially useful because people tend to narrow their attention in stressful situations. Salience could make smart signs more noticeable and more resistant to habituation (availability only when necessary, for instance). In the scope of this work, we focus on smart warnings in building sites.
Gamified environments have previously been used as a tool for training risk management. They have been used, for example, to help students interpret their activity, both in real time and later, through cause-consequence analysis [67,68,69]. Additionally, some VR environments have been used to evaluate products [19,55,70]. However, the efficacy of warning signals in a simulated environment has rarely been explored in research [60,71]. Additionally, very few works have used a gamified VR environment with a qualitative assessment to understand and evaluate usability and cause-consequence during the assessment.

3. Materials and Methods

From a methodological point of view, we adopted a user-centered design. The user´s experience was appraised with a qualitative approach. The objective was to evaluate behavior and define its causes [31]. This study privileged measures of behavior (attention, performance, errors and learnability) and satisfaction (qualitative assessment and subjective responses after game interaction), as summarized in Table 1.
The experiment consisted of testing the VR environment. Users crossed an office where an accident had just occurred, acting around the risks (run away, face or solve) and warnings (perceiving, reading and obey). Their explicit task was to cross the maze in the shortest time. They were informed that they could solve some risks (e.g., extinguish a fire), but this was not a mandatory action.
Prior to the game experience stage, demographic data and characteristics were collected with the aim of discarding individuals who, in some way, could compromise the research and identifying characteristics that could jeopardize the results. Additionally, data on behavioral traits, such as behavioral intention and compliance, were collected. This information clarified when signaling variables could improve users’ behavior and when the behavioral increase is the result of a propensity of individual behavior.
Once the users left the immersive office, a retention task was assessed in which the participants had to name the signals presented at the virtual office. More details will be explained next.

3.1. Components

Regarding the components required for this proposal, we took into account two premises:
  • Implementing a device system that allows easy use anywhere and is easily recognized, familiar and affordable;
  • Technological devices, tracking and mobile systems with a high degree of accuracy in stimuli reproduction to create a sense of presence [72,73].
Based on this, we chose sensor-type HDMs for VR headsets. We used the HTC Vive headset because it is designed to track movement freely with high precision through a space of 4 m × 4 m [74]. It consists of a headset with two controllers and two base stations that emit infrared pulses at 60 Hz, providing submillimeter tracking precision to the headset and controllers [30]. The headset has the following characteristics: two OLED panels, each with a display resolution of 1080 × 1200 pixels per eye update at 90 Hz cover a send out horizontal and vertical infrared laser sweeps spanning 120° in each direction, a mass of 470 g, G-sensor, gyroscope, proximity, remote control with a battery with a range of 6 h of play, and SteamVR tracking sensors [75]. The differences in time at which the laser hits the various photodiodes allows recovery of the position and orientation of headset.
HTC Vive has development compatibility with the Unity and Unreal Engine 4 game engines. For this work, we chose Unreal because of its more straightforward coding method (C++ coding allows the possibility of coding through blueprints and Visual Studio) [30,76,77]. Unreal is developed by Epic Games and is free for use for academic purposes. Figure 1 shows the components selected to implement the experiment.
The character movement adopted is lineal character movement. We choose this as it is the simplest method and reduces VR sickness. It consists of making the character move forward immediately when the user pushes the forward button [30].

3.2. Definition of the Participants’ Profile

To define the sample size of this study, it was necessary to take into account the saturation principle [65], which defines the extent to which the data collection provides new information in terms of the setting and sample. The purpose of the research was to obtain an in-depth understanding of users’ perceptions and to obtain better precision in the description. A large sample size may lead to a repetition of information [56]. On the other hand, users’ actions have fundamental relevance for the study. Therefore, special attention was paid to selecting an expert, active sample of professionals with a recognized reputation.
Considering this, the sample was composed of 10 (5 male and 5 female) volunteer specialists in risk management/architecture/engineering with more than 5 years of experience. All of them agreed to collaborate with the research and approved the informed consent to conduct the evaluation. Twenty percent of the sample had never used an HMD, while 80% had tried it once. All participants reported normal or corrected-to-normal vision and ranged in age from 31 to 65, as demonstrated in Figure 2. They all reported that they had no physical or mental conditions that they believed would prevent them from participating in a VR immersive game simulation.
With regard to game experience and technological knowledge, 20% of the users were frequent players. 3D technologies were frequently used by the participants, not for game purposes but for professional reasons, as shown in Figure 3.

3.3. Virtual Environment Game for Safety

To increase the involvement of the subjects, a game-based virtual environment was designed by a multi-disciplinary team composed of designers, programmers, multimedia animators, and specialists in educational technology and human computer interaction (HCI). We aimed to create an atmosphere of fear, because emotional arousal (especially with negative emotions) positively affects presence [78,79,80] and retention [79,81]. However, we aimed to prevent discomfort for the users to avoid diminished performance on difficult tasks [82]. Special attention was paid to the context, design, navigation, task to be performed, presence, involvement of the subject, and preventing sickness.
Based on this, we opted to create one environment composed of just one entrance and one exit, but a range of different halls to choose. The halls are similar, but the user does not know this. This helps to increase the user disorientation without produce physical discomforts. In addition, it allows evaluation of the behaviors of the player after receiving different types of information.
The application is based on a framework, which involves the following game elements [83]:
  • Dynamics—a narrative context communicated by both prior oral orientation and posters on the pre-experimental training room.
  • Mechanics—the challenge of cross the maze in the shortest which drive the action forward and generate engagement.
  • Components—the specific instantiations of mechanics and dynamics. In Game for Safety, particle effects were used to show the extra time obtained by the user during the game.
The setting, a maze, is reminiscent of a workplace, with walls similar to office partitions and a vinyl floor (purple, gray and black, with a different color for each section) with office elements (photocopier, coffee machine and water). The users were free to act based on their decisions. The character movement was linear; when the user pushed the forward button, the character immediately moved forward.
The first action in the environment was a pre-experimental training section. It consisted of a navigation task designed to eliminate any stress and anxiety and to introduce and familiarize the user with the technology adopted. This task was performed in a room composed of a main hall with a similar layout, as shown in Figure 4, that was specifically designed for training purposes. Posters were displayed that reported an incident that occurred in the office as well the necessary commands for navigation and the task.
After testing the commands, the subject immediately entered the principal maze, which was composed of three areas with the same configurations and same hazards but distinct variable signs. The signals were displayed near a referenced hazard (for which the user was alerted about the risk) or at strategic points (educative and route signals). Figure 5 shows the disposition of hazards, routes and educative signals. To guarantee that the subjects went through all the signals (experimental conditions), the labyrinth was mirrored (if the user chose right or left, he or she went through the same risk and the same warning). A command was included that added walls to prevent the user from returning to the same section. The walls appeared discreetly and were imperceptible to the user.
The VR Game for Safety allowed individuals to execute the tasks while security signals and hazards were presented, as shown in Figure 6. The warning design variables respected the following characteristics and were placed in such a way that each subject could visualize all signal types:
  • Static Sign—respect ANSI and ISO rules;
  • Dynamic Sign—the ISO sign with intermittent red-, yellow- and white-LED lights;
  • Smart signals—signal available only after crossing a certain point.
The experimental conditions were created with reference to conventional ANSI Z535.2 and ISO-type warnings of a wet floor, fire, height, fire classification and indicative arrow. The selected warnings and a description of their variables can be seen in Table 2. They were displayed near the referenced hazard (for which the user was alerted about the risk) or at strategic points (educative and route signals).
The participants were asked to cross the maze in the shortest time possible. The time was used to increase the challenge aspect. They were verbally warned that there were hazards on the way. Specifically, with regard to the fire risks, they were informed that they would be able to solve the issue using the fire extinguisher (and earn a bonus of an extra 10 s) or just pass by. The player knows the amount of bonus time gained through an explosion of a green “+10”. For this we used Cascade, the participle effect creation software from Unreal 4. If they assumed a risky position, they lost 10 s.
To evaluate both the virtual environment and the variables of the signals, we proceeded to evaluate the user-product relationship. In this particular case, the sample selected was the user and the virtual environment, and the warnings were the products/services. The users´ desires, needs and goals were assessed by techniques for obtaining and systematizing data to assess the users’ experience and identify necessary improvements [29].

3.4. The Experiment Protocol

The experiment was conducted over 5 days (two users per day) at the end of the morning and at the end of the afternoon to maintain maximum similarity of the conditions. The experimental session was divided into four phases: the introduction, pre-experimental training session, experimental session, and follow-up interview, as follows:

3.4.1. Stage 1 Introduction (5 min)

Participants gave informed consent in an anonymous form and then completed a questionnaire about demographics, behavioral intentions (safety behavior, safety experience, work situational assessment), game experience, and technological knowledge. The participants were told that the main objective of the study was to evaluate the warnings and the virtual environment, main tasks, and game mechanism.

3.4.2. Stage 2 Pre-Experimental Training Session (3 min)

For safety reasons arising from the current situation of COVID-19, we proceeded to sanitize and disinfect the devices. Then, the participants were placed in the pre-salon and asked to explore this room freely until they felt they were able to control the input device. When the exploration ended, the recording of participant´s navigation data was launched.

3.4.3. Stage 3 Experimental Session (10 min)

Stage 3 began after the participants reported that they were comfortable with the devices and able to begin. They crossed the START point, and the counting activation began. The users were free to verbalize their actions in this phase and we carefully took notes for posterior analyses.

3.4.4. Stage 4 Follow-up Interview (10 min)

The participants were asked about their first impression and then began the BLA. With this method, we aimed to determine how the product, the consequences of its use, and the personal assessment of the product were related to the user´s thinking [84]. For the BLA technique, we followed three steps:
  • Collection: This step consisted of a blank template for the positive elements (strengths) and another identical template for the negative elements (weaknesses). The interviewer asked the participants to state which aspects of the experiment and signals they liked best or that helped them in their tasks and what elements they disliked or that disturbed the task. We limited the elements to five positives and five negatives for each person. The elements were summarized in one word or a short sentence.
  • Assessment: Once the list of positive and negative aspects was finished, the interviewer asked the user to rate each aspect on a scale using a score between 0 (lowest) and 10 (most).
  • Definition: At this point, the interviewer read aloud the elements of both lists to the user and asked him or her for a justification. The answer had to be a specific explanation of the exact characteristics that made the mentioned elements strengths or weaknesses of the product [29]. Then, the user was asked for a solution to the problems (for negative elements) or improvements (for positive elements).
The users then completed a survey, which was composed of a simulator sickness questionnaire, usability aspects of warnings, and satisfaction questionnaires. The users named the warnings they remembered and relevant aspects and indicated how much they liked each warning variable on a 5-point Likert scale from 1 (very bad) to 5 (great). We thanked the users for participating in the study.

4. Results

Regarding the potentially influential users’ characteristics, our sample presented the following characteristics:
  • Situational awareness coefficient of 0.89 for work situational awareness (reliability, which is acceptable if Cronbach’s alpha = 0.86) [85];
  • Safety behavior—According to the mean scores, the respondents´ agreement/disagreement levels ranged from strongly disagree (mean = 1.75) to strongly agree (mean = 4.13);
  • Safety experience—In terms of safety experience and, consequently, greater propensity to attention to prevent risk [86], 90% of the users had suffered or knew someone who had suffered a labor accident.
From the results obtained during immersion, it was possible to verify that the Game for Safety was engaging. By observing the video recordings, we found that warnings that were more perceptible were those that informed about hazards, as these were the first place that users looked. The signal with text was perceived by users, but they only stopped to read the dynamic signals (with LEDs). The signals on the floor were not perceived by the users; they claimed that this was because they were more concerned about the risks. These presented a high perception level. Figure 7 shows an example of a video recording extracted by navigation data.
With regard to simulation sickness, the data showed a low rate of discomfort, as shown in Figure 8. One possible cause of this distress could be if the experiment was conducted just after the user had eaten.
The evaluation of smart signals was discarded because of its low perceived ratings, caused by the elevated cognitive task due to its location. With regard to the static warnings scores, 80% of users considered them clear and 90% thought they were pleasant. The dynamic (LED) scores were 60% clear and 60% pleasant.
The results obtained in the BLA were classified according to the method with the following premises:
  • Common positive elements (positive elements cited by more than one user—CPx);
  • Particular positive elements (those mentioned by only one user—PPx);
  • Common negative elements (negative elements cited by more than one user—CNx);
  • Particular negative elements (items mentioned by only one user—PNx).
According to their positive or negative evaluations, the common elements mentioned at higher rates (except the CP average score 10.0) were the most important aspects to improve, use, adjust or modify. Table 3 shows the highlighted aspects identified by the users and their respective classifications. Additionally, a mention index was obtained to determine the elements that were perceived by the users (a high mention index indicates that the element was perceived more by users). The combination of these elements determined the most relevant items obtained from the BLA.
As shown in Table 3, the higher consensus on the positive aspects identified with an MI of 60% referred to the assessment of decision making. The second most rated characteristic was learning without risk (50% MI). This fact could prove the usefulness of Game for Safety to evaluate causes and consequences. Other positive issues cited with rates above 30% were realism, usability, time for the task, and the presence of warnings (all 40% MI). This last one deserves attention because it is related to issues that also appeared in particular comments with the highest average (LED warnings and warning pictograms). This fact can verify the importance of and attention to warnings in decision making. It is important to note that, for positive elements, negative scores tend to apply to elements that are perceived as bonus features that work badly [32]. For example, in the case of realism (Av. 4.75), the users perceived it to be positive as a concept, but their final experience was not pleasant.
For the negative issues, there was no MI above 40%. The most frequently mentioned issue was the presence of distractors. The users stated, for example, that the presence of hourglasses (extra bonus) aroused the desire to trigger them, although it was not a mandatory action. Additionally, in real accident situations, there is no extra time awarded for safety.
One interesting point is the fact of both realism and virtual environment were positively considered by women, while male participants rated them mostly with negative scores. This difference is aligned with previously findings in research about gender difference behaviors in virtual environments [87,88]. We consider this an important aspect which deserves further exploration in a quantitative study
The improvements and suggestions by the users were classified as common solutions (CS) when they were mentioned by more than one user and as particular solutions (PP) when they were cited by just one user, as shown in Table 4. It is interesting to note in this phase that the same suggestion was presented by more than one user as a solution or improvement for more than one item.
The three most cited suggestions were to adjust interactions to the user’s profile, reduce elements, and improve the layout. The users commented that the character’s locomotion should change according to the user’s profile by enabling or disabling the lateral displacement of the character according to how often the user utilized the technology. The suggestion of more information was related to sickness (inform users not to eat just before the experiment).

5. Conclusions and Future Work

The main contribution of this paper is the evaluation of variables to improve the design of serious virtual games to analyze the behavior of users in a risk environment as well as variable warnings to evaluate their effect on users’ interaction. We assessed the experience by measuring both subjective and behavioral measures. With regard to the results obtained from the video recording originated from navigation data, physical reaction, behavioral evidence, and questionnaire ratings, we addressed the following question:
  • RQ1: Are there differences in the perception, behavior and usability of users for different variable signs as a function of the level of danger?
The outcomes of the survey showed a higher-level perception of warnings when they were endowed with dynamic technology variables. In this sense, we found that high percentages of participants in a gamified and stressful condition directed their vision first to a signal with an intermittent LED variable. In a textual poster, the LED variables were related to higher perception and reaction time. However, neither the color nor the disposition generated substantial effects on users.
Gravity and eminence did not produce distinct reactions in perception, behavior or decision making. All participants’ behavior remained constant for different types of hazards (solved or escaped).
Considering the data obtained in the study and our research question, we can provide the following insights.
  • RQ2: What elements could improve the design of a virtual environment for the analysis of user behavior in a gamified risk context?
In this sense, we identified two main elements: time and stress.
It has been shown that the pressure of finishing the principal task (crossing the maze) in less time in addition to the personal tension of addressing risks constitutes conflicting objectives.
According to the BLA evaluation, stress had three principal causes: repetitive sensation, time pressure and excess elements. A long maze with a similar layout provokes the feeling of constantly being in the same place. The disposition of various elements works as a strong distractor that deviates users. In this way, the simple reason of having an extra time bonus element added by the gamification surrounding encourages individuals to achieve the goal of earning time even if they do not need more time.
Therefore, we can say that the environment layout plays an important role in virtual environments. It is very important to consider the selection of the elements and their locations to motivate, distract or focus the user.
By crossing the three phase dates and considering our third research question, we can conclude the following:
  • RQ3: Which components of an immersive serious risk game can impact the user?
Our results suggest that the possibility of evaluating cause and effect constitutes a powerful argument for the use of an immersive virtual environment. Additionally, the engagement produced by gamification stimulates training tools.
We also found that motivators in a virtual environment consist of better adjusting objectives to user profiles rather than enhancing elements. This happens, for example, because the system requirements of a person with game familiarity differ from those of a person without game familiarity. Game mechanics and the virtual environment therefore provide a level of engagement and interactivity that makes it a promising tool in any phase of the design process.
In relation to warnings, the present study demonstrates that technological advances provide sensors capable of increasing warnings. Therefore, the mode in which the signal is presented influences users’ perceptions. Sensors aimed at providing dynamic warnings are a good solution to improve the efficiency of signals. However, some changes should be made to evaluate smart signals. Based on the results of this study, we will investigate the range of possibilities of smart variables and their effects (e.g., changing the horizontal form at the floor, personalizing warnings, and implementing audio cues).
As a next step for this study, we will work to implement Game for Safety in a quantitative study. In this way, we will be able to conduct an objective analysis of the influences of warning variables to reduce accidents. We also want to explore the influence of gender in the behaviours.

Author Contributions

Conceptualization, J.C., M.C. and D.F.; methodology, J.C. and D.F.; software, V.V. and J.C.; validation, J.C., M.C. and D.F.; formal analysis, J.C. and D.F.; investigation, J.C.; resources, J.C., V.V. and M.C.; writing—original draft preparation, all authors; writing—review and editing, J.C., M.C. and D.F.; supervision, M.C. and D.F.; funding acquisition, D.F. All authors have read and agreed to the published version of the manuscript.


This research was funded by La Salle Ramon Llull University and La Salle Catalunya through the project EduEnginy. This research paper was also supported by the GRETEL research group and the Secretaria d’Universitats i Recerca del Departament d’Empresa i Coneixement de la Generalitat de Catalunya (grant 2017 SGR 934), and the National Program of Research, Development and Innovation aimed at the Society Challenges with the reference BIA2016-77464-C2-2-R of the National Plan for Scientific Research, Development and Technological Innovation 2013-2016, Government of Spain, titled “Diseño Gamificado de visualización 3D con sistemas de realidad virtual para el estudio de la mejora de competencias motivacionales, sociales y espaciales del usuario (EduGAME4CITY)”. (AEI/FEDER, UE).

Institutional Review Board Statement

The research presented, as well as the design, collection and management of its data, has been POSITIVE evaluated and APPROVED, by the Ethics Committee of the Ramon Llull University with the file number: CER URL_2020_2021_009.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data of the study can be available at this link: (accessed on 5 June 2021).


Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.


  1. Zielinska, O.A.; Mayhorn, C.B.; Wogalter, M.S. Connoted Hazard and Perceived Importance of Fluorescent, Neon, and Standard Safety Colors. Appl. Ergon. 2017, 65, 326–334. [Google Scholar] [CrossRef]
  2. Asensio, Á.; Blanco, T.; Blasco, R.; Marco, Á.; Casas, R. Managing Emergency Situations in the Smart City: The Smart Signal. Sensors 2015, 15, 14370–14396. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  3. Hui, X.; Galea, E.R.; Lawrence, P.J. Experimental and Survey Studies on the Effectiveness of Dynamic Signage Systems. Fire Safety Sci. 2014, 11, 1129–1143. [Google Scholar] [CrossRef] [Green Version]
  4. Laughery, K.R. Safety Communications: Warnings. Appl. Ergon. 2006, 37, 467–478. [Google Scholar] [CrossRef]
  5. Rebelo, F.; Noriega, P.; Duarte, E.; Soares, M. Using Virtual Reality to Assess User Experience. Hum. Factors 2012, 54, 964–982. [Google Scholar] [CrossRef]
  6. Escalona, F.; Martinez-Martin, E.; Cruz, E.; Cazorla, M.; Gomez-Donoso, F. EVA: EVAluating at-Home Rehabilitation Exercises Using Augmented Reality and Low-Cost Sensors. Virtual Real. 2020, 24. [Google Scholar] [CrossRef]
  7. Du Vignaux, M.M.; Léger, P.-M.; Charland, P.; Salame, Y.; Durand, E.; Bouillot, N.; Pardoen, M.; Sénécal, S. An Exploratory Study on the Impact of Collective Immersion on Learning and Learning Experience. Multimodal Technol. Interact. 2021, 5, 17. [Google Scholar] [CrossRef]
  8. Merlino, G.; Bruneo, D.; Distefano, S.; Longo, F.; Puliafito, A.; Al-Anbuky, A. A Smart City Lighting Case Study on an OpenStack-Powered Infrastructure. Sensors 2015, 15, 16314–16335. [Google Scholar] [CrossRef] [Green Version]
  9. Marín-Morales, J.; Llinares, C.; Guixeres, J.; Alcañiz, M. Emotion Recognition in Immersive Virtual Reality: From Statistics to Affective Computing. Sensors 2020, 20, 5613. [Google Scholar] [CrossRef]
  10. Olmos, E.; Cavalcanti, J.F.; Soler, J.-L.; Contero, M.; Alcañiz, M. Mobile Virtual Reality: A Promising Technology to Change the Way We Learn and Teach. In Mobile and Ubiquitous Learning: An International Handbook; Yu, S., Ally, M., Tsinakos, A., Eds.; Springer: Singapore, 2018; pp. 95–106. [Google Scholar] [CrossRef]
  11. Varela-Aldás, J.; Palacios-Navarro, G.; Amariglio, R.; García-Magariño, I. Head-Mounted Display-Based Application for Cognitive Training. Sensors 2020, 20, 6552. [Google Scholar] [CrossRef]
  12. Lovreglio, R.; Duan, X.; Rahouti, A.; Phipps, R.; Nilsson, D. Comparing the Effectiveness of Fire Extinguisher Virtual Reality and Video Training. Virtual Real. 2021, 25, 133–145. [Google Scholar] [CrossRef]
  13. Winn, W.; Windschitl, M.; Fruland, R.; Lee, Y. When Does Immersion Ina a Virtual Environmment Helps Students Construct Understanding? In Proceedings of the International Conference of the Learning Societis, Seattle, WA, USA, 23–26 October 2002; University of Washington: Seattle, WA, USA, 2002; pp. 497–503. [Google Scholar]
  14. Reiners, T.; Wood, L.C.; Chang, V.; Gütl, C.; Herrington, J.; Teräs, H.; Gregory, S. Operationalising Gamification in an Educational Authentic Environment. IADIS Int. Conf. Internet Technol. Soc. 2012, 2012, 93–100. [Google Scholar]
  15. Bell, J.T.; Fogler, H.S.; Arbor, A. The Application of Virtual Reality to Chemical Engineering Education. In Proceedings of the IEEE Virtual Reality 2004, Chicago, IL, USA, 27–31 March 2004. [Google Scholar]
  16. Tate, D.L.; Sibert, L.; King, T. Using Virtual Environments to Train Firefighters. IEEE Comput. Graph. Appl. 1997, 17, 23–29. [Google Scholar] [CrossRef]
  17. Vince, J. Introduction to Virtual Reality [Electronic Resource], 1st ed.; Springer: London, UK, 2004. [Google Scholar] [CrossRef]
  18. Ye, J.; Badiyani, S.; Raja, V.; Schlegel, T. Applications of Virtual Reality in Product Design Evaluation BT—Human-Computer Interaction. HCI Applications and Services; Jacko, J.A., Ed.; Springer: Berlin/Heidelberg, Germany, 2007; pp. 1190–1199. [Google Scholar]
  19. Galán, J.; García-García, C.; Felip, F.; Contero, M. Does a Presentation Media Influence the Evaluation of Consumer Products? A Comparative Study to Evaluate Virtual Reality, Virtual Reality with Passive Haptics and a Real Setting. Int. J. Interact. Multimed. Artif. Intell. 2011. [Google Scholar] [CrossRef]
  20. Ayanoglu, H.; Duarte, E.; Teles, J.; Noriega, P.; Rebelo, F. Hazard Perception of 3D Household Packages: A Study Using a Virtual Environment; CRC Press: Boca Raton, FL, USA, 2016. [Google Scholar] [CrossRef]
  21. Fonseca, D.; Pifarré, M.; Redondo, E.; Alitany, A.; Riera, A. Combination of Qualitative and Quantitative Techniques in the Analysis of New Technologies Implementation in Education: Using Augmented Reality in the Visualization of Architectural Projects. In Proceedings of the Iberian Conference on Information Systems and Technologies (CISTI), Lisboa, Portugal, 19–22 June 2013. [Google Scholar]
  22. Connolly, T.M.; Boyle, E.A.; MacArthur, E.; Hainey, T.; Boyle, J.M. A Systematic Literature Review of Empirical Evidence on Computer Games and Serious Games. Comput. Educ. 2012, 59, 661–686. [Google Scholar] [CrossRef]
  23. Merchant, Z.; Goetz, E.T.; Cifuentes, L.; Keeney-Kennicutt, W.; Davis, T.J. Effectiveness of Virtual Reality-Based Instruction on Students’ Learning Outcomes in K-12 and Higher Education: A Meta-Analysis. Comput. Educ. 2014, 70, 29–40. [Google Scholar] [CrossRef]
  24. Lavoué, É.; Monterrat, B.; Desmarais, M.; George, S. Adaptive Gamification for Learning Environments. IEEE Trans. Learn. Technol. 2019, 12, 16–28. [Google Scholar] [CrossRef]
  25. Koster, R. A Theory of Fun for Game Design, 2nd ed.; O’Reilly Media, Inc.: Newton, MA, USA, 2013. [Google Scholar]
  26. Gee, J.P. Are Video Games Good for Learning? Nord. J. Digit. Lit. 2006, 1, 172–183. [Google Scholar] [CrossRef]
  27. Shute, V.J.; Ventura, M.; Zapata-rivera, D.; Bauer, M. Melding the Power of Serious Games and Embeded Assessment to Monitor and Foster Learning Flow and Grow. In Serious Games: Mechanisms and Effects; Routkedge: New York, NY, USA, 2009; pp. 295–321. [Google Scholar] [CrossRef]
  28. Nedel, L.; Menin, A.; de Souza, V.C.; Sebben, L.; Oliveira, J.; Faria, F.; Maciel, A. Using Immersive Virtual Reality to Reduce Work Accidents in Developing Countris. Vis. Comput. Prog. Dev. Ctries. 2016, 36–46. [Google Scholar] [CrossRef]
  29. Fonseca, D.; Redondo, E.; Villagrasa, S. Mixed-Methods Research: A New Approach to Evaluating the Motivation and Satisfaction of University Students Using Advanced Visual Technologies. Univers. Access Inf. Soc. 2015, 14, 311–332. [Google Scholar] [CrossRef] [Green Version]
  30. Norman, D.; Miller, J.; Henderson, A. What You See, Some of What’s in the Future, And How We Go About Doing It: HI at Apple Computer; Association for Computing Machinery: New York, NY, USA, 1995. [Google Scholar] [CrossRef]
  31. Fonseca, D.; Cavalcanti, J.; Peña, E.; Valls, V.; Sanchez-Sepúlveda, M.; Moreira, F.; Navarro, I.; Redondo, E. Mixed Assessment of Virtual Serious Games Applied in Architectural and Urban Design Education. Sensors 2021, 21, 3102. [Google Scholar] [CrossRef] [PubMed]
  32. Pifarreacute, M.; Sorribas, X.; Villegas, E. BLA (Bipolar Laddering) Applied to YouTube. Performing Postmodern Psychology Paradigms in User Experience Field. In Advanced Technologies; Intech Open: Rijeka, Croatia, 2009. [Google Scholar] [CrossRef] [Green Version]
  33. Redondo, E.; Fonsec, D.; Vall, F.; Olivares, A. Mobile Device Based Teaching. New Challenges in Architectural Representation Education. Case of Study: The “Tianguis” in Tonalá, Jalisco, Mexico. EGA Rev. Expr. Graf. Arquit. 2016, 21, 64–73. [Google Scholar] [CrossRef] [Green Version]
  34. Redondo, E.; Fonseca, D.; Sánchez-Sepúlveda, M.; Zapata, H.; Navarro, I.; Gimenez, L.; Pérez, M.A. Edugame4city. A Gamification for Architecture Students. Viability Study Applied to Urban Design. In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Springer: Cham, Switzerland, 2020; Volume 12206. [Google Scholar] [CrossRef]
  35. Redondo, E.; Zapata, H.; Navarro, I.; Fonseca, D.; Gimenez, L.; Pérez, M.Á.; Sánchez-Sepúlveda, M. Game4city. Gamification for Citizens through the Use of Virtual Reality Made Available to the Masses. Viability Study in Two Public Events. In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Springer: Cham, Switzerland, 2020; Volume 12206. [Google Scholar] [CrossRef]
  36. Sanchez-Sepulveda, M.; Fonseca, D.; Franquesa, J.; Redondo, E. Virtual Interactive Innovations Applied for Digital Urban Transformations. Mixed Approach. Futur. Gener. Comput. Syst. 2019, 91, 371–381. [Google Scholar] [CrossRef]
  37. Calvo, X.; Fonseca, D.; Sánchez-Sepúlveda, M.; Amo, D.; Llorca, J.; Redondo, E. Programming Virtual Interactions for Gamified Educational Proposes of Urban Spaces. In LCT 2018, LNCS 10925; Zaphiris, P., Ioannou, A.I., Eds.; Springer International Publishing AG: Cham, Switzerland, 2018; pp. 1–13. [Google Scholar] [CrossRef]
  38. Fonseca, D.; Valls, F.; Redondo, E.; Villagrasa, S. Informal Interactions in 3D Education: Citizenship Participation and Assessment of Virtual Urban Proposals. Comput. Human Behav. 2016, 55, 504–518. [Google Scholar] [CrossRef] [Green Version]
  39. Fonseca, D.; Navarro, I.; de Renteria, I.; Moreira, F.; Ferrer, Á.; de Reina, O. Assessment of Wearable Virtual Reality Technology for Visiting World Heritage Buildings: An Educational Approach. J. Educ. Comput. Res. 2018, 56, 940–973. [Google Scholar] [CrossRef]
  40. Navarro, I.; De Reina, O.; Fonseca, D.; Gómez, M.; Ferrer, Á. Virtual Reality Using Smart-Devices in Educational Frameworks: Case Study: Museum Casa Batlló. Int. J. Technol. Hum. Interact. 2017, 13, 50–61. [Google Scholar] [CrossRef]
  41. Redondo, E.; Fonseca, D.; Giménez, L.; Santana, G.; Navarro, I. Digital Literacy for the Teaching of Architecture. A Case Study. Arquitetura Rev. 2012, 8. [Google Scholar] [CrossRef] [Green Version]
  42. Llorca, J.; Zapata, H.; Redondo, E.; Alba, J.; Fonseca, D. Bipolar Laddering Assessments Applied to Urban Acoustics Education. In Advances in Intelligent Systems and Computing; Springer: Cham, Switzerland, 2018; Volume 747, pp. 287–297. [Google Scholar] [CrossRef]
  43. Centeno, E.; Alaez, M.; Fonseca, D.; Fernández-Bernal, F.; Wogrin, S.; Romero, S.; Ferrán, A.; Alsina, M.; Necchi, S. A Comparative Study of the Application of Lesson Study in Different University Learning Environments. In Learning and Collaboration Technologies. Designing, Developing and Deploying Learning Experiences; Zaphiris, P., Ioannou, A., Eds.; Springer International Publishing: Cham, Switzerland, 2020; pp. 425–441. [Google Scholar]
  44. Gomez, M.J.; Ruipérez-Valiente, J.A.; Martínez, P.A.; Kim, Y.J. Applying Learning Analytics to Detect Sequences of Actions and Common Errors in a Geometry Game. Sensors 2021, 21, 25. [Google Scholar] [CrossRef]
  45. Moreira, F.; Ferreira, M.J.; Escudero, D.F.; Pereira, C.S.; Durao, N. Teaching and Learning Modelling and Specification Based on Gamification. Iber. Conf. Inf. Syst. Technol. Cist. 2020, 2020, 24–27. [Google Scholar] [CrossRef]
  46. Deliyannis, I.; Kaimara, P. Developing Smart Learning Environments Using Gamification Techniques and Video Game Technologies. In Didactics of Smart Pedagogy: Smart Pedagogy for Technology Enhanced Learning; Daniela, L., Ed.; Springer International Publishing: Cham, Switzerland, 2019; pp. 285–307. [Google Scholar] [CrossRef]
  47. Uskov, V.; Howlett, R.; Jain, L. Smart Education and E-Learning 2020; Springer: Cham, Switzerland, 2020. [Google Scholar] [CrossRef]
  48. Sen, A.; Chuen, C.L.K.; Zay Hta, A.C. Toward Smart Learning Environments: Affordances and Design Architecture of Augmented Reality (AR) Applications in Medical Education. In Smart Innovation, Systems and Technologies; Springer: Cham, Switzerland, 2018; Volume 79. [Google Scholar] [CrossRef]
  49. Park, S.-J.; Joo, M.-I.; Kim, H.-C. Virtual Reality Based Smart Learning System Design for Forensic Medicine. Adv. Sci. Lett. 2018, 24. [Google Scholar] [CrossRef]
  50. Sun, H.; Li, S.; Zhu, Y.; Hsiao, B. The Effect of User’s Perceived Presence and Promotion Focus on Usability for Interacting in Virtual Environments. Appl. Ergon. 2015, 50, 126–132. [Google Scholar] [CrossRef]
  51. Chittaro, L.; Buttussi, F. Assessing Knowledge Retention of an Immersive Serious Game vs. A Traditional Education Method in Aviation Safety. IEEE Trans. Vis. Comput. Graph. 2015, 21, 529–538. [Google Scholar] [CrossRef] [PubMed]
  52. Le, Q.T.; Pedro, A.; Lim, C.R.; Park, H.T.; Park, C.S.; Kim, H.K. A Framework for Using Mobile Based Virtual Reality and Augmented Reality for Experiential Construction Safety Education. Int. J. Eng. Educ 2015, 31, 713–725. [Google Scholar]
  53. Bell, J.T.; Fogler, H.S. A Virtual Reality Safety and Hazard Analysis Simulation. In Proceedings of the ASEE Annual Conference, St. Louis, MI, USA, 18–21 June 2000; pp. 1–6. [Google Scholar]
  54. Bernardes, S.M.F.; Rebelo, F.; Vilar, E.; Noriega, P.; Borges, T. Methodological Approaches for Use Virtual Reality to Develop Emergency Evacuation Simulations for Training, in Emergency Situations. Procedia Manuf. 2015, 3, 6313–6320. [Google Scholar] [CrossRef] [Green Version]
  55. Méndez, J.I.; Mata, O.; Ponce, P.; Meier, A.; Peffer, T.; Molina, A. Multi-Sensor System, Gamification, and Artificial Intelligence for Benefit Elderly People. In Challenges and Trends in Multimodal Fall Detection for Healthcare; Ponce, H., Martínez-Villaseñor, L., Brieva, J., Moya-Albor, E., Eds.; Springer International Publishing: Cham, Switzerland, 2020; pp. 207–235. [Google Scholar] [CrossRef]
  56. Villegas, E.; Fonseca, D.; Peña, E.; Bonet, P.; Fernández-guinea, S. Qualitative Assessment of Effective Gamification Design Processes Using Motivators to Identify Game Mechanics. Sensors 2021, 21, 2556. [Google Scholar] [CrossRef]
  57. Vilar, E.; Rebelo, F.; Noriega, P. Smart Systems in Emergency Wayfinding: A Literature Review. In Design, User Experience, and Usability: Designing Interactions; Springer: Cham, Switzerland, 2018; pp. 379–388. [Google Scholar] [CrossRef]
  58. Duarte, E.; Rebelo, F.; Teles, J.; Wogalter, M.S. Behavioral Compliance for Dynamic versus Static Signs in an Immersive Virtual Environment. Appl. Ergon. 2013, 45, 5–14. [Google Scholar] [CrossRef] [PubMed]
  59. Vilar, E.; Duarte, E.; Rebelo, F.; Noriega, P.; Vilar, E.F. A Pilot Study Using Virtual Reality to Investigate the Effects of Emergency Egress Signs Competing with Environmental Variables on Route Choices. Des. User Exp. Usability. User Exp. Des. Everyday Life Appl. Serv. 2014, 8519, 530–541. [Google Scholar] [CrossRef]
  60. Duarte, E.; Rebelo, F. Potential Uses of Virtual Reality to Evaluate Behavioral Compliance with Warnings. In Proceedings of the 2008 AHFE International Conference, Las Vegas, NV, USA, 14–17 July 2008. [Google Scholar]
  61. Duarte, E.; Rebelo, F.; Wogalter, M.S. Virtual Reality and Its Potential for Evaluating Warning Compliance. Hum. Factors Ergon. Manuf. Serv. Ind. 2010, 20, 526–537. [Google Scholar] [CrossRef]
  62. Smith-Jackson, T.L.; Wogalter, M.S. Potential Uses of Technology to Communicate Risk in Manufacturing. Hum. Factors Ergon. Manuf. 2004, 14, 1–14. [Google Scholar] [CrossRef]
  63. Wogalter, M.S.; Mayhorn, C.B. Providing Cognitive Support with Technology-Based Warning Systems. Ergonomics 2005, 48, 522–533. [Google Scholar] [CrossRef]
  64. Schall, D.L.; Doll, D.; Mohnen, A. Caution! Warnings as a Useless Countermeasure to Reduce Overconfidence? An Experimental Evaluation in Light of Enhanced and Dynamic Warning Designs. J. Behav. Decis. Mak. 2017, 30, 347–358. [Google Scholar] [CrossRef]
  65. Olander, J.; Ronchi, E.; Lovreglio, R.; Nilsson, D. Dissuasive Exit Signage for Building Fire Evacuation. Appl. Ergon. 2017, 59, 84–93. [Google Scholar] [CrossRef]
  66. Hockey, G.R. Applied Attention Theory. Ergonomic 2009, 52, 270–271. [Google Scholar] [CrossRef]
  67. Barot, C.; Burkhardt, J.-M.; Lourdeaux, D.; Lenne, D. V3S, a Virtual Environment for Risk Management Training. In Proceedings of the JVRC11: Joint Virtual Reality Conference of EGVE-EuroVR, Nottingham, UK, 20–21 September 2011; pp. 95–102. [Google Scholar] [CrossRef]
  68. Amokrane, K.; Lourdeaux, D.; Burkhardt, J. HERA: Lear Ner Tr Acking in a Vir Tual Environment. Int. J. Virtual Real. 2008, 7, 23–30. [Google Scholar]
  69. Lesch, M.F. Warning Symbols as Reminders of Hazards: Impact of Training. Accid. Anal. Prev. 2008, 40, 1005–1012. [Google Scholar] [CrossRef] [PubMed]
  70. Xu, Z.; Lu, X.Z.; Guan, H.; Chen, C.; Ren, A.Z. A Virtual Reality Based Fire Training Simulator with Smoke Hazard Assessment Capacity. Adv. Eng. Softw. 2014, 68, 1–8. [Google Scholar] [CrossRef] [Green Version]
  71. Mantovani, G.; Gamberini, L.; Martinelli, M.; Varotto, D. Exploring the Suitability of Virtual Environments for Safety Training: Signals, Norms and Ambiguity in a Simulated Emergency Escape. Cogn. Technol. Work 2001, 3, 33–41. [Google Scholar] [CrossRef]
  72. Slater, M. Place Illusion and Plausibility Can Lead to Realistic Behaviour in Immersive Virtual Environments. Philos. Trans. R. Soc. Lond. B. Biol. Sci. 2009, 364, 3549–3557. [Google Scholar] [CrossRef] [Green Version]
  73. Slater, M.; Perez-Marcos, D.; Ehrsson, H.H.; Sanchez-Vives, M.V. Towards a Digital Body: The Virtual Arm Illusion. Front. Hum. Neurosci. 2008, 2, 6. [Google Scholar] [CrossRef] [Green Version]
  74. Vive. What Is the Recommended Space for the Play Area? Available online: (accessed on 30 December 2020).
  75. Niehorster, D.C.; Li, L.; Lappe, M. The Accuracy and Precision of Position and Orientation Tracking in the HTC Vive Virtual Reality System for Scientific Research. Iperception 2017, 8. [Google Scholar] [CrossRef] [Green Version]
  76. Unity Technologies. Powerful 2D, 3D, VR, & AR Software for Cross-Platform Development of Games and Mobile Apps. Available online: (accessed on 30 December 2020).
  77. Šmíd, A. Comparison of Unity and Unreal Engine; Czech Tecnhical University: Prague, Czechia, 2017. [Google Scholar]
  78. Saunders, B.; Saunders, B.; Sim, J.; Sim, J.; Kingstone, T.; Kingstone, T.; Baker, S.; Baker, S.; Waterfield, J.; Waterfield, J.; et al. Saturation in Qualitative Research: Exploring Its Conceptualization and Operationalization. Qual. Quant. 2018, 52, 1893–1907. [Google Scholar] [CrossRef] [PubMed]
  79. Peperkorn, H.M.; Diemer, J.; Mühlberger, A. Temporal Dynamics in the Relation between Presence and Fear in Virtual Reality. Comput. Human Behav. 2015, 48, 542–547. [Google Scholar] [CrossRef]
  80. Bouchard, S.; St-Jacques, J.; Robillard, G.; Renaud, P. Anxiety Increases the Feeling of Presence in Virtual Reality. In Presence: Teleoperators and Virtual Environments; MIT Press: Cmbridge, UK, 2008; pp. 376–391. [Google Scholar] [CrossRef]
  81. Chittaro, L.; Ranon, R. Serious Games for Training Occupants of a Building in Personal Fire Safety Skills. In Proceedings of the 2009 Conference in Games and Virtual Worlds for Serious Applications, Coventry, UK, 23–24 March 2009; pp. 76–83. [Google Scholar] [CrossRef]
  82. Won, A.S.; Friend, M.; Perone, B.; Bailenson, J.N. Identifying Anxiety Through Tracked Head Movemnts in a Virtual Classroom. Cyberpsychol. Behav. Soc. Netw. 2016, 19, 380–387. [Google Scholar] [CrossRef] [PubMed]
  83. Werbach, K.; Hunter, D. The Gamification Toolkit: Dynamics, Mechanics, and Components for the Win; Wharton Digital Press: Philadelhia, PA, USA, 2015. [Google Scholar]
  84. Llorca-Bofí, J.; Redondo, E.; Vorlaender, M. Learning Room Acoustics by Design: A Project-Based Experience. Int. J. Eng. Educ. 2019, 35, 417–423. [Google Scholar] [CrossRef]
  85. Valerie Jane, G. Measures of Situational Awareness; CRC Press: Boca Raton, FL, USA, 2019. [Google Scholar] [CrossRef]
  86. Cavalcanti, J.; Soares, M. Ergonomic Analysis of Safety Signs: A Focus of Informational and Cultural Ergonomics. Work 2012, 41 (Suppl. 1), 3427–3432. [Google Scholar] [CrossRef] [Green Version]
  87. Stanney, K.; Fidopiastis, C.; Foster, L. Virtual Reality Is Sexist: But It Does Not Have to Be. Front. Robot. AI 2020, 7, 4. [Google Scholar] [CrossRef] [Green Version]
  88. Castelli, L.; Latini Corazzini, L.; Geminiani, G.C. Spatial Navigation in Large-Scale Virtual Environments: Gender Differences in Survey Tasks. Comput. Human Behav. 2008, 24, 1643–1667. [Google Scholar] [CrossRef]
Figure 1. Scheme of immersive virtual reality components.
Figure 1. Scheme of immersive virtual reality components.
Sensors 21 04663 g001
Figure 2. Percentage of age range.
Figure 2. Percentage of age range.
Sensors 21 04663 g002
Figure 3. Use frequency of 3D platform.
Figure 3. Use frequency of 3D platform.
Sensors 21 04663 g003
Figure 4. Pre-experiment training room.
Figure 4. Pre-experiment training room.
Sensors 21 04663 g004
Figure 5. Hazard distribution, route and educational signs in the principal maze.
Figure 5. Hazard distribution, route and educational signs in the principal maze.
Sensors 21 04663 g005
Figure 6. Example of hazards.
Figure 6. Example of hazards.
Sensors 21 04663 g006
Figure 7. Screenshot of a video recording showing user’s path.
Figure 7. Screenshot of a video recording showing user’s path.
Sensors 21 04663 g007
Figure 8. Simulator Sickness.
Figure 8. Simulator Sickness.
Sensors 21 04663 g008
Table 1. Summary of the metrics.
Table 1. Summary of the metrics.
Dependent MeasuresStageResearch Tools
Behavioral IntentionsPretestQuestionnaires
Perception Times; Response; Reaction Time; Behavior; Compliance; Accuracy; Selective and Divided AttentionDuring the gameNavigation data, physical reactions, behavioral evidence
Memory; Ratings of Perceived Appropriateness; Qualitative SatisfactionPost-experimentQuestionnaire ratings; Bipolar laddering
Table 2. Signals used in Game for Safety.
Table 2. Signals used in Game for Safety.
Static SignalDynamic VariableSmart Variable
Sensors 21 04663 i001Intermittent LEDs at the extremes of the poster.
Intermittent LEDs at the extremes of each cell information
Sensors 21 04663 i002Intermittent white color LEDs
Intermittent yellow color LEDs
Intermittent red color LEDs
Sensors 21 04663 i003Intermittent white color LEDs at the postersPoster and warning band appear when user advances until the danger
Sensors 21 04663 i004Intermittent LED lights around text
Intermittent LED lights around pictograms
Sensors 21 04663 i005Intermittent LED lights on the arrowSignal appears when user advances until the danger
Table 3. BLA positive common (PCx), negative common (NCx), positive particular (PPx), and negative particular (NPx) elements of the game and sensor warning variables.
Table 3. BLA positive common (PCx), negative common (NCx), positive particular (PPx), and negative particular (NPx) elements of the game and sensor warning variables.
Item IDDescriptionAv. Score
Mention Index (MI) %
PC1Learning without risk8.4050
PC2Assessment of decision making9.7060
PC5Virtual environment8.5020
PC7Time x Task8.5040
PC10LED warning10.0020
PP1Warning pictograms10.0010
PP4No aggressive/Pleasant experience6.0010
NC1Get used to technology6.0020
NC3Fire extinguisher command2.5020
NC5Fire extinguisher usability5.0020
NC8Bonus time2.5020
NC9Graphic environment2.5020
NP2Time information4.0010
NP3Warnings with texts6.0010
NP6Environmental Lay-out4.0010
Table 4. Common and particular solutions identified for negative and positive elements.
Table 4. Common and particular solutions identified for negative and positive elements.
Item IDDescriptionMention Index (MI) %
CS1Longer experience20
CS2Adjust interaction to user profile40
CS3More initial information20
CS4Change time display20
CS5Reduce elements40
CS6Improve layout by area (colors, elements, events)30
CS7Less text on warnings20
PS1Limit time for fire extinguisher10
PS2Another level10
PS3Improve speed10
PS4More difficult to access fire extinguisher10
PS5More severe negative effect10
PS6More stressful elements10
PS8Change the water effect10
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Cavalcanti, J.; Valls, V.; Contero, M.; Fonseca, D. Gamification and Hazard Communication in Virtual Reality: A Qualitative Study. Sensors 2021, 21, 4663.

AMA Style

Cavalcanti J, Valls V, Contero M, Fonseca D. Gamification and Hazard Communication in Virtual Reality: A Qualitative Study. Sensors. 2021; 21(14):4663.

Chicago/Turabian Style

Cavalcanti, Janaina, Victor Valls, Manuel Contero, and David Fonseca. 2021. "Gamification and Hazard Communication in Virtual Reality: A Qualitative Study" Sensors 21, no. 14: 4663.

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop