6.1. Student Evaluations
We were primarily interested in measuring changes in (1) players’ opinions of the importance of spatial thinking skills and (2) their confidence in their own spatial abilities when evaluating the game. The two changes were assessed via a pre- and post-game experience questionnaire, a common technique used in game evaluation [43
]. The study group was comprised of 10 students, of which 50% was male and 50% was female, who were graduate or undergraduate students. This sample size that has been used in past GIS disaster management task assessments [41
]. The pre- and post-game experience questionnaire allowed participants to self-rate specific spatial thinking skills using a five-point Likert scale of 1–5 (1 = strongly disagree; 2 = disagree; 3 = neutral; 4 = agree; 5 = strongly agree) via four questions:
Q1: I am confident in my ability to navigate an unfamiliar area.
Q2: I am confident in my ability to navigate without my phone or using a paper map.
Q3: I am confident in my ability to translate verbal directions into visual directions.
Q4: I am confident in my ability to find an alternate route or to reroute when a path is obstructed.
graphically represents the summary results of the pre- and post-game questionnaire related to these four questions, which we discuss next along with other results.
As seen in Figure 6
, results from the preliminary questionnaire generally indicated a lack of confidence in navigations skills in terms of responses ranging from strongly disagree to neutral, as particularly seen is responses to question 2 (I am confident in my ability to navigate without my phone or using a paper map
) and question 4 (I am confident in my ability to find an alternate route or to reroute when a path is obstructed
). In addition, 90% of the study group self-identified that they had no previous experience with spatial thinking skills before participating in the study.
Qualitative assessment of the game was done through play testing and “think aloud” method, a technique that has been used in past serious game evaluation research [26
]. Players were debriefed about the project goals and given an overview of the gameplay in terms of the Dickinson disaster response scenario during Hurricane Harvey. Then, they were asked to play through the game and verbalize their thoughts about what they were experiencing. A facilitator was present to answer any questions and encourage users to reflect on their reasoning as they carried out the missions. The game play and think aloud dialogs were screen-recorded for follow-up analysis.
Qualitative assessment of the recorded game play and spatial think aloud data indicated that the game helped players become aware of their spatial awareness skills, but that the level of spatial thinking challenges actually needed to increase. The qualitative assessment also revealed insight into accessibility issues, game mechanics, and the extent to which players used spatial thinking. For example, one player commented that the lack of beacons made it difficult to determine their location within Dickinson.
Despite these challenges, however, most of the users were able to orient themselves within the game environment using the provided map, as seen in Figure 4
b. One player completed all of the missions on the second day by cutting through the fields, using the map to keep track of how many roads they had travelled. They expressed that the timer drove them to take shortcuts with the boat in order to reach people in time, but felt that there should be more obstacles to really engage the player with the gameplay. Conversely, another player claimed to have “no sense of direction” and that “our culture” is not really concerned with teaching people navigation skills. They had trouble navigating the scenario, and took longer than other players to complete the missions.
Various play testers expressed their appreciation of the landmarks, a common item used for navigation [45
]. They noted that the big, recognizable buildings offered them a sense of direction. Some of them used the high school and the highway as points of reference, looking for them whenever they got lost. Other strategies included using the stickers as placeholders for the character’s location and destination to help guide them in the right direction, as seen in Figure 4
b. They also expressed their concerns with navigating in the boat through areas of higher elevation to reach lily pads.
Finally, we gained insight into how the players viewed their ability to think spatially from the results of the post-questionnaire. The post-questionnaire asked the same questions as the preliminary questionnaire, as well as three additional questions about gameplay and three questions about the game’s relationship with disaster resilience. As seen in Figure 6
, player confidence in their ability to navigate an unfamiliar area after the test increased (question 1) from 60% indicating that they agree or strongly agree that they are confident to 70%. Additionally, player confidence in their ability to navigate without their phone or using a paper map (question 2) increased from just 20% indicating that they agree or strongly agree that they are confident to 50%. The feedback from question 2 was useful because players indicated that they felt the weakest in this area of spatial thinking on the pre-questionnaire. Player confidence in their ability to translate verbal directions into visual directions (question 3) showed an interesting result of a slight decrease from 55% indicating that they agree or strong agree that they are confident to 40%, although one respondent indicated a strong increase in confidence after conducting the evaluation. Question 4 indicated that confidence increased in a player’s ability to find an alternate route or reroute when a path is obstructed (question 4), with 60% indicating that they agree or strongly agree that they are confident after playing Project Lily Pad. The increase in self-reported spatial ability confidence, as seen in changes to responses to questions 1, 2, and 4, reflect results from previous research on how providing visual, interactive, and geographic environments (games or otherwise) can improve spatial reasoning and decision making generally, but also for disaster management tasks specifically [23
]. The slight dual decrease and increase in confidence in translating verbal directions into visual direction (question 3) could be explained by issues with design of the verbal directions themselves or the general challenge of navigation derived from spatial cognition of environments based on external representations, such as language, maps, and text [47
The additional post-game experience questions also indicated that 50% of players thought that the game made them think more about how they navigated their environment. Furthermore, 90% of the players indicated the game made them think more about what they would do in a disaster and 70% of the game players indicated the game made them think more about how disaster response actually works in practice, results that generally reflect the established benefits serious games have shown for increasing disaster management domain knowledge [9
]. Additionally, as discussed in Section 2
, Harvey survivors we interviewed felt skeptical and abandoned by official response agencies. The fact that the Project Lily Pad game gave players a deeper understanding of disaster response could indicate how Project Lily Pad can potentially be used as a learning device to empower local communities to make learning about disaster response accessible and more realistic, and how they can use these training experiences if disasters were to occur again in their communities.
6.2. Practicioner Evaluations
The Project Lily Pad game was also evaluated by the emergency management professionals from the Galveston County Office of Emergency Management, who were actual 2017 Hurricane Harvey responders. These evaluations were more broadly focused on the utility of Project Lily Pad for emergency management practice, as serious games have a well-established record of being used for emergency management practitioner training [9
]. Two emergency management professionals spent one hour each playing Project Lily Pad and provided feedback using the think aloud method (Figure 7
Although the small sample size was small, the ecological validity of Project Lily Pad being reviewed by the actual people who were the motivation for the game’s creation shed valuable insight into the utility of the game for emergency management practice. Specific game design feedback included introducing more realism based on actual disaster response experiences, for example making it more difficult to navigate the environment in terms of less reference point such as street signs; and better game interface controls and visibility for non-gamers, such as emergency management professionals, who would use the game.
The emergency management professionals also indicated that the game would be valuable to their practice in terms of having a game as a device to teach spatial thinking and awareness of a given environment for people that are new to a given disaster response area. For example, during Hurricane Harvey, several National Guard units were deployed to the Galveston County Emergency Operations center before being deployed into the community. Many of the National Guard members did not have any previous spatial knowledge of Galveston County or Dickinson in terms of the neighborhoods, streets, and overall spatial layout of the emergency response area. A game such as Project Lily Pad could have been used to teach National Guard and others about the area that they were going to operate in as they were waiting to be deployed from the emergency operations center.