3.1. Research Question 1: Does MSP Challenge 2050 Offer a Platform for Participants to Learn about MSP by Helping Them Understand Information Derived from Data, Analyses, and Models? (i.e., Individual Added Value)
A small sample of the different interactions between participants during the game can be found in Table 2
below. It identifies how these interactions show that the game helped participants’ understanding of data, analysis, and models, and how they relate to the knowledge co-creation cycle. There were three main types of interactions recorded that totaled 102 interactions among the teams. The first involved players not understanding a certain word or concept presented in the game and asking their teammates for additional information (see, for example, 0:03:32, 0:18:00, and 0:21:40). It represented close to a quarter of interactions recorded (24.5%). This is externalization of knowledge that helps the participants understand the data and the model in the game. Another type of interaction observed involved players helping each other understand features of the game (see 0:09:30, 0:11:09, 1:24:40, and 1:27:40). These types of interactions represented the bulk (51%) of the interactions between players. These interactions helped the players understand the model in the game and corresponds to socialization, because they would not have had access to this knowledge had they not been experiencing the game together. The last major type of interaction recorded happened when teams extrapolated beyond the scope of the game to reflect on how their plans should be implemented in a real-world setting (see 0:07:30, 0:39:40, 0:45:00, and 1:00:10). These interactions are examples of analysis within the teams and the combination of knowledge provided by the game to apply it to more complex and realistic situations. This last type of interaction accounted for close to a quarter of interactions (24.5%) and mostly occurred when the teams discussed how to present their plan to the G.O.D.
The post-game survey questions and answers relating to learning outcomes from the three game events are presented in Table 3
below. The possible answers that participants chose from range from 1 (strongly disagree) to 5 (strongly agree), where a 3 is neutral, meaning the player neither agreed nor disagreed with the statement. The answers have been grouped based on the players’ experiences working in MSP. The data is divided by experience because statistical analysis (i.e., Mann-Whitney U test) revealed that prior experience in MSP most influenced the players’ answers. The participants had for choice: Less than a year, from one to two years, two to three years, three to five years, five to 10 years, and 10 or more years of experience working in MSP. When unsure, the participants were told to round down their level of experience. One column presents the average answer for players with less than two years of experience (less than two years), and the other for players with two years or more (two years and more). The first three questions ask the players how much they have learned about MSP from the game artefact, which corresponds to externalization of knowledge. A Mann-Whitney U test on the first three questions indicated that the players with less experience of MSP learnt significantly more about MSP from playing the game than the players with more experience (U = 116, p = 0.05). That being said, both cohorts chose, on average, an answer above “neutral”, meaning that they report some learning about MSP from playing the game. Questions 3 to 7 pertain to the social aspect of playing the game, or the socialization part of the knowledge co-creation cycle. The questions relate to the interactions between the players, which acted as practitioners of MSP. There was no statistical difference between the answers of the two cohorts for these questions (U = 116, p = 0.05), with averages above 3 (“neutral”). Furthermore, the players agreed that the game helped them understand the different barriers to the development of a good MSP process (question 5), which can be linked to the combination of knowledge about different systems involved in MSP and how they can conflict with each other. Finally, survey question 8 asks the players about internal reflection throughout the game, which corresponds to an internalization of knowledge and information provided by the game. Although there is no statistical difference between the two cohorts for this answer, it is important to note that participants from the Copenhagen event scored significantly lower (mean = 3.05, SD = 0.94) on this question than players from Newfoundland (mean = 3.67, SD = 0.50, U = 48, p = 0.05) and Venice (mean = 4.20, SD = 0.77, U = 90, p = 0.05). The post-game survey written responses of the Copenhagen event indicate that the players wanted the game to be based on realistic events and conflicts. Two players suggested introducing different planning scenarios that would be specific to a region. Another player suggested using a real case study in the game. Finally, three players thought that the game was too focused on national planning and that there was not enough time for transnational planning. Overall, the interaction analysis and the post-game survey answers showed that playing MSP Challenge 2050 helped players with little work experience in MSP better understand the challenges of MSP and the game helped facilitate the MSP process within each team.
The answers from the mid-game survey of the Newfoundland game event indicate that the teams were struggling to define their goals because of the overwhelming amount of data the game provided them. Team Indigo admitted that they did not know “How to geographically spread out some of the new facilities”. To help work through this cognitive loading, the teams identified different strategies, such as “Us[ing] a more integrated approach” and “leaving on [the] most important layers (things that can’t be moved) to plan our work” (team Indigo), “Taking on specific roles” (team Orange), and “Dividing tasks” (team Purple). These findings demonstrate that although the teams were overwhelmed by the amount of data at the end of the planning phase, they developed strategies to redress the situation, such as: Integrated planning, limiting the amount of information on the screen to only show what they deemed most important, and dividing the specific roles and tasks between the players within a team to limit information overload. The game was therefore a good learning tool that helped the players develop skills to adjust and adapt to working with large data sets required for MSP.
3.2. Research Question 2: Does playing MSP Challenge 2050 Promote Quality Interactions and Cooperation Between Participants Facilitating the Knowledge Co-creation Cycle? (i.e., Group Added Value)
First, from the videos of the Newfoundland event, a quantitative analysis of the interactions in each team was performed for the planning phase of the game (defined as the first 90 min of gameplay where the time in the game did not advance). To graphically represent the data, this 90-min timespan was divided into nine sub-phases that show how interactions changed over the course of the planning phase. Due to increased movement of players between teams after the planning phase, interactions past this point could not be counted. Table 4
below shows the quantitative evolution of interactions for each team during the initial planning phase of the MSP Challenge 2050 simulation. There are three points to keep in mind when looking at Table 4
below. The first is that the numbers represent the total number of interactions, although it should be noted that some teams had less players than others. Having one less player in a team surprisingly did not affect the number of interactions. Interaction analysis reveals that this could be because the teams with three players only used one computer instead of two and therefore had to communicate with each other more than a team that used two computers. The player on the computer tended to speak less than the others and instead take instructions from other teammates. Second, the bolded numbers represent moments where the team experienced technical difficulty and a facilitator had to come and reboot their computer. A technical difficulty is always accompanied by a dip in the number of interactions (see teams, OR, IND, YEL). A third thing to note is that each team met with G.O.D. individually towards the end of the first 90 min of gameplay to go over their country plans. These meetings took between 10–15 min, represented in the table as G.O.D.
From the interaction analysis, it was noted that after meeting with G.O.D., team members were more stressed and retreated into their chosen roles, which appeared to isolate them from other players within their team. Therefore, the number of interactions dropped as players focused their attention on changing their national plans to match G.O.D.’s demands before discussing with players with similar roles in other teams how to coordinate on an international scale. With that in mind, the table shows that interactions within teams are constantly changing, but that interactions stay relatively high throughout the simulation gaming event.
The most important information coming from the quantitative interaction analysis is that MSP Challenge 2050 fosters numerous interactions between teammates during the planning phases. These interactions are necessary for the socialization stage of the knowledge co-creation cycle. Furthermore, it shows that the main hindrance to interactions throughout the game are technical difficulties. Therefore, the utmost care and caution must be exercised when planning these events to reduce instances of technical difficulties to promote the greatest amount of interactions. However, looking at quantity is not enough to come to definitive conclusions, instead, the nature and quality of these interactions must be evaluated. To do this, five qualitative indicators were developed for this research to describe the quality of the interactions. These indicators are described in Table 5
and the results are tallied in Figure 2
Once again, it is important to note that team Indigo and team Red had three players while the other teams had four players. Furthermore, interactions between teams during the planning phase were recorded though minimal in number (less than 10% of total interactions). The results for each indicator are similar for each team, with one outlier that outperformed the other teams in each category. Moments of explicit knowledge transfer and of reflection were the most common type of interaction between teams making up 22% and 37% of the quality interactions, respectively. This supports the hypothesis that playing MSP Challenge 2050 can lead to knowledge co-creation through explicit knowledge transfer and combination. Going back to the first research question, it also supports the idea that this SG could be used as a PSS providing added value while it helps participants learn to interpret the data, models, and analysis through interactions that lead to explicit knowledge transfer and reflection. These findings show that MSP Challenge 2050 can serve as a learning tool and as an innovative tool to support planning.
As can be noted by looking at Table 5
above, a great deal of interactions occurred between players and teammates; however, when looking at Figure 2
above, one can see a great deal less in terms of quality interactions. This is not to say that the game is ineffective at fostering quality interactions, rather, a large number of interactions are required to support these quality interactions. Previous research has shown similar results using a different SG (see [36
]). Certain quality interactions are made of multiple smaller interactions, which can also explain the discrepancy in occurrence. Survey results from the mid- and post-game questionnaires were also analyzed to determine how the players felt the game helped them collaborate. The answers from the mid-game survey of the Newfoundland game event show that most teams identified team cohesion and collaboration as being helpful to develop and achieve MSP. Teams reported that much of their strategies relied on “lots of communication” (team Red), and “delegation, cooperation” (team Purple). When asked which strategies they were using to develop their MSP, team Yellow responded by saying: “Focus on communication between teams and between different ministers”, while team Orange reported that: “[They] are still having fun working together easily and have chosen some key roles”.
The answers to the mid-game survey indicate that teams valued cooperation during the game. The answers from the post-game survey in Table 6
below reinforce the findings of the mid-game survey and the interaction analysis, i.e. that cooperation took place between participants during the game event. Table 6
summarizes answers from the post-game survey from all three events to questions about the level of collaboration within teams. As before, the possible answers range from 0 (strongly disagree) to 5 (strongly agree), where a 3 is neutral. The Newfoundland and Venice groups ranked the questions similarly (U = 34, p = 0.05). However, we see a statistically significant dip in the values for the Copenhagen group for the second question relating to how well players worked together (UV
= 90, UN
= 48, p = 0.05). This may be attributed to the fact that for both the Newfoundland and the Venice event, the players were mostly students who knew each other well prior to the game, while the Copenhagen event was attended mostly by MSP professionals with more distant or non-existent relationships. The standard deviation for these questions remains under 1, meaning that most participants shared a similar experience. The high number of interactions and the teams’ focus on collaboration indicates a potential for MSP Challenge 2050 to promote knowledge co-creation through socialization, externalization, and combination. In fact, as noted above, players identified communication as an important strategy for the next steps of the game. Overall, the game promoted ongoing collaboration within teams, despite each player choosing a distinct role for themselves.
The key insights from the debriefing session revolved around economic considerations, collaboration between different teams, and the feasibility of the task given to them. The players reflected on the complexity of MSP, which was made easier by removing economic components during the game, something the players were thankful for, but that removed from the realism of the task given to them. Even without economic considerations, the teams acknowledged that they had to make tradeoffs between different tasks and prioritize certain goals. In response to the complexity of MSP and learning to make decisions in a complex system, player RED2 noted that “It’s not that people are useless—interactions between good and moral people make weird outcomes”. The facilitator reflected on this statement by explaining that political and research cycles have different lengths, therefore, planners often find themselves trying to make decisions that fit within the short span of a political cycle without necessarily having as much scientific research to help them as they would like. This is a reality that most planners deal with, and the game illustrated that reality. From the interaction analysis of the debriefing session, it seems that the complexity of the subject mixed with the fast-paced game allowed some participants to reflect critically on what was happening, but that the game process could be improved to allow for more space for critical thinking. An interesting development occurred during the Newfoundland event, which can be explained by looking at the interaction analysis of the game event and the debrief. A feature of the game requires teams to make their plans visible to other teams to receive feedback and approval. As the game currently stands, it is suggested that teams seek approval from other teams before implementing plans in shared waters (based on the Kyiv (SEA) protocol that states that Member States must notify and consult each other on all major projects under construction that may have adverse environmental impacts across borders). However, the teams realized, one by one, that they did not need to wait for approval to go ahead with implementation of their plans (see Table 7
The role and importance of this feature of the game will be further explored in the Discussion Section. Overall the interaction analysis showed that players helped each other understand the terms used by the game, the game itself, as well as reflected on how their plans could be implemented in the real world.
3.3. Research Question 3: What are the Characteristics of the Plans Developed While Playing the Game and How Do the Plans Differ From Team-to-team? (i.e., Outcome Added Value)
To assess the outcome of added value of MSP Challenge 2050, the national plans developed by each team during the planning phase provide valuable information. Table 8
below provides an overview of the national plans developed by each team in the first 90 min of game-play and presented to G.O.D. The plans below are divided into five sectors: Ecology, fishing and recreation, oil and gas, renewable energy, shipping, and an additional miscellaneous category.
When looking at the above plans, some teams’ plans are more comprehensive than others. For example, the Orange team planned for all five sectors as well as having miscellaneous goals. Similarly, the Purple team planned for all sectors, save shipping, although they managed to include timelines for most of their plans. Team Indigo also created a rather comprehensive plan. On the other end of the spectrum, we see that the Yellow and Red teams’ plans are rather lacking, with the Yellow team failing to plan for two of the sectors while the Red team planned for all sectors, though vaguely.
Comparing the quantitative and quality interactions obtained in question 2 shows how these interactions may have supported the development of their plans. Table 9
below shows the total quality and quantitative interactions for all teams.
above shows that the teams with more comprehensive plans interacted more throughout the game event. These teams generally had more quality interactions than other teams. The exception here being the Orange team that had less interactions (both quantitative and quality). However, it should be noted that for two phases, team Orange was out of view of the recording devices, and the team interacted more than the numbers show. In fact, in their mid-game survey, they reported “we looked at everything as a team, looked at what is missing, what needs more information, to make sure to talk with key partners to avoid conflict.” The Yellow team is also an exception, with the second highest quality interactions, yet they still had an underdeveloped plan. Despite this, the Yellow team noted in their mid-game survey that one of the challenges they faced was “confusion over how to prioritize things”, which may have led to confusion over how to develop their national plan. The Yellow team did, however, have the most instances of knowledge sharing, meaning that although their MSP plans may not have been as comprehensive as most, there was the highest exchange of knowledge occurring.
The outcome added value does not end at the planning phase, it extends into the implementation phase. Although not recorded, teams discussed the implementation phase during the debriefing period, which was recorded as well. The teams discussed an example of international cooperation that occurred during the implementation phase, namely the international Sea of Colours Energy Grid that was a joint creation by all teams. Proposed by the Orange team, a so-called “international summit” was held where representatives from each country were brought together. As a result of this summit, all five countries successfully linked their energy grids in a hub in the center of the Sea of Colours. The Orange team was composed of one student, two local stakeholders, and one postdoctoral fellow; it was the team with the most combined experience. The other teams benefitted from their expertise and were able to learn from their experience and create new knowledge between themselves, resulting in a successfully implemented Energy Grid for the Sea of Colours.
3.4. Summary of Research Results
Though the results and findings for each individual research question are essential to achieve the research objective, it is also important to look at the inter-connections between these questions. An important aspect of this study was to investigate the added values of MSP Challenge 2050 as defined by Pelzer, Geertman, van der Heijden, and Rouwette [14
]. Namely, our research questions sought to investigate the individual, group, and outcome added values. By isolating these added values, we add credence to the idea that SG (specifically MSP Challenge 2050) can be used as PSS.
The findings from the first research question allowed for a deeper investigation into the individual added value that PSS can bring out in users. Results and findings relating to this first question indicated that by playing MSP Challenge 2050, participants are given the opportunity to acquire knowledge and information from the game artefact and game experience itself. The results have shown that this opportunity is greater for participants with less experience and formal training in MSP, though, generally, the game experience proved advantageous to all participants. The second research question allowed the researchers to investigate the added group benefits as defined by Pelzer, Geertman, van der Heijden, and Rouwette [14
]. A quantitative analysis shows that players are offered a venue in which interactions are common and stable (unless teams experienced technical difficulty). Furthermore, an analysis of quality interactions has shown that knowledge between participants is exchanged either through explicit knowledge transfer or anecdotal exchange. The game experience also fosters moments of reflection, allowing for players to come together and reflect on the information being transmitted to them via the game artefact that was determined in research question 1. This knowledge transfer and reflection therefore allows for the different stages of knowledge co-creation to take place. Table 10
shows the stages of the knowledge conversion cycle and summarizes if and how these stages occurred during the MSP Challenge 2050 game event.
Finally, research question 3 investigated the added benefits of the outcome level by analyzing the plans that were developed by teams throughout the planning process. These plans were informed by both the individual and group levels as they are a synthesis of all the information and knowledge developed by players throughout the planning stage of the game. Findings from this question indicate that teams that interacted more both in quantity and quality generally created more thorough MSP plans for their respective countries. Overall, it can be concluded that the MSP Challenge 2050 SG displays the added values required for a PSS on an individual, group, and outcome level. It offers opportunities for interaction and discussion within and between teams. This interaction and discussion increase the chances of creating new tacit or explicit knowledge on an individual and group level. The results of this study are preliminary, although this research provides insight into further research opportunities on how SG can effectively be used as a PSS or how PSS can be gamified to allow for learning outcomes that may not be present in traditional PSS.