Next Article in Journal
Characteristics and Trends of Ambient Ozone and Nitrogen Oxides at Urban, Suburban, and Rural Sites from 2011 to 2017 in Shenzhen, China
Previous Article in Journal
Dry Bulk Carrier Investment Selection through a Dual Group Decision Fusing Mechanism in the Green Supply Chain
Open AccessReview

On Evaluating Social Learning Outcomes of Serious Games to Collaboratively Address Sustainability Problems: A Literature Review

Department of Design, Production and Management, Faculty of Engineering Technology, University of Twente, Drienerlolaan 5, 7522NB Enschede, The Netherlands
*
Author to whom correspondence should be addressed.
Sustainability 2018, 10(12), 4529; https://doi.org/10.3390/su10124529
Received: 12 October 2018 / Revised: 19 November 2018 / Accepted: 26 November 2018 / Published: 1 December 2018
(This article belongs to the Section Sustainable Urban and Rural Development)

Abstract

Serious games are increasingly explored as collaborative tools to enhance social learning on sustainable management of land and natural resources. A systematic literature review was conducted to examine the current state of the art of the different methods and procedures used to assess social learning outcomes of collaborative serious games. Forty-two publications were identified and included in the review following study selection and quality assessment steps. Extracted data from the publications were categorized in relation to five research questions. Approaches that were used to assess cognitive, normative, and relational learning outcomes of collaborative serious games were subsequently identified based on the categorizations. As a result, these approaches distinguished between the nature of learning in the assessment of collaborative serious games. Combined, these approaches provide an overview of how to assess social learning outcomes of collaborative serious games, including the methods and procedures that can be used, and may serve as a reference for scholars designing and evaluating collaborative serious games.
Keywords: serious gaming; simulation gaming; sustainability; natural resources management; social learning; learning environment; assessment; evaluation serious gaming; simulation gaming; sustainability; natural resources management; social learning; learning environment; assessment; evaluation

1. Introduction

Environmental sustainability problems are typically complex and multi-scale, concern inherent uncertainty, and affect multiple stakeholders and agencies. Solving sustainability problems concerns the management of land and natural resources in a way that creates and maintains prosperous social, economic, and ecological systems [1]. To address such problems, decision-making needs to be adaptive to deal with the uncertainties and needs to include the diversity of knowledge and values of all affected stakeholders. To achieve this, scholars have advocated active experimentation and continuous evaluation, summarized as learning-by-doing, in natural resources management [2,3,4,5]. Central to these approaches is collaboration between and learning among researchers, resource managers, and resource users in order to find sustainable solutions [1,6,7,8].
Learning, in particular social learning, is therefore seen as a prominent driver and normative goal in natural resources management [2,9,10,11,12]. Although there is debate to find a commonly shared definition for social learning (see e.g., [3,11,13,14]), scholars increasingly agree that social learning has occurred when a change in understanding—related to for example the system, problem at hand, agreement, and collective action—is achieved through interaction in collaborative and participatory settings [11,13,15,16,17]. Social learning outcomes therefore require deliberative interactions, where multiple stakeholders work together and build relationships, which should ultimately lead to collective action [14,15,18]. Baird et al. [17] define three types of learning outcomes in relation to social learning: cognitive, the acquisition of new or restructuring of existing knowledge; normative, a shift in viewpoints, values or paradigms; and relational, an improved understanding of others’ mind-sets and enhanced trust and ability to cooperate between stakeholders.
Serious games are increasingly explored as a method to establish social learning on sustainable natural resources management and urban planning [19,20,21,22]. Serious games are generally referenced to as games that have a primary purpose other than entertainment, such as educating, training or informing players [23,24,25]. More specifically towards policy-making, Mayer ([26], p. 825) defines games as “experi(m)ent(i)al, rule-based, interactive environments, where players learn by taking actions and by experiencing their effects through feedback mechanisms that are deliberately built into and around the game”. A recognized strength of such serious games is that games can include both the techno-physical complexity—the underlying physical elements of the system and its uncertainties—and the socio-political complexity—the strategic interactions between stakeholders in the policy arena—by combining role-play with in-game feedback mechanisms [20,21,26,27]. Therefore, serious games fit well with the learning-by-doing approach in natural resources management; serious games offer stakeholders a place to negotiate, deliberate, exchange perspectives used in decision-making and learn about the trade-offs between decisions in the safe experimentation environment of a game [26,27]. Moreover, multiplayer and multi-role serious games thereby offer the collaborative and participatory stakeholder interactions that are needed to establish social learning [20,22]. The term collaborative serious games is used in the remainder of the paper to refer to serious games that follow the above definition by Mayer [26], but are particularly focused to collaboratively—through mutual engagement of stakeholders in a coordinated effort to solve the problem ([28], p. 70)—explore sustainable management strategies.
The assumption is that any learning that occurs from playing collaborative serious games is transferable to the world outside the game [26,29]. A collaborative serious game can therefore be seen as an intervention or a transitional object [29] that may lead to social learning outcomes. While calls exist for more systematic assessment of learning through serious games (see e.g., [30]), assessing social learning outcomes of collaborative serious games is challenging. On a practical level, gathering data occurs in a collaborative setting that introduces many confounding variables such as players’ prior relations, players’ attitudes towards serious gaming, and facilitators who have to make decisions on the spot while guiding sessions [30,31,32]. On a higher level, the lack of consensus on the definition of social learning itself has made it difficult to assess [8], and indeed few studies empirically and directly assess the learning effects of interventions [3,12,17].
The research presented in this paper adds to this gap by answering the main research question: What is the current state of the art of the different methods and procedures used to assess social learning outcomes of collaborative serious games? To this end, a systematic literature review was conducted to survey the empirical assessment of social learning through collaborative serious games. As such, the contributions of this paper include: (1) summarizing and categorizing different evaluation procedures applied and methods used to assess social learning outcomes of collaborative serious games; and (2) presenting a state-of-the-art overview of approaches to assess social learning outcomes of collaborative serious games.
The next section presents the method of the systematic review and describes all the steps taken in detail. Section 3 and Section 4 cover the search and categorization results of the review, respectively. Section 5 discusses approaches used to assess cognitive, normative, and relational learning outcomes of collaborative serious games. Section 6 finally sums up the conclusions drawn from the review and provides an overview of the identified assessment approaches.

2. Materials and Methods

To perform the literature review, the guidelines by Kitchenham and Charters [33]—an updated version of Kitchenham [34]—were used. These guidelines are an established procedure for conducting systematic reviews, particularly in software engineering. Although collaborative serious games are not exclusively digital—a board game can also be used to explore sustainable management solutions—the use of these guidelines is appropriate as many serious games do make use of software and the guidelines are based on guidelines used in other disciplines.
Kitchenham and Charters [33] prescribe three phases of a systematic review: planning, conducting, and reporting. The planning phase covers confirming the need for a review, defining the research questions, and producing a review protocol. The conducting phase involves developing a search strategy, executing as well as documenting the search, performing a study selection, and extracting data relevant to the research questions. The reporting phase then covers reporting and disseminating the results of the study. In the next subsections, all the steps taken in performing the systematic literature review are described.

2.1. Related Work

As a starting point in the study, a search was executed to confirm the need for a review. One literature review by Calderón and Ruiz [35] on the different methods and procedures used to evaluate serious games was identified in this step. Notably, the serious games included in this review are mostly evaluated using a questionnaire as the main assessment method and are mostly evaluated on the educational effectiveness—defined by Calderón and Ruiz [35] as the learning outcomes, usability, and user’s experience. The study is similar in terms of its scope and research questions covered, but lacks the focus on social learning and the policy setting. The study was therefore used as a reference for the overall search strategy, such as to sharpen the research questions and to formulate inclusion and exclusion criteria. Other reviews, both literature and meta reviews, focused on the effectiveness of serious games [36,37,38,39,40], not on the assessment procedures.
In addition to these studies, five review papers were identified which focus on serious games in relation to sustainable development [41], climate change [42,43,44], and sustainable water resources management [45]. While these do not review the assessment approaches of learning outcomes, the serious games reviewed in these papers do all relate to sustainability and may therefore be of interest to this review. These studies were therefore used to inform the search strategy, for example to define keywords, and as snowball resources.
Concluding, no review was found that focuses specifically on evaluating collaborative serious games and this study therefore adds an in-depth review with this explicit focus to the literature.

2.2. Research Questions

To examine the current state of the art of the different methods and procedures used to assess social learning outcomes of collaborative serious games, the following research questions were addressed to analyse publications:
RQ1.
How is learning through collaborative serious games conceptualized?
RQ2.
When is data collected in the evaluation of learning through collaborative serious games?
RQ3.
What methods are used in the evaluation of learning through collaborative serious games?
RQ4.
Do evaluations of learning through collaborative serious games use quantitative, qualitative or a combination between quantitative and qualitative data?
RQ5.
What are the learning effects of collaborative serious games according to their evaluations in relation to social learning?
Learning is used over social learning in these questions to not exclude publications that use a different learning conceptualization but do fall within the scope of this review. For all research questions, data was extracted from the publications and subsequently categorized, which is further explained in the data extraction and data analysis Section 2.7 and Section 2.8.

2.3. Search Strategy

The search strategy defines the systematic approach to identify publications to include in the literature review. Two categories of keywords were used to create search combinations. The first category (A) covered three keywords to identify publications that covered a game-based approach: serious gam* (A1), simulation gam* (A2), and policy gam* (A3). The asterisk was used in each keyword to cover for both game and gaming. Simulation and policy gaming were added to the search terms as pilot searches showed this term, unlike alternatives such as applied gaming, returned search results that were relevant to the scope.
The second category (B) covered 18 keywords to identify publications in relation to sustainability and disciplines dealing with sustainability issues: sustainability (B1), sustainable use (B2), sustainable development (B3), climate change (B4), climate adaptation (B5), environmental change (B6), energy transition (B7), adaptive *management (B8), collaborative management (B9), natural resource management (B10), environmental management (B11), spatial planning (B12), urban planning (B13), urban development (B14), water management (B15), water resources (B16), river management (B17), and river basin (B18). These keywords were selected based on the five identified review papers on games and sustainability [41,42,43,44,45], pilot searches, and publications covering collaborative serious games already known to the authors. The asterisk in adaptive *management was added to also cover for both comanagement and co-management.
The keywords were subsequently used in search strings to search for all possible combinations between one keyword from category A and one keyword from category B. Using Boolean statements, the search strings therefore covered “(A1 OR A2 OR A3) AND (B1 OR B2 … OR B18)”.
Searches were limited to publications published since 2007 to review the state of the art; 2007 was chosen in order to cover the last ten years and as pilot searches found few publications providing sufficient information for the scope of the review before 2007—consistent with the results of Calderón and Ruiz [35].
Six databases were similarly determined through pilot searches for conducting the review: ACM Digital Library, IEEE Xplore Digital Library, ISI Web of Science, ScienceDirect, Scopus and SpringerLink. Searches were executed on title, abstract, and keywords.

2.4. Study Selection

To narrow down the publications obtained following the search strategy, an initial selection and a final selection step were performed following inclusion and exclusion criteria (Table 1). In the initial selection, each study was reviewed based on its title, abstract, and keywords. This way, publications clearly not covering the scope of the literature review—i.e., a publication describing the economy and markets as a serious game—were excluded. In the final selection, the publications included in the initial selection were reviewed more rigorously according to the same criteria by scanning through each study and readings its conclusion. The entire study was read if necessary to determine inclusion or exclusion.

2.5. Study Quality Assessment

Following the study selection steps, the selected publications were looked at in more detail in relation to the research questions in a quality assessment step. Specifically, questions that can be answered with yes or no were formulated in relation to the research questions:
  • Does the publication provide a conceptualization of learning? (RQ1)
  • Does the publication state when data is collected in the evaluation of the collaborative serious game? (RQ2)
  • Does the publication discuss a method, technique or theory used to evaluate the collaborative serious game? (RQ3)
  • Does the publication discuss whether the evaluation used a quantitative or qualitative approach, or both in the evaluation of the collaborative serious game? (RQ4)
  • Does the publication describe learning outcomes of the collaborative serious game following evaluation? (RQ5)
The publications included in the study selection step were assessed on each of these questions as yes or no, corresponding to a 1 or 0 score. If a publication included all relevant aspects in regard to the scope of the review, it would therefore score five out of five. Publications that scored four or higher were deemed of sufficient quality and included in the review. Publications that scored exactly three, and were thus excluded, were checked again after completing the quality assessment to confirm exclusion was the correct decision.

2.6. Snowballing

To cover for relevant publications missed in the database searches, a snowballing approach was used to identify further relevant publications [46]. Firstly, backward snowballing was applied by identifying possible relevant publications from the reference lists of publications passing the quality assessment step. Secondly, forward snowballing was applied by checking all publications citing the publications deemed of sufficient quality using Google Scholar. Thirdly, the five review papers on sustainable development, climate change, and sustainable water resources management [41,42,43,44,45] identified earlier were used to snowball from as serious games covered in these reviews may be of interest to the scope of this review. All papers identified through snowballing were subsequently reviewed following the same study selection and quality assessment steps.

2.7. Data Extraction

In the data extraction step, all the publications included in the study selection and deemed of sufficient quality were read in its entirety. Data was extracted from the publications in regard to the research questions. All extracted data, for all publications, were stored in a spreadsheet. At this point, if publications were identified covering the same collaborative serious game without providing additional insight into its evaluation—i.e., a conference paper covering a limited evaluation and a journal paper covering a more extensive evaluation by the same authors—the publication with the lower-quality assessment was excluded from the review.

2.8. Data Analysis and Categorization

All data was subsequently categorized in regard to each research question. An inductive approach was used to create categories based on the data retrieved by identifying common themes, for example applied methods as surveyed in RQ3. On the first research question, for example, in regard to the conceptualization of learning, categories included social learning and experiential learning. For RQ5, the typology on social learning by Baird et al. [17] (see Table 2) was used as a framework to define categories that differentiate between the three types of learning. From an assessment perspective, the typology is beneficial as it views learning outcomes from their nature—cognitive, normative or relational—rather than their perceived value. Moreover, the typology separates relational learning as an explicit learning outcome, which is of particular interest in the multi-stakeholder context of sustainable land and natural resources management.
The categorizations were subsequently used to identify assessment approaches used in the evaluation of learning outcomes of collaborative serious games.

3. Search Results

The entire review, from developing the scope and performing the search, selection, data analysis, and synthesis, was executed between October 2017 and November 2018. Initial database searches were executed on or a few days before the 7th of January 2018, and all publications extracted through these searches were analysed. After processing all retrieved publications, all database searches were updated on the 7th of November 2018. Figure 1 provides an overview of the number of publications found and subsequently included after each step.
As Figure 1 shows, 78 publications retrieved through the database searches were included in the selection steps following the inclusion criteria, of which 41 publications were included following quality assessment. Sixty-six additional publications were identified through snowballing, 45 publications from the publications identified through database searches, and 21 from the five review papers [41,42,43,44,45]. Six of these 66 publications passed the same selection steps and quality assessment. Five publications were subsequently removed from the total publications as these covered the same collaborative serious game without providing additional insight into its evaluation. Forty-two publications were therefore included in the review.
Figure 2a shows an overview of how many publications included in the review were retrieved through each database as well as through snowballing. Some publications appeared in multiple databases and are added to these databases in the figure, explaining why the numbers added up are higher than 42. Scopus, as can be expected, provided the most publications included in the review. No publications retrieved from the IEEE database passed the selection criteria and quality assessment steps.
Figure 2b shows an overview of the publications included in the review in regard to their year of publication. The figure suggests an increasing trend in the use of collaborative serious games to explore sustainable management strategies. It must be noted that 2018 has the highest publications included in the review (10 publications), even though the review only covers the first 10 months of 2018.
Table 3 provides an overview of the publications included in the review and the game each publication evaluates.

4. Categorization Results

This section covers the categorized findings in relation to the formulated research questions. Through an inductive approach, between three and 13 categories were identified for each research question. A complete overview of all the publications and assigned categories can be found in Table A1 in the Appendix A. An overview of the underlying data can be found as a spreadsheet in the Supplementary Materials.
The next subsections present the results of categorization for each research question.

4.1. How Is Learning through Collaborative Serious Games Conceptualized?

The aim of this question was to obtain an overview of how learning through collaborative serious games is conceptualized. Thirteen categories were identified in the analysis. In most cases, publications mentioned the categories explicitly and included relevant citations, while in some cases publications lacked these citations. In the latter case, publications were still assigned to the category if a definition or description was included that matches the category definition. Only the categories social learning and experiential learning covered more than one or two publications. Table 4 provides the categories, their definitions, and the publications assigned to these categories. Categories covering only one or two publications are grouped into the other category in Table 4, but can be found in full in Table A1.

4.2. When is Data Collected in the Evaluation of Learning through Collaborative Serious Games?

The aim of this research question was to get an overview of when data is gathered in the assessment of learning through collaborative serious games. Categorizations were chosen based on game sessions as a reference point, with pre, during, post, and post-post data collection. Table 5 provides the categories, their definitions, and the publications assigned to these categories. Notably, 37 out of 42 publications gather data immediately after game sessions (post). Twenty-five of these 37 publications combine the post data collection with data collection during game sessions, while 21 of these combine it with a data collection before game sessions (pre) and 15 combine all three. Only 10 publications collected data well after game sessions (post-post).

4.3. What Methods are Used in the Evaluation of Learning through Collaborative Serious Games?

The aim of this research question was to obtain an overview of the methods and techniques that are used to evaluate collaborative serious games. Table 6 provides the categories used to represent different methods, their definitions, and the publications assigned to these categories. Thirty-one out of the 42 publications used multiple methods—combinations of the five most used methods; questionnaires, observations, debriefings, interviews, and data logging.

4.4. Do Evaluations of Learning through Collaborative Serious Games Use Quantitative, Qualitative or a Combination between Quantitative and Qualitative Data?

The aim of this research question was to see whether the evaluation of collaborative serious games relied on qualitative or quantitative or qualitative and quantitative data. Table 7 provides an overview of which publications based evaluations on either of these three categories. Thirty-seven out of the 42 publications used qualitative data. Sixteen of these 37 publications combined the qualitative data with quantitative data. Qualitative data resulted from questionnaires (open questions), observations (unstructured observations by researchers), debriefings, and interviews. Quantitative data on the other hand resulted from questionnaires (closed questions using Likert scales), observations (structured observations e.g., noting and counting certain behaviour), and data logging (e.g., player choices and group decisions).

4.5. What Are the Learning Effects of Applying Collaborative Serious Games According to Their Evaluations in Relation to Social Learning?

The aim of this research question was to get an overview of the learning outcomes of collaborative serious games. Categories were formulated based on the data extracted from the publications and in relation to the learning typology by Baird et al. [17]. Table 8 provides the categories, their definitions and relation to the social learning typology, and the publications assigned to these categories. Thirty-seven out of 42 publications report on cognitive learning outcomes, increased system understanding or raised awareness or both. Only five publications report that playing a collaborative serious game led to a change in views, the category linked to normative learning. Seventeen publications report on relational learning outcomes in the form of understanding other perspectives or building relationships and trust or both.
To formulate and assign categories for this research question, the choice was made to assign publications to categories based only on the text as written in the publications—the data—regardless of whether or not publications described their results as representative for the entire population or as highly or weakly significant. For example, Keijser et al. [65] and Lawrence and Haasnoot [74] both noted results related to the category increased system understanding although Keijser et al. only noted it for those less familiar with the topic, yet both publications were assigned to this category. Making distinctions would otherwise add a selection mechanism susceptible to subjective interpretations of the publications. The results as presented here should therefore only be viewed in relation to the learning outcomes of collaborative serious games can have in general and should not be used to draw direct conclusions on individual publications.

5. Discussion

This section covers the findings on the approaches to assess social learning outcomes of collaborative serious games. The findings are discussed in relation to the three types of learning as defined by Baird et al. [17]; cognitive, relational, and normative learning. In addition, as social learning should lead to collective action, the real-world impact of collaborative serious games—as reported in the reviewed publications—are discussed. The section ends with a reflection on and limitations of the review.
Forty-two publications were included in the review. In relation to the first research question, social learning and experiential learning are the only conceptualizations of learning used by more than two publications. While some publications provide learning conceptualization related to social learning—i.e., collective learning [49,63] and boundary crossing [48]—other publications provide conceptualizations that do not explicitly include the relational learning component of social learning. This is reflected in the review as the latter publications, as well as the publications that do not provide an identifiable conceptualization of learning, mostly report on cognitive learning outcomes. The next subsections discuss the assessments of cognitive, relational, and normative learning outcomes of collaborative serious games in detail.

5.1. Cognitive Learning

Cognitive learning relates to the acquisition of new or the restructuring of existing knowledge. Almost all of the publications in the review—37 out of 42—report on cognitive learning outcomes of collaborative serious games. Two categories were linked to cognitive learning, increased system understanding, and raised awareness, although both can relate to acquiring new knowledge and restructuring existing knowledge. Three common approaches were identified in the assessment: (1) self-reflective questions after game sessions; (2) pre-post measurements of self-reported knowledge on issues; and (3) observed acquisition or restructuring of knowledge.
Firstly, 21 publications describe an approach where participants were asked self-reflective questions in questionnaires, interviews or debriefings after game sessions, sometimes in combination with other methods [49,50,52,55,57,58,62,63,64,67,68,72,73,74,76,77,78,80,82,86,89]. For example, Dionnet et al. [77] assessed their TADLA approach, which aims to facilitate farmers in the collective modernization of their irrigation system, by combining a questionnaire after game sessions with follow-up interviews. They show that their approach helped farmers to acquire a more comprehensive understanding of how the different system components work together. Becu et al. [62] asked self-reflective questions on what local policymakers learned during their structured, round-table debriefing of LottoSim, a game to support social learning on coastal risk prevention measures. According to their evaluation, the participants especially learned about the dynamics of water expansion in flooding events as a result of the game’s integrated simulations. Moreover, the game helped participants to discuss long-term strategies, although participants did not pursue other prevention measures than those already applied in practice. Keijser et al. [65] in turn assessed the ability of the Marine Spatial Planning Challenge to communicate the dynamic and complex interactions between shipping and spatial planning in the maritime environment. Questionnaires after game sessions were applied to assess self-reported learning through closed questions and a five-point Likert scale. Their results show that the game worked well as an introduction to marine spatial planning where participants gain a grasp of its complexities, particularly for those with limited knowledge on the topic.
Secondly, 12 publications analysed self-reported learning through pre-post measurements of participants’ knowledge on the issues addressed in the game [58,59,60,65,70,76,78,80,81,82,87,88]. Questionnaires were the common method in this approach, complemented with other methods, although Salvini et al. [81] and Haug et al. [82] instead used interviews and concept maps respectively. Salvini et al. [81] assessed the effects on social learning of their game, that aims to induce Brazilian farmers to explore agroforestry practices, by conducting interviews with participants. The interviews covered the same questions before and after game sessions and included questions on the farmers’ practices, knowledge of different systems, and opinions on forming cooperatives. The farmers learned about the technical aspects of agroforestry, such as the amount of investments needed for new infrastructure, and its benefits, such as increased productivity, product quality, and profitability. Haug et al. [82] assessed social learning outcomes through a policy game on EU climate change policy and the member states’ sharing arrangements. Participants were asked to draw concept maps [92] before and after game sessions. The analysis of the concept maps was complemented with self-reflective questions asked in questionnaires and interviews after game sessions. Their results show there were some cognitive learning outcomes as certain issues became more central and specific in the post session concept maps—an indication of restructuring of knowledge.
Pre-post measurements of cognitive learning outcomes through questionnaires were based on qualitative assessment of open questions or quantitative assessment of closed questions using Likert scales or a combination of the two. For example, Douven et al. [70] assessed the impact on both raising awareness and upgrading knowledge of playing the Shariva game, which aims to stimulate collaboration among water and related professionals and to resolve transboundary river basin issues. They used questionnaires with both open and closed questions before and after game sessions in this assessment. Their results show that participants acquired knowledge on addressing and resolving transboundary river basin issues. Onencan and Van de Walle [87] assessed whether or not participants gained increased situational awareness—the perception of system elements, comprehension of their meaning and projection of their near future status—from playing WeShareIt Nzoia. To this end, they applied questionnaires based on SART [95], using 10-dimensional subjective ratings, before and after game sessions in combination with in-game performance measures to measure the game’s effects. Their results show an increase in participants’ situational awareness, on all dependent variables, between the pre-test and post-test of local policymakers.
Thirdly, 10 publications assessed cognitive learning outcomes without asking participants, but rather assessed observed acquisition or restructuring of knowledge by analysing observations and data resulting from game sessions [51,53,54,56,61,71,79,83,84,85]. To assess whether or not participants improved their understanding of the complexity of planning for a sustainable future through their game Futura, Antle et al. [54] analysed field notes on participant behaviour and quotes by identifying themes through coding. Their results show that participants did not necessarily acquire new knowledge from playing the game, but did obtain a better sense of the sheer complexity of sustainable development. Sausse et al. [84] developed a game for local stakeholders to explore management of both genetically modified and conventional crops in an area. To analyse how farmers manage the coexistence of both crop types, they analysed data collected from gameplay, such as the simulation results, player actions and maps drawn by the participants, complemented with observations from notes and audio recordings, to understand why players made certain decisions and why certain events occurred. They show that farmers learned about how to technically implement management measures and to better assess the risk of coexistence of both genetically modified and conventional crops in fields next to each other.

5.2. Normative Learning

Normative learning relates to a shift in viewpoints, values or paradigms. Assessing changes in viewpoints and values is particularly challenging as these are difficult to measure and require reflection—and thus time—to occur. How normative learning has been assessed in collaborative serious games is therefore of particular interest. Five publications report on normative learning outcomes [60,74,75,78,88]. In addition, Haug et al. [82] did assess normative learning, but did not find evidence of it taking place. Their assessment was based on pre-post measurements, participants rating the extent they agreed or disagreed on propositions before and after game sessions, in combination with participant interviews after game sessions.
Three of the five publications that report on normative learning outcomes used either pre-post measurements or self-reflective questions after gameplay to assess individual changes in views and values. In the evaluation of Ter’Aguas, a game used to simulate negotiations related to land-use planning in a Brazilian municipality, Ducrot, et al. [78] applied a short-term and long-term learning assessment. In the long-term assessment, through interviews conducted eight months after the game sessions, they found that participants had changed their opinion on their role and position in relation to water management issues. For example, a participant noted that (s)he no longer threw away oil after becoming aware of a water quality issue through playing the game.
Two publications applied pre-post measurements that asked participants to indicate to what extent they agreed with propositions, an approach similar to Haug, et al. [82]. Meya and Eisenack [60] assessed how playing the board game KEEP COOL, which aims to enhance public understanding of climate change science and to raise awareness among public, scientific and environmental organizations, changes participants’ beliefs on international climate politics. To this end, they asked participants to rate their opinions on for example personal responsibility towards climate change mitigation and the confidence in politics to act against climate change on a five-point Likert scale in questionnaires before and after game sessions. Their results show that participants perceived increased responsibility and became more confident in the potential of politics to act. Similarly, Sterman et al. [88] used pre-post measurements of closed question questionnaires to assess changes in attitudes towards climate change from playing the game WORLD CLIMATE, which aims to help participants understand the dynamics and geopolitical implications of climate change. Their results show that participants became more worried about climate change, believed it to be more personally important, and were more likely to urge for immediate action.
The other two publications that report normative learning outcomes cover different versions of the Sustainable Delta Game, which aims to help participants learn about preparing water management strategies for an uncertain future. Lawrence and Haasnoot [74] applied an adapted version of the game in a regional water management project in New Zealand to test how dynamic pathways, pro-active planning to adapt to uncertain future developments, can be adopted in decision-making. Based on observations during game sessions complemented with analyses of the debriefings and interviews with participants after game sessions, they conclude that the game led to group convergence on the necessities to make decisions in uncertain conditions. Van der Wal et al. [75] in turn explicitly assessed such group convergence by measuring changes in group perspectives. They applied perspective mapping, a method rooted in Cultural Theory to classify, interpret, and analyse different individual perspectives [91]. Specifically, participants were asked to select statements they agreed with in relation to a topic during each phase of the game. The method is therefore quite similar to the agreement or disagreement ratings applied by Haug, et al. [82], Meya and Eisenack [60] and Sterman et al. [88]. Group convergence was subsequently assessed by comparing the agreement on statements between participants. The analysis was complemented with analysing recordings of game sessions to understand why perspectives changed and participant interviews after the game to validate that participants found the game and its underlying model credible. Their results show that individual perspectives changed during gameplay, leading to group convergence in most game sessions.

5.3. Relational Learning

Relational learning relates to obtaining an improved understanding of others’ mind-sets, enhanced trust and ability to cooperate. Around half of the publications—17 out of 42—report on relational learning outcomes. Two categories were identified in regard to relational learning outcomes: Understanding other roles and perspectives and building relationships and trust. The first category, reported by 13 publications, relates to participants acquiring an increased understanding on other participants’ roles, mind-sets, and points of view [48,49,56,58,63,64,66,70,74,78,80,82,89]. The second category, reported by seven publications, instead relates to enhancing participants’ ability to collaborate by building relationships and enhancing trust [48,49,52,55,69,80,81]. Both learning outcomes were assessed mostly qualitatively, based on self-reflective questions in questionnaires, interviews and debriefings conducted after game sessions or well after game sessions (post-post). However, building relationships and trust was reported more often when collaborative serious games were used in a community or an existing project where participants addressed a shared problem that directly impacts them.
For example, in the same long-term assessment as described in the previous section by Ducrot et al. [78], participants indicated in the interviews that they learned about the relationships between issues and stakeholders, and obtained a better understanding of other stakeholders’ positions. Similarly, Rumore et al. [58] interviewed participants four to six weeks after game sessions in the assessment of a tailored role-playing game to enhance engagement in the NECAP project, a project that aims to increase the climate change adaptation readiness of coastal communities in New England. In the interviews, participants showed increased empathy and appreciation for other perspectives as a result of playing the game. In particular, participants noted these learning outcomes as a result of taking on another role, looking at the issue from another perspective, and engaging openly with other stakeholders’ point of views. Different from these examples, Carson et al. [80] used questionnaires both directly after game sessions and three months later to assess the effect of the Multi Hazard Tournament on decision-making, social learning, and relationship building in watershed management. The post-game questionnaire showed that learning about other stakeholders’ perspectives was the main learning outcome. Moreover, participants indicated in the follow-up questionnaire that game sessions had led to them pursuing potential projects with other participants.
Exception to this approach is Jean et al. [48], who combined self-reflective questions with interaction and social network analysis to assess how playing Aqua Republica—a game focused on sustainable watershed management—enhanced collaboration and knowledge co-creation. In particular, participants were asked to rate the collaboration in their game session both during—after completed phases—and after the session. Interaction analysis and social network analysis were applied on session recordings of the game phases to analyse the amount of interactions between participants, the quality of interactions and the effect pre-existing relationships had on group dynamics. Their analysis shows that participants shared ideas and forged relationships during game sessions, providing evidence that collaborative serious games can help to enhance connections and develop a mutual understanding between stakeholders.

5.4. Collective Action

Apart from cognitive, relational, and normative learning outcomes, social learning should ultimately lead to collective action [14,15,18]. Arguably, the best indicator of collective action is showing that using a collaborative serious game led to real-world impact. Indeed, a few publications describe how game sessions within a project or community led to real-world initiatives and decisions. For example, the application of the Sustainable Delta Game in a water management project in New Zealand by Lawrence and Haasnoot [74] led to the adoption of adaptive pathways strategy in the project’s climate adaptation strategy. In the interviews conducted before game sessions by Salvini et al. [81], farmers indicated that they had had negative experiences with farmer cooperatives in the past. In interviews after game sessions, farmers individually indicated that they were considering to create a new cooperative together with other participants.
One publication however set out to measure the real-world impact of using a collaborative serious game. Meinzen-Dick et al. [83] used two different follow-ups after game sessions to improve local community understanding of groundwater interrelationships and to stimulate collective governance of groundwater in India: (1) interviewing locals who did not participate in game sessions to measure whether or not there were any spill-over learning effects; and (2) gathering community-level data of different communities in the region, both communities where the collaborative serious game was used and where it was not used, to evaluate whether or not the game led to changes in community practices. In relation to the latter, more communities—statistically significant—where the game was used adopted lessons from the game, leading Meinzen-Dick et al. [83] to conclude that playing the game led to real-world impact.

5.5. Reflection and Limitations

All literature reviews aim to provide the complete overview of relevant publications and the state-of-the-art. There are however some threats and limitations to note in that regard. Firstly, a literature review is always limited in the keywords that it uses in the document search. In this case, a combination of generic and specific keywords was used in relation to sustainability and disciplines dealing with sustainability issues. The generic keywords aimed to make sure collaborative serious games on a variety of sustainability topics were found. The specific keywords were added based on relevant review papers and collaborative serious games known to the researchers. A threat here is that the prior knowledge results in a bias. the generic keywords in combination with snowballing off identified publications however limited any influence of such a bias.
Another threat is that the analysis of the identified publications, and particularly during the study selection and quality assessment steps, contains some subjectivity. After all, the majority of these analysis steps were conducting by a single researcher, the first author of this paper. To limit this threat, a systematic approach to the review was applied, following the guidelines by Kitchenham and Charters [33]. Firstly, the study selection protocol enabled an initial selection that excluded publications clearly out of scope of the review, but included publications that required more scrutiny and in-depth assessment. Secondly, the quality assessment step added a structured way of selecting the publications relevant to include in the review and while answering the quality assessment questions may still have included some subjectivity, this could only affect publications that scored just short of the threshold and were thus excluded. These publications were carefully looked at again after completing the entire quality assessment to check whether or not exclusion was a correct conclusion. None of these publications were subsequently added to the review.

6. Conclusions

Serious games are increasingly explored as a method to establish social learning on sustainable natural resources management and urban planning. The aim of this review was to answer the main research question: What is the current state of the art of the different methods and procedures used to assess social learning outcomes of collaborative serious games? A systematic literature review was conducted, which reviewed 42 publications that applied serious gaming to collaboratively explore sustainable management strategies of land and natural resources. The publications were analysed in order to determine how three types of learning outcomes—cognitive, normative, and relational learning—were assessed. To this end, data was extracted from the publications in relation to five research questions and were categorized following an inductive approach. The categorizations made it possible to identify approaches used in relation to assessing cognitive, normative, and relational learning outcomes of collaborative serious games.
Most evaluations of collaborative serious games focus on assessing cognitive learning, which relates to the acquisition of new or the restructuring of existing knowledge, for example whether or not participants obtain a better understanding of the complexities of climate change. Few evaluations focus explicitly on normative learning, which relates to a shift in viewpoints, values or paradigms, for example on an individual’s feeling of responsibility towards climate change mitigation. That few evaluations assess normative learning is not surprising as it generally requires experience, reflection, and time before a person would shift viewpoints, values or paradigms. Nevertheless, the reviewed publications do show that collaborative serious games can lead to normative learning. About half of the reviewed publications report on relational learning outcomes, which relates to obtaining an improved understanding of others’ mind-sets, enhanced trust, and ability to cooperate. That only half assess this learning type seems to relate to whether or not relational learning is seen as part of the conceptualization of learning and whether or not the collaborative serious game is used in an existing project or community. Table 9 provides an overview of the approaches used in the assessment of the three learning outcomes, including the common methods and procedures used.
The review shows that three common approaches are used to assess cognitive learning outcomes. Firstly, asking participants self-reflective questions such as what they learned after game sessions, either in a qualitative assessment through open questions or a quantitative assessment using closed questions. Secondly, applying measurements of participants’ knowledge before and after game sessions, also based on self-reporting by asking open or closed questions to participants. Thirdly, assessing observed acquisition or restructuring of knowledge by analysing observations, either qualitatively or quantitatively, and in-game collected data.
Two approaches are used to assess normative learning outcomes. Firstly, asking participants to rate their views on propositions in pre–post measurements, optionally complemented with such ratings during game sessions, through closed questions questionnaires. Next to assessing individual shifts in viewpoints, pre–post measured individual shifts can be compared within the group of participants to determine whether or not convergence of group opinion occurs. Secondly, interviews with participants well after game sessions—weeks to months—can be used to reflect on the impact of collaborative serious games on normative learning.
Two approaches are also used to assess relational learning outcomes. Firstly, asking participants self-reflective questions following game sessions—the approach used in all but one publication reviewed—in order to determine whether or not participants gain a better understanding of other stakeholders. Secondly, asking participants to self-report the level of collaboration in combination with analysing participants’ interactions during game sessions. This approach quantitatively analyses the group dynamics during game sessions in order to determine whether or not relationships are enhanced and a mutual understanding is developed.
Of course, evaluation approaches are not exclusive from one another. Many of the reviewed publications in fact combine approaches or use multiple methods to assess learning outcomes. A multi-method evaluation approach is also recognized to fit well with serious gaming in general. However, explicitly assessing cognitive, normative, and relational learning may help to separate the nature of learning through collaborative serious games. The overview provided in this paper may therefore serve as a reference for scholars designing and evaluating collaborative serious games, particularly those that address sustainability problems, but may also be useful in relation to serious games that aim to collaboratively address problems in complex systems in general.

Supplementary Materials

Table A1: Overview of all publications included in the review, their quality assessments and categorization of the data in relation to the research questions. A spreadsheet with the underlying data of Table A1 can be found at https://www.mdpi.com/2071-1050/10/12/4529/s1.

Funding

This research is part of the research programme RiverCare: towards self-sustaining multifunctional rivers, supported by the Netherlands Organization for Scientific Research (NWO) and partly funded by the Ministry of Economic Affairs under grant number P12-14 (Perspective Programme).

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Table A1. Overview of all publications included in the review, the quality assessment and the categorization of the data in relation to the research questions.
Table A1. Overview of all publications included in the review, the quality assessment and the categorization of the data in relation to the research questions.
Game NameReferenceQuality AssessmentLearning Conceptualization (RQ1)Data Collection (RQ2)Assessment Methods (RQ3)Quantitative or Qualitative Data (RQ4)Social Learning Effects (RQ5)
Aqua Republica[48]5Boundary crossingPre & During & PostQuestionnaires & interaction analysis & social network analysisQuantitative & qualitativeUnderstanding other perspectives
Building relationships and trust
CauxOperation[49]5Collective learningPostDebriefingQualitativeRaised awareness
Increased system understanding
Understanding other perspectives
Building relationships and trust
Climate Game[50]5Links to experiential learningPre & During & PostQuestionnaires & observations & debriefsQuantitative & qualitativeIncreased system understanding
Community Cooperation Game[51]5Links to experiential learningDuringObservationsQualitativeIncreased system understanding
Raised awareness
Forage Rummy[52]5Knowledge co-productionPostDirect feedbackQualitativeIncreased system understanding
Raised awareness
Building relationships and trust
FOWIS[53]4Not conceptualizedDuring & PostObservations & debriefs & InterviewsQualitativeIncreased system understanding
Futura[54]5Experiential learningDuringObservationsQualitativeRaised awareness
GO2Zero[55]4Not conceptualizedPre & During & PostQuestionnaires & ObservationsQuantitative & qualitativeIncreased system understanding
Building relationships and trust
Grazing game[56]5Social learning & Anticipatory learning During & PostObservations & debriefQualitativeIncreased system understanding
Understanding other perspectives
Invitational Drought Tournament[57]5Social learningPostQuestionnaires & debriefsQualitativeIncreased system understanding
IUP & NECAP[58]5Social learningPre & Post & Post-postQuestionnaires & debriefs & interviewsQuantitative & qualitativeIncreased system understanding
Raised awareness
Understanding other perspectives
KEEP COOL[59]4Not conceptualizedPre & During & PostQuestionnaires & observationsQualitativeIncreased system understanding
[60]4Not conceptualizedPre & During & PostQuestionnaire & ObservationsQuantitativeIncreased system understanding
Raised awareness
Change in views
Lords of the Valley[61]4Social learningPre & During & PostQuestionnaires & observationsQuantitative & qualitativeNo learning effects reported
LottoSIM[62]5Links to social learning & experiential learningPostDebriefsQualitativeIncreased system understanding
MAE SALAE RPG 1 & 2[63]5Collective learningDuring & Post & Post-postObservations & InterviewsQualitativeIncreased system understanding
Understanding other perspectives
Marine Spatial Planning Challenge[64]5Constructivist learning philosophyDuring & PostQuestionnaires & observationsQuantitative & qualitativeIncreased system understanding
[65]5Policy-oriented learningPre & During & PostQuestionnaires & observations & data logging & interviewsQuantitative & qualitativeIncreased system understanding
Building relationships and trust
New-District[66]5Experiential learningPre & PostQuestionnairesQuantitative & qualitativeUnderstanding other perspectives
REEFGAME[67]5Links to social learningDuring & PostDebriefsQualitativeIncreased system understanding
ReHab[68]5Social learning & experiential learningDuring & PostObservations & debriefingsQuantitative & qualitativeIncreased system understanding
RESORTES[69]4Not conceptualizedPre & During & Post & Post-postQuestionnaires & observations & debrief & interviewQualitativeBuilding relationships and trust
Shariva[70]5Meaningful playPre & PostQuestionnairesQuantitative & qualitativeRaised awareness
Increased system understanding
Understanding other perspectives
SimPhy[71]4Not conceptualizedDuringData loggingQuantitativeIncreased system understanding
SPRINTCITY[72]5Policy oriented learningDuring & PostQuestionnaires & observations & data logging & interviewsQualitativeRaised awareness
Increased system understanding
Sustainable Delta Game (and adaptations)[73]4Not conceptualizedPre & During & PostObservations & debriefs & perspective mappingQualitativeIncreased system understanding
Understanding other perspectives
[74]5Social learning During & PostObservations & interviews & debriefsQualitativeIncreased system understanding
Change in views
[75]5Social learning & experiential learningPre & During & PostObservations & data logging & interviews & perspective mappingQuantitative & qualitativeChange in views
[76]4Not conceptualizedPre & During & Post & Post-post & Control GroupQuestionnaires & observations & debriefs & interviews & control groupQuantitative & qualitativeIncreased system understanding
Raised awareness
Establish experiential environment
TADLA[77]5Individual learning & Social learningPost & Post-postQuestionnaires & InterviewsQualitativeIncreased system understanding
Ter’Aguas[78]5Social learningPre & During & Post & Post-postQuestionnaires & observations & debriefs & interviewsQualitativeIncreased system understanding
Understanding other perspectives
Change in views
The Floodplain Management Game[79]5Social learningDuring & PostObservations & debriefsQualitativeRaised awareness
The Multi-Hazard Tournament (MHT)[80]5Social learningPre & Post & Post-postQuestionnairesQuantitative & qualitativeIncreased system understanding
Raised awareness
Understanding other perspectives
Building relationships and trust
Unnamed game[81]5Social learningPre & PostInterviewsQualitativeIncreased system understanding
Building relationships and trust
Unnamed game[82]5Social learningPre & During & Post & Post-postQuestionnaires & Observations & Debriefs & Interviews & Concept mapsQualitativeIncreased system understanding
Understanding other perspectives
Unnamed game[83]5Social learningDuring & Post & Post-post & Control groupQuestionnaires & data logging & interviews & real-world dataQuantitative & qualitativeIncreased system understanding
Real-world adoption
Unnamed game[84]4Not conceptualizedDuringObservations & data loggingQuantitative & QualitativeIncreased system understanding
Unnamed game[85]4Not conceptualizedPre & DuringQuestionnaires & data logging & control groupQuantitativeIncreased system understanding
WeShareIt[86]5Effective learningPre & During & PostQuestionnaires & observations & debriefQuantitative & qualitativeIncreased system understanding
WeShareIt Nzoia[87]5Situational awarenessPre & During & PostQuestionnaires & data loggingQuantitativeRaised awareness
Increased system understanding
WORLD CLIMATE[88]5Links to experiential learningPre & Post-postQuestionnairesQuantitativeIncreased system understanding
Raised awareness
Change in views
WSP game[89]5Experiential learningPostQuestionnaires & debriefQualitativeRaised awareness
Building relationships and trust

References

  1. Folke, C.; Carpenter, S.; Elmqvist, T.; Gunderson, L.; Holling, C.S.; Walker, B.; Bengtsson, J.; Berkes, F.; Colding, J.; Danell, K.; et al. Resilience and Sustainable Development: Building Adaptive Capacity in a World of Transformations. Ambio 2002, 31, 437–440. [Google Scholar] [CrossRef][Green Version]
  2. Berkes, F. Evolution of co-management: Role of knowledge generation, bridging organizations and social learning. J. Environ. Manag. 2009, 90, 1692–1702. [Google Scholar] [CrossRef] [PubMed]
  3. Rodela, R. Social learning and natural resource management: The emergence of three research perspectives. Ecol. Soc. 2011, 16, 30. [Google Scholar] [CrossRef]
  4. Pahl-Wostl, C. The implications of complexity for integrated resources management. Environ. Model. Softw. 2007, 22, 561–569. [Google Scholar] [CrossRef][Green Version]
  5. Armitage, D.; Berkes, F.; Doubleday, N. Adaptive Co-Management: Collaboration, Learning, and Multi-Level Governance; UBC Press: Vancouver, BC, Canada, 2007. [Google Scholar]
  6. Folke, C.; Hahn, T.; Olsson, P.; Norberg, J. Adaptive governance of social-ecological systems. Annu. Rev. Environ. Resour. 2005, 30, 441–473. [Google Scholar] [CrossRef]
  7. Olsson, P.; Folke, C.; Berkes, F. Adaptive comanagement for building resilience in social–ecological systems. Environ. Manag. 2004, 34, 75–90. [Google Scholar] [CrossRef] [PubMed]
  8. Crona, B.I.; Parker, J.N. Learning in support of governance: Theories, methods, and a framework to assess how bridging organizations contribute to adaptive resource governance. Ecol. Soc. 2012, 17, 32. [Google Scholar] [CrossRef]
  9. Pahl-Wostl, C. The importance of social learning in restoring the multifunctionality of rivers and floodplains. Ecol. Soc. 2006, 11, 10. [Google Scholar] [CrossRef]
  10. Armitage, D. Adaptive capacity and community-based natural resource management. Environ. Manag. 2005, 35, 703–715. [Google Scholar] [CrossRef] [PubMed]
  11. Reed, M.S.; Evely, A.C.; Cundill, G.; Fazey, I.; Glass, J.; Laing, A.; Newig, J.; Parrish, B.; Prell, C.; Raymond, C. What is social learning? Ecol. Soc. 2010, 15. Available online: https://www.ecologyandsociety.org/vol15/iss4/resp1/ (accessed on 1 October 2018). [CrossRef]
  12. Armitage, D.; Marschke, M.; Plummer, R. Adaptive co-management and the paradox of learning. Glob. Environ. Chang. 2008, 18, 86–98. [Google Scholar] [CrossRef]
  13. Muro, M.; Jeffrey, P. A critical review of the theory and application of social learning in participatory natural resource management processes. J. Environ. Plan. Manag. 2008, 51, 325–344. [Google Scholar] [CrossRef]
  14. Cundill, G.; Rodela, R. A review of assertions about the processes and outcomes of social learning in natural resource management. J. Environ. Manag. 2012, 113, 7–14. [Google Scholar] [CrossRef] [PubMed]
  15. Keen, M.; Brown, V.A.; Dyball, R. Social Learning in Environmental Management: Towards a Sustainable Future; Routledge: Abingdon, UK, 2005. [Google Scholar]
  16. Pahl-Wostl, C.; Craps, M.; Dewulf, A.; Mostert, E.; Tabara, D.; Taillieu, T. Social learning and water resources management. Ecol. Soc. 2007, 12, 5. [Google Scholar] [CrossRef]
  17. Baird, J.; Plummer, R.; Haug, C.; Huitema, D. Learning effects of interactive decision-making processes for climate change adaptation. Glob. Environ. Chang. 2014, 27, 51–63. [Google Scholar] [CrossRef]
  18. Pahl-Wostl, C.; Tàbara, D.; Bouwen, R.; Craps, M.; Dewulf, A.; Mostert, E.; Ridder, D.; Taillieu, T. The importance of social learning and culture for sustainable water management. Ecol. Econ. 2008, 64, 484–495. [Google Scholar] [CrossRef]
  19. Furber, A.; Medema, W.; Adamowski, J. Assessing the benefits of serious games to support sustainable decision-making for transboundary watershed governance. Can. Water Resour. J./Rev. Can. Ressour. Hydr. 2018, 1–15. [Google Scholar] [CrossRef]
  20. Medema, W.; Furber, A.; Adamowski, J.; Zhou, Q.; Mayer, I. Exploring the potential impact of serious games on social learning and stakeholder collaborations for transboundary watershed management of the St. Lawrence River Basin. Water 2016, 8, 175. [Google Scholar] [CrossRef]
  21. Marini, D.; Medema, W.; Adamowski, J.; Veissière, S.; Mayer, I.; Wals, A. Socio-Psychological Perspectives on the Potential for Serious Games to Promote Transcendental Values in IWRM Decision-Making. Water 2018, 10, 1097. [Google Scholar] [CrossRef]
  22. Ampatzidou, C.; Gugerell, K.; Constantinescu, T.; Devisch, O.; Jauschneg, M.; Berger, M. All Work and No Play? Facilitating Serious Games and Gamified Applications in Participatory Urban Planning and Governance. Urban Plan. 2018, 3, 34–46. [Google Scholar] [CrossRef]
  23. Susi, T.; Johannesson, M.; Backlund, P. Serious Games: An Overview; Institutionen för Kommunikation och Information: Lund, Sweden, 2007. [Google Scholar]
  24. Michael, D.R.; Chen, S.L. Serious Games: Games That Educate, Train, and Inform; Thomson Course Technology: Boston, MA, USA, 2006. [Google Scholar]
  25. Abt, C.C. Serious Games; University Press of America: Lanham, MA, USA, 1987. [Google Scholar]
  26. Mayer, I. The gaming of policy and the politics of gaming: A review. Simul. Gaming 2009, 40, 825–862. [Google Scholar] [CrossRef]
  27. Bekebrede, G. Experiencing Complexity: A Gaming Approach for Understanding Infrastructure System. Ph.D. Thesis, TU Delft, Delft University of Technology, Delft, The Netherlands, 2010. [Google Scholar]
  28. Roschelle, J.; Teasley, S.D. The construction of shared knowledge in collaborative problem solving. In Proceedings of the Computer Supported Collaborative Learning; Springer: Berlin/Heidelberg, Germany, 1995; pp. 69–97. [Google Scholar]
  29. Geurts, J.L.; Duke, R.D.; Vermeulen, P.A. Policy gaming for strategy and change. Long Range Plan. 2007, 40, 535–558. [Google Scholar] [CrossRef]
  30. Mayer, I.; Bekebrede, G.; Harteveld, C.; Warmelink, H.; Zhou, Q.; van Ruijven, T.; Lo, J.; Kortmann, R.; Wenzler, I. The research and evaluation of serious games: Toward a comprehensive methodology. Br. J. Educ. Technol. 2014, 45, 502–527. [Google Scholar] [CrossRef]
  31. De Caluwé, L.; Geurts, J.; Kleinlugtenbelt, W.J. Gaming research in policy and organization: An assessment from the Netherlands. Simul. Gaming 2012, 43, 600–626. [Google Scholar] [CrossRef]
  32. Hofstede, G.J.; De Caluwé, L.; Peters, V. Why simulation games work-in search of the active substance: A synthesis. Simul. Gaming 2010, 41, 824–843. [Google Scholar] [CrossRef]
  33. Kitchenham, B.; Charters, S. Guidelines for Performing Systematic Literature Reviews in Software Engineering; Version 2.3; University of Keele (Software Engineering Group, School of Computer Science and Mathematics) and University of Durham (Department of Computer Science): Keele and Durham, UK, 2007. [Google Scholar]
  34. Kitchenham, B. Procedures for Performing Systematic Reviews; Keele University: Keele, UK, 2004; Volume 33, pp. 1–26. [Google Scholar]
  35. Calderón, A.; Ruiz, M. A systematic literature review on serious games evaluation: An application to software project management. Comput. Educ. 2015, 87, 396–422. [Google Scholar] [CrossRef]
  36. Connolly, T.M.; Boyle, E.A.; MacArthur, E.; Hainey, T.; Boyle, J.M. A systematic literature review of empirical evidence on computer games and serious games. Comput. Educ. 2012, 59, 661–686. [Google Scholar] [CrossRef]
  37. Backlund, P.; Hendrix, M. Educational games-are they worth the effort? A literature survey of the effectiveness of serious games. In Proceedings of the 2013 5th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES), Poole, UK, 11–13 September 2013; pp. 1–8. [Google Scholar]
  38. Boyle, E.A.; Hainey, T.; Connolly, T.M.; Gray, G.; Earp, J.; Ott, M.; Lim, T.; Ninaus, M.; Ribeiro, C.; Pereira, J. An update to the systematic literature review of empirical evidence of the impacts and outcomes of computer games and serious games. Comput. Educ. 2016, 94, 178–192. [Google Scholar] [CrossRef]
  39. Wouters, P.; Van der Spek, E.D.; Van Oostendorp, H. Current practices in serious game research: A review from a learning outcomes perspective. In Games-Based Learning Advancements for multi-Sensory Human Computer Interfaces: Techniques and Effective Practices; IGI Global: Hershey, PA, USA, 2009; pp. 232–250. [Google Scholar]
  40. Girard, C.; Ecalle, J.; Magnan, A. Serious games as new educational tools: How effective are they? A meta-analysis of recent studies. J. Comput. Assist. Learn. 2013, 29, 207–219. [Google Scholar] [CrossRef]
  41. Katsaliaki, K.; Mustafee, N. A survey of serious games on sustainable development. In Proceedings of the Winter Simulation Conference, Berlin, Germany, 9–12 December 2012; p. 136. [Google Scholar]
  42. Reckien, D.; Eisenack, K. Climate change gaming on board and screen: A review. Simul. Gaming 2013, 44, 253–271. [Google Scholar] [CrossRef]
  43. Flood, S.; Cradock-Henry, N.A.; Blackett, P.; Edwards, P. Adaptive and interactive climate futures: Systematic review of ‘serious games’ for engagement and decision-making. Environ. Res. Lett. 2018, 13, 063005. [Google Scholar] [CrossRef]
  44. Wu, J.S.; Lee, J.J. Climate change games as tools for education and engagement. Nat. Clim. Chang. 2015, 5, 413. [Google Scholar] [CrossRef]
  45. Aubert, A.H.; Bauer, R.; Lienert, J. A review of water-related serious games to specify use in environmental Multi-Criteria Decision Analysis. Environ. Model. Softw. 2018, 105, 64–78. [Google Scholar] [CrossRef]
  46. Wohlin, C. Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, London, UK, 13–14 May 2014; p. 38. [Google Scholar]
  47. Moher, D.; Liberati, A.; Tetzlaff, J.; Altman, D.G. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Ann. Intern. Med. 2009, 151, 264–269. [Google Scholar] [CrossRef] [PubMed]
  48. Jean, S.; Medema, W.; Adamowski, J.; Chew, C.; Delaney, P.; Wals, A. Serious games as a catalyst for boundary crossing, collaboration and knowledge co-creation in a watershed governance context. J. Environ. Manag. 2018, 223, 1010–1022. [Google Scholar] [CrossRef] [PubMed]
  49. Souchère, V.; Millair, L.; Echeverria, J.; Bousquet, F.; Le Page, C.; Etienne, M. Co-constructing with stakeholders a role-playing game to initiate collective management of erosive runoff risks at the watershed scale. Environ. Model. Softw. 2010, 25, 1359–1370. [Google Scholar] [CrossRef]
  50. Zhou, Q.; Bekebrede, G.; Mayer, I.; Warmerdam, J.; Knepflé, M. The Climate Game: Connecting Water Management and Spatial Planning through Simulation Gaming? In Water Governance as Connective Capacity; Edelenbos, J., Bressers, N., Scholten, P., Eds.; Routledge: Abingdon, UK, 2013; pp. 109–127. [Google Scholar]
  51. Tanwattana, P.; Toyoda, Y. Contributions of gaming simulation in building community-based disaster risk management applying Japanese case to flood prone communities in Thailand upstream area. Int. J. Disaster Risk Reduct. 2018, 27, 199–213. [Google Scholar] [CrossRef]
  52. Martin, G. A conceptual framework to support adaptation of farming systems–Development and application with Forage Rummy. Agric. Syst. 2015, 132, 52–61. [Google Scholar] [CrossRef]
  53. Hertzog, T.; Poussin, J.-C.; Tangara, B.; Kouriba, I.; Jamin, J.-Y. A role playing game to address future water management issues in a large irrigated system: Experience from Mali. Agric. Water Manag. 2014, 137, 1–14. [Google Scholar] [CrossRef]
  54. Antle, A.N.; Bevans, A.; Tanenbaum, J.; Seaborn, K.; Wang, S. Futura: Design for collaborative learning and game play on a multi-touch digital tabletop. In Proceedings of the Fifth International Conference on Tangible, Embedded, and Embodied Interaction, Funchal, Portugal, 22–26 January 2011; pp. 93–100. [Google Scholar]
  55. Bekebrede, G.; van Bueren, E.; Wenzler, I. Towards a Joint Local Energy Transition Process in Urban Districts: The GO2Zero Simulation Game. Sustainability 2018, 10, 2602. [Google Scholar] [CrossRef]
  56. Villamor, G.B.; Badmos, B.K. Grazing game: A learning tool for adaptive management in response to climate variability in semiarid areas of Ghana. Ecol. Soc. 2016, 21, 39. [Google Scholar] [CrossRef]
  57. Hill, H.; Hadarits, M.; Rieger, R.; Strickert, G.; Davies, E.G.; Strobbe, K.M. The Invitational Drought Tournament: What is it and why is it a useful tool for drought preparedness and adaptation? Weather Clim. Extremes 2014, 3, 107–116. [Google Scholar] [CrossRef]
  58. Rumore, D.; Schenk, T.; Susskind, L. Role-play simulations for climate change adaptation education and engagement. Nat. Clim. Chang. 2016, 6, 745. [Google Scholar] [CrossRef]
  59. Eisenack, K. A climate change board game for interdisciplinary communication and education. Simul. Gaming 2013, 44, 328–348. [Google Scholar] [CrossRef]
  60. Meya, J.N.; Eisenack, K. Effectiveness of gaming for communicating and teaching climate change. Clim. Chang. 2018, 149, 319–333. [Google Scholar] [CrossRef]
  61. Magnuszewski, P.; Królikowska, K.; Koch, A.; Pająk, M.; Allen, C.; Chraibi, V.; Giri, A.; Haak, D.; Hart, N.; Hellman, M. Exploring the Role of Relational Practices in Water Governance Using a Game-Based Approach. Water 2018, 10, 346. [Google Scholar] [CrossRef]
  62. Becu, N.; Amalric, M.; Anselme, B.; Beck, E.; Bertin, X.; Delay, E.; Long, N.; Marilleau, N.; Pignon-Mussaud, C.; Rousseaux, F. Participatory simulation to foster social learning on coastal flooding prevention. Environ. Model. Softw. 2017, 98, 1–11. [Google Scholar] [CrossRef]
  63. Barnaud, C.; Promburom, T.; Trébuil, G.; Bousquet, F. An evolving simulation/gaming process to facilitate adaptive watershed management in northern mountainous Thailand. Simul. Gaming 2007, 38, 398–420. [Google Scholar] [CrossRef][Green Version]
  64. Mayer, I.; Zhou, Q.; Lo, J.; Abspoel, L.; Keijser, X.; Olsen, E.; Nixon, E.; Kannen, A. Integrated, ecosystem-based Marine Spatial Planning: Design and results of a game-based, quasi-experiment. Ocean Coast. Manag. 2013, 82, 7–26. [Google Scholar] [CrossRef]
  65. Keijser, X.; Ripken, M.; Mayer, I.; Warmelink, H.; Abspoel, L.; Fairgrieve, R.; Paris, C. Stakeholder Engagement in Maritime Spatial Planning: The Efficacy of a Serious Game Approach. Water 2018, 10, 724. [Google Scholar] [CrossRef]
  66. Becu, N.; Frascaria-Lacoste, N.; Latune, J. Experiential Learning Based on the Newdistrict Asymmetric Simulation Game: Results of a Dozen Gameplay Sessions. 2015. Available online: https://hal.archives-ouvertes.fr/hal-01253024/document (accessed on 7 January 2018).
  67. Cleland, D.; Dray, A.; Perez, P.; Cruz-Trinidad, A.; Geronimo, R. Simulating the dynamics of subsistence fishing communities: REEFGAME as a learning and data-gathering computer-assisted role-play game. Simul. Gaming 2012, 43, 102–117. [Google Scholar] [CrossRef]
  68. Page, C.L.; Dray, A.; Perez, P.; Garcia, C. Exploring how knowledge and communication influence natural resources management with REHAB. Simul. Gaming 2016, 47, 257–284. [Google Scholar] [CrossRef]
  69. Speelman, E.; García-Barrios, L.; Groot, J.; Tittonell, P. Gaming for smallholder participation in the design of more sustainable agricultural landscapes. Agric. Syst. 2014, 126, 62–75. [Google Scholar] [CrossRef]
  70. Douven, W.; Mul, M.L.; Son, L.; Bakker, N.; Radosevich, G.; Hendriks, A. Games to create awareness and design policies for transboundary cooperation in river basins: Lessons from the shariva game of the Mekong river commission. Water Resour. Manag. 2014, 28, 1431–1447. [Google Scholar] [CrossRef]
  71. Ayadi, H.; Le Bars, M.; Le Grusse, P.; Mandart, E.; Fabre, J.; Bouaziz, A.; Bord, J. SimPhy: A simulation game to lessen the impact of phytosanitaries on health and the environment—The case of Merja Zerga in Morocco. Environ. Sci. Pollut. Res. 2014, 21, 4950–4963. [Google Scholar] [CrossRef] [PubMed]
  72. Duffhues, J.; Mayer, I.S.; Nefs, M.; van der Vliet, M. Breaking barriers to transit-oriented development: Insights from the serious game SPRINTCITY. Environ. Plan. B Plan. Des. 2014, 41, 770–791. [Google Scholar] [CrossRef]
  73. Valkering, P.; van der Brugge, R.; Offermans, A.; Haasnoot, M.; Vreugdenhil, H. A perspective-based simulation game to explore future pathways of a water-society system under climate change. Simul. Gaming 2013, 44, 366–390. [Google Scholar] [CrossRef]
  74. Lawrence, J.; Haasnoot, M. What it took to catalyse uptake of dynamic adaptive pathways planning to address climate change uncertainty. Environ. Sci. Policy 2017, 68, 47–57. [Google Scholar] [CrossRef]
  75. Van der Wal, M.M.; De Kraker, J.; Kroeze, C.; Kirschner, P.A.; Valkering, P. Can computer models be used for social learning? A serious game in water management. Environ. Model. Softw. 2016, 75, 119–132. [Google Scholar] [CrossRef][Green Version]
  76. Van Pelt, S.; Haasnoot, M.; Arts, B.; Ludwig, F.; Swart, R.; Biesbroek, R. Communicating climate (change) uncertainties: Simulation games as boundary objects. Environ. Sci. Policy 2015, 45, 41–52. [Google Scholar] [CrossRef]
  77. Dionnet, M.; Kuper, M.; Hammani, A.; Garin, P. Combining role-playing games and policy simulation exercises: An experience with Moroccan smallholder farmers. Simul. Gaming 2008, 39, 498–514. [Google Scholar] [CrossRef]
  78. Ducrot, R.; Van Paassen, A.; Barban, V.; Daré, W.s.; Gramaglia, C. Learning integrative negotiation to manage complex environmental issues: Example of a gaming approach in the peri-urban catchment of Sao Paulo, Brazil. Reg. Environ. Chang. 2015, 15, 67–78. [Google Scholar] [CrossRef]
  79. Stefanska, J.; Magnuszewski, P.; Sendzimir, J.; Romaniuk, P.; Taillieu, T.; Dubel, A.; Flachner, Z.; Balogh, P. A Gaming Exercise to Explore Problem-Solving versus Relational Activities for River Floodplain Management. Environ. Policy Gov. 2011, 21, 454–471. [Google Scholar] [CrossRef]
  80. Carson, A.; Windsor, M.; Hill, H.; Haigh, T.; Wall, N.; Smith, J.; Olsen, R.; Bathke, D.; Demir, I.; Muste, M. Serious gaming for participatory planning of multi-hazard mitigation. Int. J. River Basin Manag. 2018, 16, 379–391. [Google Scholar] [CrossRef]
  81. Salvini, G.; Van Paassen, A.; Ligtenberg, A.; Carrero, G.; Bregt, A. A role-playing game as a tool to facilitate social learning and collective action towards Climate Smart Agriculture: Lessons learned from Apuí, Brazil. Environ. Sci. Policy 2016, 63, 113–121. [Google Scholar] [CrossRef]
  82. Haug, C.; Huitema, D.; Wenzler, I. Learning through games? Evaluating the learning effect of a policy exercise on European climate policy. Technol. Forecast. Soc. Chang. 2011, 78, 968–981. [Google Scholar] [CrossRef]
  83. Meinzen-Dick, R.; Janssen, M.A.; Kandikuppa, S.; Chaturvedi, R.; Rao, K.; Theis, S. Playing games to save water: Collective action games for groundwater management in Andhra Pradesh, India. World Dev. 2018, 107, 40–53. [Google Scholar] [CrossRef]
  84. Sausse, C.; Le Bail, M.; Lecroart, B.; Remy, B.; Messéan, A. How to manage the coexistence between genetically modified and conventional crops in grain and oilseed collection areas? Elaboration of scenarios using role playing games. Land Use Policy 2013, 30, 719–729. [Google Scholar] [CrossRef]
  85. Moser, S.; Mußhoff, O. Ex-ante evaluation of policy measures: Effects of reward and punishment for fertiliser reduction in palm oil production. J. Agric. Econ. 2016, 67, 84–104. [Google Scholar] [CrossRef]
  86. Onencan, A.; Van de Walle, B.; Enserink, B.; Chelang’a, J.; Kulei, F. WeShareIt Game: Strategic foresight for climate-change induced disaster risk reduction. Procedia Eng. 2016, 159, 307–315. [Google Scholar] [CrossRef]
  87. Onencan, A.M.; Van de Walle, B. From Paris Agreement to Action: Enhancing Climate Change Familiarity and Situation Awareness. Sustainability 2018, 10, 1929. [Google Scholar] [CrossRef]
  88. Sterman, J.; Franck, T.; Fiddaman, T.; Jones, A.; McCauley, S.; Rice, P.; Sawin, E.; Siegel, L.; Rooney-Varga, J.N. World Climate: A role-play simulation of climate negotiations. Simul. Gaming 2015, 46, 348–382. [Google Scholar] [CrossRef]
  89. Ferrero, G.; Bichai, F.; Rusca, M. Experiential Learning through Role-Playing: Enhancing Stakeholder Collaboration in Water Safety Plans. Water 2018, 10, 227. [Google Scholar] [CrossRef]
  90. Kolb, D.A. Experiential Learning: Experience as the Source of Learning and Development; Prentice Hall: Englewood Cliffs, NJ, USA, 1984. [Google Scholar]
  91. Offermans, A.; Haasnoot, M.; Valkering, P. A method to explore social response for sustainable water management strategies under changing conditions. Sustain. Dev. 2011, 19, 312–324. [Google Scholar] [CrossRef]
  92. Ruiz-Primo, M.A.; Shavelson, R.J. Problems and issues in the use of concept maps in science assessment. J. Res. Sci. Teach. Off. J. Natl. Assoc. Res. Sci. Teach. 1996, 33, 569–600. [Google Scholar] [CrossRef]
  93. Jordan, B.; Henderson, A. Interaction analysis: Foundations and practice. J. Learn. Sci. 1995, 4, 39–103. [Google Scholar] [CrossRef]
  94. Serrat, O. Social network analysis. In Knowledge Solutions; Springer: Singapore, 2017; pp. 39–43. [Google Scholar]
  95. Taylor, R. Situational awareness rating technique (SART): The development of a tool for aircrew systems design. In Situation Awareness in Aerospace Operations; NATO—Advisory Group for Aerospace Research and Development: Neuilly-Sur-Seine, France, 1990; pp. 1–17. [Google Scholar]
Figure 1. Number of publications included after each step. Flow chart based on the PRISMA guidelines [47].
Figure 1. Number of publications included after each step. Flow chart based on the PRISMA guidelines [47].
Sustainability 10 04529 g001
Figure 2. Overview of the publications retrieved and included in the review: (a) overview of publications retrieved per database or through snowballing (note: some publications appeared in more than one database); (b) overview of the amount of publications included in the review per year of publication.
Figure 2. Overview of the publications retrieved and included in the review: (a) overview of publications retrieved per database or through snowballing (note: some publications appeared in more than one database); (b) overview of the amount of publications included in the review per year of publication.
Sustainability 10 04529 g002
Table 1. Inclusion and exclusion criteria.
Table 1. Inclusion and exclusion criteria.
Inclusion criteria
  • Publications that describe a collaborative serious game on sustainable management of land or natural resources, or otherwise related to a complex sustainability issues (e.g., climate change)
  • Publications that describe the assessment of learning outcomes of a collaborative serious game
  • Publications that describe the procedure and methods applied to assess learning outcomes of a collaborative serious game
Exclusion criteria
  • Publications that describe a serious game, but which do not cover a collaborative serious game (e.g., a single-player serious game)
  • Publications that describe a collaborative serious game, but the game is not in any way related to sustainability
  • Publications that describe a collaborative serious game, but which do not assess its learning outcomes
  • Publications that describe the evaluation of the learning outcomes from a collaborative serious game, but offer no information on how this assessment was performed
  • Publications that describe the assessment of a larger learning process of which a collaborative serious game is a component that cannot be seen as separated from the larger process and its evaluation
  • Publications that describe a formative evaluation that only focuses on improving the design of a collaborative serious game and do not assess learning outcomes in any way
  • Publications that describe a serious game designed only for use in an educational setting
  • Publications that are not written in English
  • Publications that are only available as abstract 1
1 Abstracts were still reviewed, if the abstract seemed applicable, an attempt was made to obtain the entire publication, exclusion occurred if this was not possible.
Table 2. Typology of learning effects (from Baird et al. [17]).
Table 2. Typology of learning effects (from Baird et al. [17]).
TypeDefinition/Indicators of Learning EffectsMeasures of Indicators
Cognitive learningAcquisition of new knowledge;
restructuring of existing knowledge
Test scores; change in centrality and specificity of knowledge
presented in concept maps
Normative learningChanges in norms;
change in values;
change in paradigms;
convergence of group opinion
Change in, and convergence around, environmental beliefs;
participant reflections;
meeting proceedings
Relational learningImproved understanding of mind sets of others;
building of relationships;
enhanced trust and cooperation
Change in social network structure;
participant reflections
Table 3. Overview of all games and publications included in the review.
Table 3. Overview of all games and publications included in the review.
Game NameReference
Aqua RepublicaJean, Medema, Adamowski, Chew, Delaney and Wals [48]
CauxOperationSouchère, Millair, Echeverria, Bousquet, Le Page and Etienne [49]
Climate GameZhou, Bekebrede, Mayer, Warmerdam and Knepflé [50]
Community Cooperation GameTanwattana and Toyoda [51]
Forage RummyMartin [52]
FOWIS Hertzog, Poussin, Tangara, Kouriba and Jamin [53]
FuturaAntle, Bevans, Tanenbaum, Seaborn and Wang [54]
GO2ZeroBekebrede, van Bueren and Wenzler [55]
Grazing gameVillamor and Badmos [56]
Invitational Drought TournamentHill, Hadarits, Rieger, Strickert, Davies and Strobbe [57]
IUP & NECAPRumore, Schenk and Susskind [58]
KEEP COOLEisenack [59]
Meya and Eisenack [60]
Lords of the ValleyMagnuszewski, et al. [61]
LottoSIMBecu, et al. [62]
MAE SALAE RPG 1 & 2Barnaud, Promburom, Trébuil and Bousquet [63]
Marine Spatial Planning ChallengeMayer, Zhou, Lo, Abspoel, Keijser, Olsen, Nixon and Kannen [64]
Keijser, Ripken, Mayer, Warmelink, Abspoel, Fairgrieve and Paris [65]
New-DistrictBecu, Frascaria-Lacoste and Latune [66]
REEFGAMECleland, Dray, Perez, Cruz-Trinidad and Geronimo [67]
ReHabPage, Dray, Perez and Garcia [68]
RESORTESSpeelman, García-Barrios, Groot and Tittonell [69]
SharivaDouven, Mul, Son, Bakker, Radosevich and Hendriks [70]
SimPhyAyadi, Le Bars, Le Grusse, Mandart, Fabre, Bouaziz and Bord [71]
SPRINTCITYDuffhues, Mayer, Nefs and van der Vliet [72]
Sustainable Delta Game (and adaptations)Valkering, van der Brugge, Offermans, Haasnoot and Vreugdenhil [73]
Lawrence and Haasnoot [74]
Van der Wal, De Kraker, Kroeze, Kirschner and Valkering [75]
Van Pelt, Haasnoot, Arts, Ludwig, Swart and Biesbroek [76]
TADLADionnet, Kuper, Hammani and Garin [77]
Ter’AguasDucrot, Van Paassen, Barban, Daré and Gramaglia [78]
The Floodplain Management GameStefanska, Magnuszewski, Sendzimir, Romaniuk, Taillieu, Dubel, Flachner and Balogh [79]
The Multi-Hazard Tournament Carson, et al. [80]
Unnamed gameSalvini, Van Paassen, Ligtenberg, Carrero and Bregt [81]
Unnamed gameHaug, Huitema and Wenzler [82]
Unnamed gameMeinzen-Dick, Janssen, Kandikuppa, Chaturvedi, Rao and Theis [83]
Unnamed gameSausse, Le Bail, Lecroart, Remy and Messéan [84]
Unnamed gameMoser and Mußhoff [85]
WeShareItOnencan, Van de Walle, Enserink, Chelang’a and Kulei [86]
WeShareIt NzoiaOnencan and Van de Walle [87]
WORLD CLIMATESterman, Franck, Fiddaman, Jones, McCauley, Rice, Sawin, Siegel and Rooney-Varga [88]
WSP gameFerrero, Bichai and Rusca [89]
Table 4. Overview of categorization for Research Question 1 (RQ1) on the conceptualization of learning through collaborative serious games.
Table 4. Overview of categorization for Research Question 1 (RQ1) on the conceptualization of learning through collaborative serious games.
Category (Number of Publications)Category DefinitionAssigned to Publications
Social learning (16)A change in understanding that goes beyond the individual to become situated within wider social units or communities of practice through social interactions between actors within social networks ([11], p. 6)[56,57,58,61,62,67,68,74,75,77,78,79,80,81,82,83]
Experiential learning (9)The process whereby knowledge is created through the transformation of experience ([90], p. 38)[50,51,54,62,66,68,75,88,89]
Other (11)Other learning conceptualization (see Table A1)[48,49,52,56,63,64,65,70,72,86,87]
Not conceptualized (10)Assessing whether or not playing the game led to led to actual changes beyond gameplay[53,55,59,60,69,71,73,76,84,85]
Table 5. Overview of categorization for RQ4 on when data is collected in the evaluation of collaborative serious games.
Table 5. Overview of categorization for RQ4 on when data is collected in the evaluation of collaborative serious games.
Category (Number of Publications)Category DefinitionAssigned to Publications
Pre (22)Data gathered before game sessions[48,50,55,58,59,60,61,64,66,69,70,73,75,76,78,80,81,82,85,86,87,88]
During (30)Data gathered during game sessions[48,50,51,53,54,55,56,59,60,61,63,64,65,67,68,69,71,72,73,74,75,76,78,79,82,83,84,86,87]
Post (37)Data gathered immediately after game sessions[48,49,50,52,53,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,72,73,74,75,76,77,78,79,80,81,82,83,86,87,89]
Post-post (10)Data gathered well after game sessions; not on the same day[58,63,69,76,77,78,80,82,83,88]
Table 6. Overview of categorization for RQ5 on the methods used in the evaluation of collaborative serious games.
Table 6. Overview of categorization for RQ5 on the methods used in the evaluation of collaborative serious games.
Category (Number of Publications)Category DefinitionAssigned to Publications
Questionnaires (25)Written or electronic questionnaires or surveys filled out by participants[48,50,55,57,58,59,60,61,64,65,66,69,70,72,76,77,78,80,82,83,85,86,87,88,89]
Observations (24)Observations performed in-situ or based on recordings[50,51,53,54,55,56,59,60,61,63,64,65,68,69,72,73,74,75,76,78,79,82,84,86]
Debriefings (18)Formal analysis of the game’s debriefing(s)[49,50,53,56,57,58,62,67,68,69,73,74,76,78,79,82,86,89]
Interviews (14)Interviews conducted with individual participants[53,58,63,64,69,72,74,75,76,77,78,81,82,83]
Data Logging (8)Data logged during gameplay, such as player decisions[64,71,72,75,83,84,85,87]
Control group (3)Data gathered through the use of a control group[76,83,85]
Perspective mapping (2)Classifying, interpreting and analysing different perspectives on promising paths for sustainable strategies [91][73,75]
Concept maps (1)Participants drawing structural representation of associations in relation to a topic [92][82]
Interaction Analysis (1)Analysis of human-to-human as well as human-environment interactions [93][48]
Social Network Analysis (1)Analysis of the affordances and hindrances to knowledge co-creation by structuring relationships [94][48]
Real-world data (1)Data gathered from the ‘real world’ research context or environment[83]
Table 7. Overview of categorization for RQ6 on use of qualitative and/or quantitative data in the evaluation of collaborative serious games.
Table 7. Overview of categorization for RQ6 on use of qualitative and/or quantitative data in the evaluation of collaborative serious games.
Category (Number of Publications)Category DefinitionAssigned to Publications
Qualitative (21)Evaluations based only on qualitative data[49,51,52,53,54,56,57,59,62,63,67,69,72,73,74,77,78,79,81,82,89]
Quantitative (5)Evaluations based only on quantitative data[60,71,85,87,88]
Qualitative and quantitative (16)Evaluations based on both qualitative and quantitative data[48,50,55,58,61,64,65,66,68,70,75,76,80,83,84,86]
Table 8. Overview of categorization for RQ5 on the learning effects of applying collaborative serious games according to their evaluations in relation to social learning.
Table 8. Overview of categorization for RQ5 on the learning effects of applying collaborative serious games according to their evaluations in relation to social learning.
Category (Number of Publications)Category DefinitionAssigned to Publications
Increased system understanding (34)Learning about the topic covered in the game, its complexity and possible trade-offs between decisions (cognitive learning)[49,50,51,52,53,55,56,57,58,59,60,62,63,64,65,67,68,70,71,72,73,74,76,77,78,80,81,82,83,84,85,86,87,88]
Raised awareness (14)Learning about the seriousness of an issue (cognitive learning)[49,51,52,54,58,60,70,72,76,79,80,87,88,89]
Change in views (5)Observing participants changing their viewpoints, values or paradigms (normative learning)[60,74,75,78,88]
Understanding other roles and perspectives (13)Learning about other participants’ roles, mind-sets and points of view (relational learning)[48,49,56,58,63,64,66,70,74,78,80,82,89]
Building relationships and trust (7)Enhancing participants’ ability to collaborate by building relationships and enhancing trust (relational learning)[48,49,52,55,69,80,81]
Table 9. Overview of approaches used to assess cognitive, relational and normative learning through collaborative serious games.
Table 9. Overview of approaches used to assess cognitive, relational and normative learning through collaborative serious games.
TypeAssessment ApproachData CollectionMethods
Cognitive learningSelf-reflective questionsPost-gameQuestionnaires (open or closed questions); interviews; debriefings
Knowledge measurementsPre- and post-gameQuestionnaires (open or closed questions); interviews; concept maps
Observed knowledge acquisitionIn-gameObservations (structured or unstructured, in-situ or from recordings); Analysing data logged during gameplay
Normative learningSelf-reflective questionsPost- and post-post gameInterviews
Shifts in opinion measurementsPre- and post-gameQuestionnaires (closed questions); perspective mapping
Relational learningSelf-reflective questionsPost- and post-post gameQuestionnaires (open or closed question); interviews; debriefings
Group dynamics analysisIn-gameQuestionnaires (closed questions) & interaction analysis and social network analysis
Back to TopTop