On Evaluating Social Learning Outcomes of Serious Games to Collaboratively Address Sustainability Problems: A Literature Review
Abstract
:1. Introduction
2. Materials and Methods
2.1. Related Work
2.2. Research Questions
- RQ1.
- How is learning through collaborative serious games conceptualized?
- RQ2.
- When is data collected in the evaluation of learning through collaborative serious games?
- RQ3.
- What methods are used in the evaluation of learning through collaborative serious games?
- RQ4.
- Do evaluations of learning through collaborative serious games use quantitative, qualitative or a combination between quantitative and qualitative data?
- RQ5.
- What are the learning effects of collaborative serious games according to their evaluations in relation to social learning?
2.3. Search Strategy
2.4. Study Selection
2.5. Study Quality Assessment
- Does the publication provide a conceptualization of learning? (RQ1)
- Does the publication state when data is collected in the evaluation of the collaborative serious game? (RQ2)
- Does the publication discuss a method, technique or theory used to evaluate the collaborative serious game? (RQ3)
- Does the publication discuss whether the evaluation used a quantitative or qualitative approach, or both in the evaluation of the collaborative serious game? (RQ4)
- Does the publication describe learning outcomes of the collaborative serious game following evaluation? (RQ5)
2.6. Snowballing
2.7. Data Extraction
2.8. Data Analysis and Categorization
3. Search Results
4. Categorization Results
4.1. How Is Learning through Collaborative Serious Games Conceptualized?
4.2. When is Data Collected in the Evaluation of Learning through Collaborative Serious Games?
4.3. What Methods are Used in the Evaluation of Learning through Collaborative Serious Games?
4.4. Do Evaluations of Learning through Collaborative Serious Games Use Quantitative, Qualitative or a Combination between Quantitative and Qualitative Data?
4.5. What Are the Learning Effects of Applying Collaborative Serious Games According to Their Evaluations in Relation to Social Learning?
5. Discussion
5.1. Cognitive Learning
5.2. Normative Learning
5.3. Relational Learning
5.4. Collective Action
5.5. Reflection and Limitations
6. Conclusions
Supplementary Materials
Funding
Conflicts of Interest
Appendix A
Game Name | Reference | Quality Assessment | Learning Conceptualization (RQ1) | Data Collection (RQ2) | Assessment Methods (RQ3) | Quantitative or Qualitative Data (RQ4) | Social Learning Effects (RQ5) |
---|---|---|---|---|---|---|---|
Aqua Republica | [48] | 5 | Boundary crossing | Pre & During & Post | Questionnaires & interaction analysis & social network analysis | Quantitative & qualitative | Understanding other perspectives Building relationships and trust |
CauxOperation | [49] | 5 | Collective learning | Post | Debriefing | Qualitative | Raised awareness Increased system understanding Understanding other perspectives Building relationships and trust |
Climate Game | [50] | 5 | Links to experiential learning | Pre & During & Post | Questionnaires & observations & debriefs | Quantitative & qualitative | Increased system understanding |
Community Cooperation Game | [51] | 5 | Links to experiential learning | During | Observations | Qualitative | Increased system understanding Raised awareness |
Forage Rummy | [52] | 5 | Knowledge co-production | Post | Direct feedback | Qualitative | Increased system understanding Raised awareness Building relationships and trust |
FOWIS | [53] | 4 | Not conceptualized | During & Post | Observations & debriefs & Interviews | Qualitative | Increased system understanding |
Futura | [54] | 5 | Experiential learning | During | Observations | Qualitative | Raised awareness |
GO2Zero | [55] | 4 | Not conceptualized | Pre & During & Post | Questionnaires & Observations | Quantitative & qualitative | Increased system understanding Building relationships and trust |
Grazing game | [56] | 5 | Social learning & Anticipatory learning | During & Post | Observations & debrief | Qualitative | Increased system understanding Understanding other perspectives |
Invitational Drought Tournament | [57] | 5 | Social learning | Post | Questionnaires & debriefs | Qualitative | Increased system understanding |
IUP & NECAP | [58] | 5 | Social learning | Pre & Post & Post-post | Questionnaires & debriefs & interviews | Quantitative & qualitative | Increased system understanding Raised awareness Understanding other perspectives |
KEEP COOL | [59] | 4 | Not conceptualized | Pre & During & Post | Questionnaires & observations | Qualitative | Increased system understanding |
[60] | 4 | Not conceptualized | Pre & During & Post | Questionnaire & Observations | Quantitative | Increased system understanding Raised awareness Change in views | |
Lords of the Valley | [61] | 4 | Social learning | Pre & During & Post | Questionnaires & observations | Quantitative & qualitative | No learning effects reported |
LottoSIM | [62] | 5 | Links to social learning & experiential learning | Post | Debriefs | Qualitative | Increased system understanding |
MAE SALAE RPG 1 & 2 | [63] | 5 | Collective learning | During & Post & Post-post | Observations & Interviews | Qualitative | Increased system understanding Understanding other perspectives |
Marine Spatial Planning Challenge | [64] | 5 | Constructivist learning philosophy | During & Post | Questionnaires & observations | Quantitative & qualitative | Increased system understanding |
[65] | 5 | Policy-oriented learning | Pre & During & Post | Questionnaires & observations & data logging & interviews | Quantitative & qualitative | Increased system understanding Building relationships and trust | |
New-District | [66] | 5 | Experiential learning | Pre & Post | Questionnaires | Quantitative & qualitative | Understanding other perspectives |
REEFGAME | [67] | 5 | Links to social learning | During & Post | Debriefs | Qualitative | Increased system understanding |
ReHab | [68] | 5 | Social learning & experiential learning | During & Post | Observations & debriefings | Quantitative & qualitative | Increased system understanding |
RESORTES | [69] | 4 | Not conceptualized | Pre & During & Post & Post-post | Questionnaires & observations & debrief & interview | Qualitative | Building relationships and trust |
Shariva | [70] | 5 | Meaningful play | Pre & Post | Questionnaires | Quantitative & qualitative | Raised awareness Increased system understanding Understanding other perspectives |
SimPhy | [71] | 4 | Not conceptualized | During | Data logging | Quantitative | Increased system understanding |
SPRINTCITY | [72] | 5 | Policy oriented learning | During & Post | Questionnaires & observations & data logging & interviews | Qualitative | Raised awareness Increased system understanding |
Sustainable Delta Game (and adaptations) | [73] | 4 | Not conceptualized | Pre & During & Post | Observations & debriefs & perspective mapping | Qualitative | Increased system understanding Understanding other perspectives |
[74] | 5 | Social learning | During & Post | Observations & interviews & debriefs | Qualitative | Increased system understanding Change in views | |
[75] | 5 | Social learning & experiential learning | Pre & During & Post | Observations & data logging & interviews & perspective mapping | Quantitative & qualitative | Change in views | |
[76] | 4 | Not conceptualized | Pre & During & Post & Post-post & Control Group | Questionnaires & observations & debriefs & interviews & control group | Quantitative & qualitative | Increased system understanding Raised awareness Establish experiential environment | |
TADLA | [77] | 5 | Individual learning & Social learning | Post & Post-post | Questionnaires & Interviews | Qualitative | Increased system understanding |
Ter’Aguas | [78] | 5 | Social learning | Pre & During & Post & Post-post | Questionnaires & observations & debriefs & interviews | Qualitative | Increased system understanding Understanding other perspectives Change in views |
The Floodplain Management Game | [79] | 5 | Social learning | During & Post | Observations & debriefs | Qualitative | Raised awareness |
The Multi-Hazard Tournament (MHT) | [80] | 5 | Social learning | Pre & Post & Post-post | Questionnaires | Quantitative & qualitative | Increased system understanding Raised awareness Understanding other perspectives Building relationships and trust |
Unnamed game | [81] | 5 | Social learning | Pre & Post | Interviews | Qualitative | Increased system understanding Building relationships and trust |
Unnamed game | [82] | 5 | Social learning | Pre & During & Post & Post-post | Questionnaires & Observations & Debriefs & Interviews & Concept maps | Qualitative | Increased system understanding Understanding other perspectives |
Unnamed game | [83] | 5 | Social learning | During & Post & Post-post & Control group | Questionnaires & data logging & interviews & real-world data | Quantitative & qualitative | Increased system understanding Real-world adoption |
Unnamed game | [84] | 4 | Not conceptualized | During | Observations & data logging | Quantitative & Qualitative | Increased system understanding |
Unnamed game | [85] | 4 | Not conceptualized | Pre & During | Questionnaires & data logging & control group | Quantitative | Increased system understanding |
WeShareIt | [86] | 5 | Effective learning | Pre & During & Post | Questionnaires & observations & debrief | Quantitative & qualitative | Increased system understanding |
WeShareIt Nzoia | [87] | 5 | Situational awareness | Pre & During & Post | Questionnaires & data logging | Quantitative | Raised awareness Increased system understanding |
WORLD CLIMATE | [88] | 5 | Links to experiential learning | Pre & Post-post | Questionnaires | Quantitative | Increased system understanding Raised awareness Change in views |
WSP game | [89] | 5 | Experiential learning | Post | Questionnaires & debrief | Qualitative | Raised awareness Building relationships and trust |
References
- Folke, C.; Carpenter, S.; Elmqvist, T.; Gunderson, L.; Holling, C.S.; Walker, B.; Bengtsson, J.; Berkes, F.; Colding, J.; Danell, K.; et al. Resilience and Sustainable Development: Building Adaptive Capacity in a World of Transformations. Ambio 2002, 31, 437–440. [Google Scholar] [CrossRef] [Green Version]
- Berkes, F. Evolution of co-management: Role of knowledge generation, bridging organizations and social learning. J. Environ. Manag. 2009, 90, 1692–1702. [Google Scholar] [CrossRef] [PubMed]
- Rodela, R. Social learning and natural resource management: The emergence of three research perspectives. Ecol. Soc. 2011, 16, 30. [Google Scholar] [CrossRef]
- Pahl-Wostl, C. The implications of complexity for integrated resources management. Environ. Model. Softw. 2007, 22, 561–569. [Google Scholar] [CrossRef] [Green Version]
- Armitage, D.; Berkes, F.; Doubleday, N. Adaptive Co-Management: Collaboration, Learning, and Multi-Level Governance; UBC Press: Vancouver, BC, Canada, 2007. [Google Scholar]
- Folke, C.; Hahn, T.; Olsson, P.; Norberg, J. Adaptive governance of social-ecological systems. Annu. Rev. Environ. Resour. 2005, 30, 441–473. [Google Scholar] [CrossRef]
- Olsson, P.; Folke, C.; Berkes, F. Adaptive comanagement for building resilience in social–ecological systems. Environ. Manag. 2004, 34, 75–90. [Google Scholar] [CrossRef] [PubMed]
- Crona, B.I.; Parker, J.N. Learning in support of governance: Theories, methods, and a framework to assess how bridging organizations contribute to adaptive resource governance. Ecol. Soc. 2012, 17, 32. [Google Scholar] [CrossRef]
- Pahl-Wostl, C. The importance of social learning in restoring the multifunctionality of rivers and floodplains. Ecol. Soc. 2006, 11, 10. [Google Scholar] [CrossRef]
- Armitage, D. Adaptive capacity and community-based natural resource management. Environ. Manag. 2005, 35, 703–715. [Google Scholar] [CrossRef] [PubMed]
- Reed, M.S.; Evely, A.C.; Cundill, G.; Fazey, I.; Glass, J.; Laing, A.; Newig, J.; Parrish, B.; Prell, C.; Raymond, C. What is social learning? Ecol. Soc. 2010, 15. Available online: https://www.ecologyandsociety.org/vol15/iss4/resp1/ (accessed on 1 October 2018). [CrossRef]
- Armitage, D.; Marschke, M.; Plummer, R. Adaptive co-management and the paradox of learning. Glob. Environ. Chang. 2008, 18, 86–98. [Google Scholar] [CrossRef]
- Muro, M.; Jeffrey, P. A critical review of the theory and application of social learning in participatory natural resource management processes. J. Environ. Plan. Manag. 2008, 51, 325–344. [Google Scholar] [CrossRef]
- Cundill, G.; Rodela, R. A review of assertions about the processes and outcomes of social learning in natural resource management. J. Environ. Manag. 2012, 113, 7–14. [Google Scholar] [CrossRef] [PubMed]
- Keen, M.; Brown, V.A.; Dyball, R. Social Learning in Environmental Management: Towards a Sustainable Future; Routledge: Abingdon, UK, 2005. [Google Scholar]
- Pahl-Wostl, C.; Craps, M.; Dewulf, A.; Mostert, E.; Tabara, D.; Taillieu, T. Social learning and water resources management. Ecol. Soc. 2007, 12, 5. [Google Scholar] [CrossRef]
- Baird, J.; Plummer, R.; Haug, C.; Huitema, D. Learning effects of interactive decision-making processes for climate change adaptation. Glob. Environ. Chang. 2014, 27, 51–63. [Google Scholar] [CrossRef]
- Pahl-Wostl, C.; Tàbara, D.; Bouwen, R.; Craps, M.; Dewulf, A.; Mostert, E.; Ridder, D.; Taillieu, T. The importance of social learning and culture for sustainable water management. Ecol. Econ. 2008, 64, 484–495. [Google Scholar] [CrossRef]
- Furber, A.; Medema, W.; Adamowski, J. Assessing the benefits of serious games to support sustainable decision-making for transboundary watershed governance. Can. Water Resour. J./Rev. Can. Ressour. Hydr. 2018, 1–15. [Google Scholar] [CrossRef]
- Medema, W.; Furber, A.; Adamowski, J.; Zhou, Q.; Mayer, I. Exploring the potential impact of serious games on social learning and stakeholder collaborations for transboundary watershed management of the St. Lawrence River Basin. Water 2016, 8, 175. [Google Scholar] [CrossRef]
- Marini, D.; Medema, W.; Adamowski, J.; Veissière, S.; Mayer, I.; Wals, A. Socio-Psychological Perspectives on the Potential for Serious Games to Promote Transcendental Values in IWRM Decision-Making. Water 2018, 10, 1097. [Google Scholar] [CrossRef]
- Ampatzidou, C.; Gugerell, K.; Constantinescu, T.; Devisch, O.; Jauschneg, M.; Berger, M. All Work and No Play? Facilitating Serious Games and Gamified Applications in Participatory Urban Planning and Governance. Urban Plan. 2018, 3, 34–46. [Google Scholar] [CrossRef]
- Susi, T.; Johannesson, M.; Backlund, P. Serious Games: An Overview; Institutionen för Kommunikation och Information: Lund, Sweden, 2007. [Google Scholar]
- Michael, D.R.; Chen, S.L. Serious Games: Games That Educate, Train, and Inform; Thomson Course Technology: Boston, MA, USA, 2006. [Google Scholar]
- Abt, C.C. Serious Games; University Press of America: Lanham, MA, USA, 1987. [Google Scholar]
- Mayer, I. The gaming of policy and the politics of gaming: A review. Simul. Gaming 2009, 40, 825–862. [Google Scholar] [CrossRef]
- Bekebrede, G. Experiencing Complexity: A Gaming Approach for Understanding Infrastructure System. Ph.D. Thesis, TU Delft, Delft University of Technology, Delft, The Netherlands, 2010. [Google Scholar]
- Roschelle, J.; Teasley, S.D. The construction of shared knowledge in collaborative problem solving. In Proceedings of the Computer Supported Collaborative Learning; Springer: Berlin/Heidelberg, Germany, 1995; pp. 69–97. [Google Scholar]
- Geurts, J.L.; Duke, R.D.; Vermeulen, P.A. Policy gaming for strategy and change. Long Range Plan. 2007, 40, 535–558. [Google Scholar] [CrossRef]
- Mayer, I.; Bekebrede, G.; Harteveld, C.; Warmelink, H.; Zhou, Q.; van Ruijven, T.; Lo, J.; Kortmann, R.; Wenzler, I. The research and evaluation of serious games: Toward a comprehensive methodology. Br. J. Educ. Technol. 2014, 45, 502–527. [Google Scholar] [CrossRef]
- De Caluwé, L.; Geurts, J.; Kleinlugtenbelt, W.J. Gaming research in policy and organization: An assessment from the Netherlands. Simul. Gaming 2012, 43, 600–626. [Google Scholar] [CrossRef]
- Hofstede, G.J.; De Caluwé, L.; Peters, V. Why simulation games work-in search of the active substance: A synthesis. Simul. Gaming 2010, 41, 824–843. [Google Scholar] [CrossRef]
- Kitchenham, B.; Charters, S. Guidelines for Performing Systematic Literature Reviews in Software Engineering; Version 2.3; University of Keele (Software Engineering Group, School of Computer Science and Mathematics) and University of Durham (Department of Computer Science): Keele and Durham, UK, 2007. [Google Scholar]
- Kitchenham, B. Procedures for Performing Systematic Reviews; Keele University: Keele, UK, 2004; Volume 33, pp. 1–26. [Google Scholar]
- Calderón, A.; Ruiz, M. A systematic literature review on serious games evaluation: An application to software project management. Comput. Educ. 2015, 87, 396–422. [Google Scholar] [CrossRef]
- Connolly, T.M.; Boyle, E.A.; MacArthur, E.; Hainey, T.; Boyle, J.M. A systematic literature review of empirical evidence on computer games and serious games. Comput. Educ. 2012, 59, 661–686. [Google Scholar] [CrossRef]
- Backlund, P.; Hendrix, M. Educational games-are they worth the effort? A literature survey of the effectiveness of serious games. In Proceedings of the 2013 5th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES), Poole, UK, 11–13 September 2013; pp. 1–8. [Google Scholar]
- Boyle, E.A.; Hainey, T.; Connolly, T.M.; Gray, G.; Earp, J.; Ott, M.; Lim, T.; Ninaus, M.; Ribeiro, C.; Pereira, J. An update to the systematic literature review of empirical evidence of the impacts and outcomes of computer games and serious games. Comput. Educ. 2016, 94, 178–192. [Google Scholar] [CrossRef]
- Wouters, P.; Van der Spek, E.D.; Van Oostendorp, H. Current practices in serious game research: A review from a learning outcomes perspective. In Games-Based Learning Advancements for multi-Sensory Human Computer Interfaces: Techniques and Effective Practices; IGI Global: Hershey, PA, USA, 2009; pp. 232–250. [Google Scholar]
- Girard, C.; Ecalle, J.; Magnan, A. Serious games as new educational tools: How effective are they? A meta-analysis of recent studies. J. Comput. Assist. Learn. 2013, 29, 207–219. [Google Scholar] [CrossRef]
- Katsaliaki, K.; Mustafee, N. A survey of serious games on sustainable development. In Proceedings of the Winter Simulation Conference, Berlin, Germany, 9–12 December 2012; p. 136. [Google Scholar]
- Reckien, D.; Eisenack, K. Climate change gaming on board and screen: A review. Simul. Gaming 2013, 44, 253–271. [Google Scholar] [CrossRef]
- Flood, S.; Cradock-Henry, N.A.; Blackett, P.; Edwards, P. Adaptive and interactive climate futures: Systematic review of ‘serious games’ for engagement and decision-making. Environ. Res. Lett. 2018, 13, 063005. [Google Scholar] [CrossRef]
- Wu, J.S.; Lee, J.J. Climate change games as tools for education and engagement. Nat. Clim. Chang. 2015, 5, 413. [Google Scholar] [CrossRef]
- Aubert, A.H.; Bauer, R.; Lienert, J. A review of water-related serious games to specify use in environmental Multi-Criteria Decision Analysis. Environ. Model. Softw. 2018, 105, 64–78. [Google Scholar] [CrossRef]
- Wohlin, C. Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, London, UK, 13–14 May 2014; p. 38. [Google Scholar]
- Moher, D.; Liberati, A.; Tetzlaff, J.; Altman, D.G. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Ann. Intern. Med. 2009, 151, 264–269. [Google Scholar] [CrossRef] [PubMed]
- Jean, S.; Medema, W.; Adamowski, J.; Chew, C.; Delaney, P.; Wals, A. Serious games as a catalyst for boundary crossing, collaboration and knowledge co-creation in a watershed governance context. J. Environ. Manag. 2018, 223, 1010–1022. [Google Scholar] [CrossRef] [PubMed]
- Souchère, V.; Millair, L.; Echeverria, J.; Bousquet, F.; Le Page, C.; Etienne, M. Co-constructing with stakeholders a role-playing game to initiate collective management of erosive runoff risks at the watershed scale. Environ. Model. Softw. 2010, 25, 1359–1370. [Google Scholar] [CrossRef]
- Zhou, Q.; Bekebrede, G.; Mayer, I.; Warmerdam, J.; Knepflé, M. The Climate Game: Connecting Water Management and Spatial Planning through Simulation Gaming? In Water Governance as Connective Capacity; Edelenbos, J., Bressers, N., Scholten, P., Eds.; Routledge: Abingdon, UK, 2013; pp. 109–127. [Google Scholar]
- Tanwattana, P.; Toyoda, Y. Contributions of gaming simulation in building community-based disaster risk management applying Japanese case to flood prone communities in Thailand upstream area. Int. J. Disaster Risk Reduct. 2018, 27, 199–213. [Google Scholar] [CrossRef]
- Martin, G. A conceptual framework to support adaptation of farming systems–Development and application with Forage Rummy. Agric. Syst. 2015, 132, 52–61. [Google Scholar] [CrossRef]
- Hertzog, T.; Poussin, J.-C.; Tangara, B.; Kouriba, I.; Jamin, J.-Y. A role playing game to address future water management issues in a large irrigated system: Experience from Mali. Agric. Water Manag. 2014, 137, 1–14. [Google Scholar] [CrossRef]
- Antle, A.N.; Bevans, A.; Tanenbaum, J.; Seaborn, K.; Wang, S. Futura: Design for collaborative learning and game play on a multi-touch digital tabletop. In Proceedings of the Fifth International Conference on Tangible, Embedded, and Embodied Interaction, Funchal, Portugal, 22–26 January 2011; pp. 93–100. [Google Scholar]
- Bekebrede, G.; van Bueren, E.; Wenzler, I. Towards a Joint Local Energy Transition Process in Urban Districts: The GO2Zero Simulation Game. Sustainability 2018, 10, 2602. [Google Scholar] [CrossRef]
- Villamor, G.B.; Badmos, B.K. Grazing game: A learning tool for adaptive management in response to climate variability in semiarid areas of Ghana. Ecol. Soc. 2016, 21, 39. [Google Scholar] [CrossRef]
- Hill, H.; Hadarits, M.; Rieger, R.; Strickert, G.; Davies, E.G.; Strobbe, K.M. The Invitational Drought Tournament: What is it and why is it a useful tool for drought preparedness and adaptation? Weather Clim. Extremes 2014, 3, 107–116. [Google Scholar] [CrossRef]
- Rumore, D.; Schenk, T.; Susskind, L. Role-play simulations for climate change adaptation education and engagement. Nat. Clim. Chang. 2016, 6, 745. [Google Scholar] [CrossRef]
- Eisenack, K. A climate change board game for interdisciplinary communication and education. Simul. Gaming 2013, 44, 328–348. [Google Scholar] [CrossRef]
- Meya, J.N.; Eisenack, K. Effectiveness of gaming for communicating and teaching climate change. Clim. Chang. 2018, 149, 319–333. [Google Scholar] [CrossRef]
- Magnuszewski, P.; Królikowska, K.; Koch, A.; Pająk, M.; Allen, C.; Chraibi, V.; Giri, A.; Haak, D.; Hart, N.; Hellman, M. Exploring the Role of Relational Practices in Water Governance Using a Game-Based Approach. Water 2018, 10, 346. [Google Scholar] [CrossRef]
- Becu, N.; Amalric, M.; Anselme, B.; Beck, E.; Bertin, X.; Delay, E.; Long, N.; Marilleau, N.; Pignon-Mussaud, C.; Rousseaux, F. Participatory simulation to foster social learning on coastal flooding prevention. Environ. Model. Softw. 2017, 98, 1–11. [Google Scholar] [CrossRef]
- Barnaud, C.; Promburom, T.; Trébuil, G.; Bousquet, F. An evolving simulation/gaming process to facilitate adaptive watershed management in northern mountainous Thailand. Simul. Gaming 2007, 38, 398–420. [Google Scholar] [CrossRef] [Green Version]
- Mayer, I.; Zhou, Q.; Lo, J.; Abspoel, L.; Keijser, X.; Olsen, E.; Nixon, E.; Kannen, A. Integrated, ecosystem-based Marine Spatial Planning: Design and results of a game-based, quasi-experiment. Ocean Coast. Manag. 2013, 82, 7–26. [Google Scholar] [CrossRef]
- Keijser, X.; Ripken, M.; Mayer, I.; Warmelink, H.; Abspoel, L.; Fairgrieve, R.; Paris, C. Stakeholder Engagement in Maritime Spatial Planning: The Efficacy of a Serious Game Approach. Water 2018, 10, 724. [Google Scholar] [CrossRef]
- Becu, N.; Frascaria-Lacoste, N.; Latune, J. Experiential Learning Based on the Newdistrict Asymmetric Simulation Game: Results of a Dozen Gameplay Sessions. 2015. Available online: https://hal.archives-ouvertes.fr/hal-01253024/document (accessed on 7 January 2018).
- Cleland, D.; Dray, A.; Perez, P.; Cruz-Trinidad, A.; Geronimo, R. Simulating the dynamics of subsistence fishing communities: REEFGAME as a learning and data-gathering computer-assisted role-play game. Simul. Gaming 2012, 43, 102–117. [Google Scholar] [CrossRef]
- Page, C.L.; Dray, A.; Perez, P.; Garcia, C. Exploring how knowledge and communication influence natural resources management with REHAB. Simul. Gaming 2016, 47, 257–284. [Google Scholar] [CrossRef]
- Speelman, E.; García-Barrios, L.; Groot, J.; Tittonell, P. Gaming for smallholder participation in the design of more sustainable agricultural landscapes. Agric. Syst. 2014, 126, 62–75. [Google Scholar] [CrossRef]
- Douven, W.; Mul, M.L.; Son, L.; Bakker, N.; Radosevich, G.; Hendriks, A. Games to create awareness and design policies for transboundary cooperation in river basins: Lessons from the shariva game of the Mekong river commission. Water Resour. Manag. 2014, 28, 1431–1447. [Google Scholar] [CrossRef]
- Ayadi, H.; Le Bars, M.; Le Grusse, P.; Mandart, E.; Fabre, J.; Bouaziz, A.; Bord, J. SimPhy: A simulation game to lessen the impact of phytosanitaries on health and the environment—The case of Merja Zerga in Morocco. Environ. Sci. Pollut. Res. 2014, 21, 4950–4963. [Google Scholar] [CrossRef] [PubMed]
- Duffhues, J.; Mayer, I.S.; Nefs, M.; van der Vliet, M. Breaking barriers to transit-oriented development: Insights from the serious game SPRINTCITY. Environ. Plan. B Plan. Des. 2014, 41, 770–791. [Google Scholar] [CrossRef]
- Valkering, P.; van der Brugge, R.; Offermans, A.; Haasnoot, M.; Vreugdenhil, H. A perspective-based simulation game to explore future pathways of a water-society system under climate change. Simul. Gaming 2013, 44, 366–390. [Google Scholar] [CrossRef]
- Lawrence, J.; Haasnoot, M. What it took to catalyse uptake of dynamic adaptive pathways planning to address climate change uncertainty. Environ. Sci. Policy 2017, 68, 47–57. [Google Scholar] [CrossRef]
- Van der Wal, M.M.; De Kraker, J.; Kroeze, C.; Kirschner, P.A.; Valkering, P. Can computer models be used for social learning? A serious game in water management. Environ. Model. Softw. 2016, 75, 119–132. [Google Scholar] [CrossRef] [Green Version]
- Van Pelt, S.; Haasnoot, M.; Arts, B.; Ludwig, F.; Swart, R.; Biesbroek, R. Communicating climate (change) uncertainties: Simulation games as boundary objects. Environ. Sci. Policy 2015, 45, 41–52. [Google Scholar] [CrossRef]
- Dionnet, M.; Kuper, M.; Hammani, A.; Garin, P. Combining role-playing games and policy simulation exercises: An experience with Moroccan smallholder farmers. Simul. Gaming 2008, 39, 498–514. [Google Scholar] [CrossRef]
- Ducrot, R.; Van Paassen, A.; Barban, V.; Daré, W.s.; Gramaglia, C. Learning integrative negotiation to manage complex environmental issues: Example of a gaming approach in the peri-urban catchment of Sao Paulo, Brazil. Reg. Environ. Chang. 2015, 15, 67–78. [Google Scholar] [CrossRef]
- Stefanska, J.; Magnuszewski, P.; Sendzimir, J.; Romaniuk, P.; Taillieu, T.; Dubel, A.; Flachner, Z.; Balogh, P. A Gaming Exercise to Explore Problem-Solving versus Relational Activities for River Floodplain Management. Environ. Policy Gov. 2011, 21, 454–471. [Google Scholar] [CrossRef]
- Carson, A.; Windsor, M.; Hill, H.; Haigh, T.; Wall, N.; Smith, J.; Olsen, R.; Bathke, D.; Demir, I.; Muste, M. Serious gaming for participatory planning of multi-hazard mitigation. Int. J. River Basin Manag. 2018, 16, 379–391. [Google Scholar] [CrossRef]
- Salvini, G.; Van Paassen, A.; Ligtenberg, A.; Carrero, G.; Bregt, A. A role-playing game as a tool to facilitate social learning and collective action towards Climate Smart Agriculture: Lessons learned from Apuí, Brazil. Environ. Sci. Policy 2016, 63, 113–121. [Google Scholar] [CrossRef]
- Haug, C.; Huitema, D.; Wenzler, I. Learning through games? Evaluating the learning effect of a policy exercise on European climate policy. Technol. Forecast. Soc. Chang. 2011, 78, 968–981. [Google Scholar] [CrossRef]
- Meinzen-Dick, R.; Janssen, M.A.; Kandikuppa, S.; Chaturvedi, R.; Rao, K.; Theis, S. Playing games to save water: Collective action games for groundwater management in Andhra Pradesh, India. World Dev. 2018, 107, 40–53. [Google Scholar] [CrossRef]
- Sausse, C.; Le Bail, M.; Lecroart, B.; Remy, B.; Messéan, A. How to manage the coexistence between genetically modified and conventional crops in grain and oilseed collection areas? Elaboration of scenarios using role playing games. Land Use Policy 2013, 30, 719–729. [Google Scholar] [CrossRef]
- Moser, S.; Mußhoff, O. Ex-ante evaluation of policy measures: Effects of reward and punishment for fertiliser reduction in palm oil production. J. Agric. Econ. 2016, 67, 84–104. [Google Scholar] [CrossRef]
- Onencan, A.; Van de Walle, B.; Enserink, B.; Chelang’a, J.; Kulei, F. WeShareIt Game: Strategic foresight for climate-change induced disaster risk reduction. Procedia Eng. 2016, 159, 307–315. [Google Scholar] [CrossRef]
- Onencan, A.M.; Van de Walle, B. From Paris Agreement to Action: Enhancing Climate Change Familiarity and Situation Awareness. Sustainability 2018, 10, 1929. [Google Scholar] [CrossRef]
- Sterman, J.; Franck, T.; Fiddaman, T.; Jones, A.; McCauley, S.; Rice, P.; Sawin, E.; Siegel, L.; Rooney-Varga, J.N. World Climate: A role-play simulation of climate negotiations. Simul. Gaming 2015, 46, 348–382. [Google Scholar] [CrossRef]
- Ferrero, G.; Bichai, F.; Rusca, M. Experiential Learning through Role-Playing: Enhancing Stakeholder Collaboration in Water Safety Plans. Water 2018, 10, 227. [Google Scholar] [CrossRef]
- Kolb, D.A. Experiential Learning: Experience as the Source of Learning and Development; Prentice Hall: Englewood Cliffs, NJ, USA, 1984. [Google Scholar]
- Offermans, A.; Haasnoot, M.; Valkering, P. A method to explore social response for sustainable water management strategies under changing conditions. Sustain. Dev. 2011, 19, 312–324. [Google Scholar] [CrossRef]
- Ruiz-Primo, M.A.; Shavelson, R.J. Problems and issues in the use of concept maps in science assessment. J. Res. Sci. Teach. Off. J. Natl. Assoc. Res. Sci. Teach. 1996, 33, 569–600. [Google Scholar] [CrossRef]
- Jordan, B.; Henderson, A. Interaction analysis: Foundations and practice. J. Learn. Sci. 1995, 4, 39–103. [Google Scholar] [CrossRef]
- Serrat, O. Social network analysis. In Knowledge Solutions; Springer: Singapore, 2017; pp. 39–43. [Google Scholar]
- Taylor, R. Situational awareness rating technique (SART): The development of a tool for aircrew systems design. In Situation Awareness in Aerospace Operations; NATO—Advisory Group for Aerospace Research and Development: Neuilly-Sur-Seine, France, 1990; pp. 1–17. [Google Scholar]
Inclusion criteria |
|
Exclusion criteria |
|
Type | Definition/Indicators of Learning Effects | Measures of Indicators |
---|---|---|
Cognitive learning | Acquisition of new knowledge; restructuring of existing knowledge | Test scores; change in centrality and specificity of knowledge presented in concept maps |
Normative learning | Changes in norms; change in values; change in paradigms; convergence of group opinion | Change in, and convergence around, environmental beliefs; participant reflections; meeting proceedings |
Relational learning | Improved understanding of mind sets of others; building of relationships; enhanced trust and cooperation | Change in social network structure; participant reflections |
Game Name | Reference |
---|---|
Aqua Republica | Jean, Medema, Adamowski, Chew, Delaney and Wals [48] |
CauxOperation | Souchère, Millair, Echeverria, Bousquet, Le Page and Etienne [49] |
Climate Game | Zhou, Bekebrede, Mayer, Warmerdam and Knepflé [50] |
Community Cooperation Game | Tanwattana and Toyoda [51] |
Forage Rummy | Martin [52] |
FOWIS | Hertzog, Poussin, Tangara, Kouriba and Jamin [53] |
Futura | Antle, Bevans, Tanenbaum, Seaborn and Wang [54] |
GO2Zero | Bekebrede, van Bueren and Wenzler [55] |
Grazing game | Villamor and Badmos [56] |
Invitational Drought Tournament | Hill, Hadarits, Rieger, Strickert, Davies and Strobbe [57] |
IUP & NECAP | Rumore, Schenk and Susskind [58] |
KEEP COOL | Eisenack [59] |
Meya and Eisenack [60] | |
Lords of the Valley | Magnuszewski, et al. [61] |
LottoSIM | Becu, et al. [62] |
MAE SALAE RPG 1 & 2 | Barnaud, Promburom, Trébuil and Bousquet [63] |
Marine Spatial Planning Challenge | Mayer, Zhou, Lo, Abspoel, Keijser, Olsen, Nixon and Kannen [64] |
Keijser, Ripken, Mayer, Warmelink, Abspoel, Fairgrieve and Paris [65] | |
New-District | Becu, Frascaria-Lacoste and Latune [66] |
REEFGAME | Cleland, Dray, Perez, Cruz-Trinidad and Geronimo [67] |
ReHab | Page, Dray, Perez and Garcia [68] |
RESORTES | Speelman, García-Barrios, Groot and Tittonell [69] |
Shariva | Douven, Mul, Son, Bakker, Radosevich and Hendriks [70] |
SimPhy | Ayadi, Le Bars, Le Grusse, Mandart, Fabre, Bouaziz and Bord [71] |
SPRINTCITY | Duffhues, Mayer, Nefs and van der Vliet [72] |
Sustainable Delta Game (and adaptations) | Valkering, van der Brugge, Offermans, Haasnoot and Vreugdenhil [73] |
Lawrence and Haasnoot [74] | |
Van der Wal, De Kraker, Kroeze, Kirschner and Valkering [75] | |
Van Pelt, Haasnoot, Arts, Ludwig, Swart and Biesbroek [76] | |
TADLA | Dionnet, Kuper, Hammani and Garin [77] |
Ter’Aguas | Ducrot, Van Paassen, Barban, Daré and Gramaglia [78] |
The Floodplain Management Game | Stefanska, Magnuszewski, Sendzimir, Romaniuk, Taillieu, Dubel, Flachner and Balogh [79] |
The Multi-Hazard Tournament | Carson, et al. [80] |
Unnamed game | Salvini, Van Paassen, Ligtenberg, Carrero and Bregt [81] |
Unnamed game | Haug, Huitema and Wenzler [82] |
Unnamed game | Meinzen-Dick, Janssen, Kandikuppa, Chaturvedi, Rao and Theis [83] |
Unnamed game | Sausse, Le Bail, Lecroart, Remy and Messéan [84] |
Unnamed game | Moser and Mußhoff [85] |
WeShareIt | Onencan, Van de Walle, Enserink, Chelang’a and Kulei [86] |
WeShareIt Nzoia | Onencan and Van de Walle [87] |
WORLD CLIMATE | Sterman, Franck, Fiddaman, Jones, McCauley, Rice, Sawin, Siegel and Rooney-Varga [88] |
WSP game | Ferrero, Bichai and Rusca [89] |
Category (Number of Publications) | Category Definition | Assigned to Publications |
---|---|---|
Social learning (16) | A change in understanding that goes beyond the individual to become situated within wider social units or communities of practice through social interactions between actors within social networks ([11], p. 6) | [56,57,58,61,62,67,68,74,75,77,78,79,80,81,82,83] |
Experiential learning (9) | The process whereby knowledge is created through the transformation of experience ([90], p. 38) | [50,51,54,62,66,68,75,88,89] |
Other (11) | Other learning conceptualization (see Table A1) | [48,49,52,56,63,64,65,70,72,86,87] |
Not conceptualized (10) | Assessing whether or not playing the game led to led to actual changes beyond gameplay | [53,55,59,60,69,71,73,76,84,85] |
Category (Number of Publications) | Category Definition | Assigned to Publications |
---|---|---|
Pre (22) | Data gathered before game sessions | [48,50,55,58,59,60,61,64,66,69,70,73,75,76,78,80,81,82,85,86,87,88] |
During (30) | Data gathered during game sessions | [48,50,51,53,54,55,56,59,60,61,63,64,65,67,68,69,71,72,73,74,75,76,78,79,82,83,84,86,87] |
Post (37) | Data gathered immediately after game sessions | [48,49,50,52,53,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,72,73,74,75,76,77,78,79,80,81,82,83,86,87,89] |
Post-post (10) | Data gathered well after game sessions; not on the same day | [58,63,69,76,77,78,80,82,83,88] |
Category (Number of Publications) | Category Definition | Assigned to Publications |
---|---|---|
Questionnaires (25) | Written or electronic questionnaires or surveys filled out by participants | [48,50,55,57,58,59,60,61,64,65,66,69,70,72,76,77,78,80,82,83,85,86,87,88,89] |
Observations (24) | Observations performed in-situ or based on recordings | [50,51,53,54,55,56,59,60,61,63,64,65,68,69,72,73,74,75,76,78,79,82,84,86] |
Debriefings (18) | Formal analysis of the game’s debriefing(s) | [49,50,53,56,57,58,62,67,68,69,73,74,76,78,79,82,86,89] |
Interviews (14) | Interviews conducted with individual participants | [53,58,63,64,69,72,74,75,76,77,78,81,82,83] |
Data Logging (8) | Data logged during gameplay, such as player decisions | [64,71,72,75,83,84,85,87] |
Control group (3) | Data gathered through the use of a control group | [76,83,85] |
Perspective mapping (2) | Classifying, interpreting and analysing different perspectives on promising paths for sustainable strategies [91] | [73,75] |
Concept maps (1) | Participants drawing structural representation of associations in relation to a topic [92] | [82] |
Interaction Analysis (1) | Analysis of human-to-human as well as human-environment interactions [93] | [48] |
Social Network Analysis (1) | Analysis of the affordances and hindrances to knowledge co-creation by structuring relationships [94] | [48] |
Real-world data (1) | Data gathered from the ‘real world’ research context or environment | [83] |
Category (Number of Publications) | Category Definition | Assigned to Publications |
---|---|---|
Qualitative (21) | Evaluations based only on qualitative data | [49,51,52,53,54,56,57,59,62,63,67,69,72,73,74,77,78,79,81,82,89] |
Quantitative (5) | Evaluations based only on quantitative data | [60,71,85,87,88] |
Qualitative and quantitative (16) | Evaluations based on both qualitative and quantitative data | [48,50,55,58,61,64,65,66,68,70,75,76,80,83,84,86] |
Category (Number of Publications) | Category Definition | Assigned to Publications |
---|---|---|
Increased system understanding (34) | Learning about the topic covered in the game, its complexity and possible trade-offs between decisions (cognitive learning) | [49,50,51,52,53,55,56,57,58,59,60,62,63,64,65,67,68,70,71,72,73,74,76,77,78,80,81,82,83,84,85,86,87,88] |
Raised awareness (14) | Learning about the seriousness of an issue (cognitive learning) | [49,51,52,54,58,60,70,72,76,79,80,87,88,89] |
Change in views (5) | Observing participants changing their viewpoints, values or paradigms (normative learning) | [60,74,75,78,88] |
Understanding other roles and perspectives (13) | Learning about other participants’ roles, mind-sets and points of view (relational learning) | [48,49,56,58,63,64,66,70,74,78,80,82,89] |
Building relationships and trust (7) | Enhancing participants’ ability to collaborate by building relationships and enhancing trust (relational learning) | [48,49,52,55,69,80,81] |
Type | Assessment Approach | Data Collection | Methods |
---|---|---|---|
Cognitive learning | Self-reflective questions | Post-game | Questionnaires (open or closed questions); interviews; debriefings |
Knowledge measurements | Pre- and post-game | Questionnaires (open or closed questions); interviews; concept maps | |
Observed knowledge acquisition | In-game | Observations (structured or unstructured, in-situ or from recordings); Analysing data logged during gameplay | |
Normative learning | Self-reflective questions | Post- and post-post game | Interviews |
Shifts in opinion measurements | Pre- and post-game | Questionnaires (closed questions); perspective mapping | |
Relational learning | Self-reflective questions | Post- and post-post game | Questionnaires (open or closed question); interviews; debriefings |
Group dynamics analysis | In-game | Questionnaires (closed questions) & interaction analysis and social network analysis |
© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Den Haan, R.-J.; Van der Voort, M.C. On Evaluating Social Learning Outcomes of Serious Games to Collaboratively Address Sustainability Problems: A Literature Review. Sustainability 2018, 10, 4529. https://doi.org/10.3390/su10124529
Den Haan R-J, Van der Voort MC. On Evaluating Social Learning Outcomes of Serious Games to Collaboratively Address Sustainability Problems: A Literature Review. Sustainability. 2018; 10(12):4529. https://doi.org/10.3390/su10124529
Chicago/Turabian StyleDen Haan, Robert-Jan, and Mascha C. Van der Voort. 2018. "On Evaluating Social Learning Outcomes of Serious Games to Collaboratively Address Sustainability Problems: A Literature Review" Sustainability 10, no. 12: 4529. https://doi.org/10.3390/su10124529