Complex Projects and Transition-Driven Evaluation: The Case of the easyRights European Project
Abstract
:1. Introduction
2. Complex Projects Evaluation: A Learning Activity
2.1. Triple-Loop Learning: An Imperative for Change
2.2. Quadruple Helix Ecosystems: An Effective Reference for Complex Projects’ Environments
2.3. Evaluation in Action: A Learning Driven Framework
3. The easyRights Project: An Action Field for Operationalizing Triple-Loop Learning in Complex Projects
3.1. A Project Overview
3.2. The easyRights Quadruple Helix Ecosystems
4. Targeting Learning in the easyRights Project
4.1. Inter-Organizational Learning as Value-Production Chain
4.2. The easyRights Methodological Framework and Its “How” Perspective in Triple-Loop Learning
4.2.1. Service Design as Value-Driven Learning Approach (Section 3.2 is Part of the Deliverable 5.1 of the easyRights Project)
4.2.2. Hackathons as a Learning Approach to Include Newcomers in Service Ecosystems
4.2.3. Interactions through Learning Dialogues
5. Applying the Evaluation Framework to the easyRights Project
5.1. Evaluation for Collaborative and Critical Reflections
5.2. Key Learning Drivers in the easyRights Project
6. Toward the easyRights Evaluation Toolbox
6.1. Mapping Evaluation Processes Toward a Shared Vision on Learning Drives
- Links between the different evaluation tasks in the course of the project and thus the need to bring together different actors and expertise for different evaluation activities;
- The key learning drivers (related to the triplet what/how/why), which are addressed as part of different evaluation activities and in varying combinations, thus influencing the choice of specific evaluation tools. These decisions have to be made in a common process and under consideration of the (project) objectives and context;
- The concrete time frame in relation to the project activities and milestones to be evaluated in the pre-peri and post-sequence;
- Evaluation results and related discussions, feedback, and learning processes, which activate dynamics and exchange between the three ecosystems (easyRights ecosystem, hackathon ecosystems, and service ecosystems) and will influence and shape further project activities.
6.2. The Definition of Context-Embedded Indicators and Evaluation Tools
6.3. Integration of Results Triggering Learning Mechanism
- Through content, impulses, reflections on (interim) project results, and methods contribute directly to the design and content of communication and interaction within the service ecosystems and thus have an activating effect;
- Evaluation results and the associated discussions influence the design and methodological implementation in the easyRights project;
- Reflection and learning processes of the different actors of the Quadruple Helix community on the project progress and impact can be documented, and associated learning processes can be traced.
7. Conclusions
Author Contributions
Funding
Acknowledgments
Conflicts of Interest
Appendix A
Appendix B
References
- Rowe, G.; Marsh, R.; Frewer, L.J. Evaluation of a Deliberative Conference. Sci. Technol. Hum. Values 2004, 29, 88–121. [Google Scholar] [CrossRef]
- Bellamy, J.A.; Walker, D.H.; McDonald, G.T.; Syme, G.J. A Systems Approach to the Evaluation of Natural Resource Management Initiatives. J. Environ. Manag. 2001, 63, 407–423. [Google Scholar] [CrossRef]
- Blackstock, K.L.; Kelly, G.J.; Horsey, B.L. Developing and Applying a Framework to Evaluate Participatory Research for Sustainability. Ecol. Econ. 2007, 60, 726–742. [Google Scholar] [CrossRef]
- Lang, D.J.; Wiek, A.; Bergmann, M.; Stauffacher, M.; Martens, P.; Moll, P.; Swilling, M.; Thomas, C.J. Transdisciplinary Research in Sustainability Science: Practice, Principles, and Challenges. Sustain Sci 2012, 7, 25–43. [Google Scholar] [CrossRef]
- Reed, M.S.; Ferré, M.; Martin-Ortega, J.; Blanche, R.; Lawford-Rolfe, R.; Dallimer, M.; Holden, J. Evaluating Impact from Research: A Methodological Framework. Res. Policy X 2020, X, 100012. [Google Scholar] [CrossRef]
- Bateson, G. Steps to an Ecology of Mind: Collected Essays in Anthropology, Psychiatry, Evolution, and Epistemology; Paladin, Granada: London, UK, 1973. [Google Scholar]
- Gupta, J. Climate Change Governance: History, Future, and Triple-Loop Learning? WIREs Clim. Chang. 2016, 7, 192–210. [Google Scholar] [CrossRef]
- Weick, K.E. Mental Models of High Reliability Systems. Ind. Crisis Q. 1989, 3, 127–142. [Google Scholar] [CrossRef]
- Loorbach, D.; Rotmans, J. The Practice of Transition Management: Examples and Lessons from Four Distinct Cases. Futures 2010, 42, 237–246. [Google Scholar] [CrossRef]
- Raven, R.P.J.M.; Heiskanen, E.; Lovio, R.; Hodson, M.; Brohmann, B. The Contribution of Local Experiments and Negotiation Processes to Field-Level Learning in Emerging (Niche) Technologies: Meta-Analysis of 27 New Energy Projects in Europe. Bull. Sci. Technol. Soc. 2008, 28, 464–477. [Google Scholar] [CrossRef] [Green Version]
- van Mierlo, B.; Beers, P.J. Understanding and Governing Learning in Sustainability Transitions: A Review. Environ. Innov. Soc. Transit. 2020, 34, 255–269. [Google Scholar] [CrossRef]
- Stacey, R.D. Strategic Management and Organisational Dynamics: The Challenge of Complexity to Ways of Thinking about Organisations; Pearson Education: London, UK, 2007. [Google Scholar]
- Kahane, A. Solving Tough Problems: An Open Way of Talking, Listening, and Creating New Realities; Berrett-Koehler Publishers: San Francisco, CA, USA, 2004. [Google Scholar]
- Schön, D.A. Organizational Learning; Addison-Wesley: Reading, MA, USA, 1978. [Google Scholar]
- Sinek, S. Start with Why: How Great Leaders Inspire Everyone to Take Action; Penguin Publishing Group: New York, NY, USA, 2009. [Google Scholar]
- Englebart, D.C. Improving Our Ability to Improve: A Call for Investment in a New Future; IBM Almaden Research Center: San Jose, CA, USA, 2003. [Google Scholar]
- Johannessen, Å.; Wamsler, C. What Does Resilience Mean for Urban Water Services? Ecol. Soc. 2017, 22. [Google Scholar] [CrossRef] [Green Version]
- Gomez, R.; Pather, S. ICT Evaluation: Are We Asking the Right Questions? Electron. J. Inf. Syst. Dev. Ctries. 2012, 50, 1–14. [Google Scholar] [CrossRef]
- Carayannis, E.G.; Campbell, D.F.J. Triple Helix, Quadruple Helix and Quintuple Helix and How Do Knowledge, Innovation and the Environment Relate To Each Other? A Proposed Framework for a Trans-Disciplinary Analysis of Sustainable Development and Social Ecology. Int. J. Soc. Ecol. Sustain. Dev. 2010, 1, 41–69. [Google Scholar] [CrossRef]
- De Liddo, A.; Concilio, G. Making Decision in Open Communities: Collective Actions in the Public Realm. Group Decis. Negot. 2017, 26, 847–856. [Google Scholar] [CrossRef] [Green Version]
- Concilio, G.; Moro, A. Trading Zones and Public Spaces Transformations the Case of Piazza Leonardo in Milan. Group Decis. Negot. 2017, 26, 933–951. [Google Scholar] [CrossRef]
- Von Hippel, E. Open Source Software Projects as User Innovation Networks. In Perspectives on free and open source software; Lessig, L., Shirky, C., Cusumano, M., Eds.; MIT Press: Cambridge, MA, USA, 2005; pp. 267–278. [Google Scholar]
- Almirall, E.; Lee, M.; Wareham, J. Mapping Living Labs in the Landscape of Innovation Methodologies. Technol. Innov. Manag. Rev. 2012, 2012, 12–18. [Google Scholar] [CrossRef]
- Wenger, E. Communities of Practice: Learning, Meaning, and Identity; Cambridge University Press: Cambridge, UK, 1999. [Google Scholar]
- Wenger, E. Communities of Practice and Social Learning Systems: The Career of a Concept. In Social Learning Systems and Communities of Practice; Blackmore, C., Ed.; Springer: London, UK, 2010; pp. 179–198. [Google Scholar] [CrossRef]
- Wholey, J.S. Formative and Summative Evaluation: Related Issues in Performance Measurement. Eval. Pract. 1996, 17, 145–149. [Google Scholar] [CrossRef]
- Tessmer, M. Planning and Conducting Formative Evaluations; Routledge: Abingdon, UK, 2013. [Google Scholar] [CrossRef]
- Carayannis, E.G.; Campbell, D.F.J. “Mode 3” and “Quadruple Helix”: Toward a 21st Century Fractal Innovation Ecosystem. Int. J. Technol. Manag. 2009, 46, 201–234. [Google Scholar] [CrossRef] [Green Version]
- Lundvall, B.-Å. National Systems of Innovation: Towards a Theory of Innovation and Interactive Learning. In The Learning Economy and the Economics of Hope; Anthem Press: London, UK, 2016. [Google Scholar]
- Kuhlmann, S. Future Governance of Innovation Policy in Europe—Three Scenarios. Res. Policy 2001, 30, 953–976. [Google Scholar] [CrossRef]
- Kaiser, R.; Prange, H. The Reconfiguration of National Innovation Systems—the Example of German Biotechnology. Res. Policy 2004, 33, 395–408. [Google Scholar] [CrossRef]
- Carayannis, E.G.; Ziemnowicz, C. (Eds.) Rediscovering Schumpeter: Creative Destruction Evolving into “Mode 3”; Palgrave Macmillan: London, UK, 2007. [Google Scholar]
- Gibbons, M.; Limoges, C.; Nowotny, H.; Schwartzman, S.; Scott, P.; Trow, M. The New Production of Knowledge: The Dynamics of Science and Research in Contemporary Societies; Sage: London, UK, 1994. [Google Scholar]
- Mohr, J.J.; Sengupta, S. Managing the Paradox Ofinter-firm Learning: The Role of Governance Mechanisms. J. Bus. Ind. Mark. 2002, 17, 282–301. [Google Scholar] [CrossRef]
- Slater, S.F.; Narver, J.C. Market-Oriented Is More than Being Customer-Led. Strateg. Manag. J. 1999, 20, 1165–1168. [Google Scholar] [CrossRef]
- Paladino, A. Analyzing the Effects of Market and Resource Orientations on Innovative Outcomes in Times of Turbulence *. J. Prod. Innov. Manag. 2008, 25, 577–592. [Google Scholar] [CrossRef]
- Sinkula, J.M.; Baker, W.E.; Noordewier, T. A Framework for Market-Based Organizational Learning: Linking Values, Knowledge, and Behavior. J. Acad. Mark. Sci. 1997, 25, 305. [Google Scholar] [CrossRef]
- Lepak, D.P.; Smith, K.G.; Taylor, M.S. Value Creation and Value Capture: A Multilevel Perspective. AMR 2007, 32, 180–194. [Google Scholar] [CrossRef] [Green Version]
- Manço, A.A. Social Development at the Local Level and Developing Public Services: Defining a Local Policy of Integration for Ethnic Minorities. Int. Sociol. 1996, 11, 79–90. [Google Scholar] [CrossRef]
- Isin, E.F.; Nielsen, G.M. Acts of Citizenship; Zed Books: London, UK, 2008. [Google Scholar]
- Owen, C.L. Design Research: Building the Knowledge Base. Des. Stud. 1998, 19, 9–20. [Google Scholar] [CrossRef]
- Beckman, S.L.; Barry, M. Innovation as a Learning Process: Embedding Design Thinking. Calif. Manag. Rev. 2007, 50, 25–56. [Google Scholar] [CrossRef]
- Kolb, D.A. Experience as the Source of Learning and Development; Pearson Prentice Hall: Upper Saddle River, NJ, USA, 1984. [Google Scholar]
- Rizzo, F.; Deserti, A.; Cobanli, O. Introducing Design Thinking in Social Innovation and in the Public Sector: A Design Based Learning Framework. Eur. Public Soc. Innov. Rev. 2017, 2, 127–143. [Google Scholar] [CrossRef]
- Bason, C. Leading Public Sector Innovation 2E: Co-Creating for a Better Society; Policy Press: Bristol, UK, 2018. [Google Scholar]
- Tõnurist, P.; Kattel, R.; Lember, V. Innovation Labs in the Public Sector: What They Are and What They Do? Public Manag. Rev. 2017, 19, 1455–1479. [Google Scholar] [CrossRef]
- Elsbach, K.D.; Stigliani, I. Design Thinking and Organizational Culture: A Review and Framework for Future Research. J. Manag. 2018, 44, 2274–2306. [Google Scholar] [CrossRef] [Green Version]
- Cohen, W.M.; Levinthal, D.A. Absorptive Capacity: A New Perspective on Learning and Innovation. Adm. Sci. Q. 1990, 35, 128–152. [Google Scholar] [CrossRef]
- Sanders, E.B.-N.; Stappers, P.J. Co-Creation and the New Landscapes of Design. CoDesign 2008, 4, 5–18. [Google Scholar] [CrossRef] [Green Version]
- Senge, P.M. The Fifth Discipline, the Art and Practice of the Learning Organization; Doubleday: New York, NY, USA, 2003. [Google Scholar]
- Checkland, P.; Scholes, J. Soft Systems Methodology in Action Chichester; John Wiley and Sons England: Chichester, UK, 1990. [Google Scholar]
- Barbat, G.; Boigey, P.; Jehan, I. Triple-loop learning: Theoretical framework, methodology andillustration. Proj. Proy. Proj. 2011, 8, 129–141. [Google Scholar]
- McArdle, K.; Coutts, N. Taking Teachers′ Continuous Professional Development (CPD) beyond Reflection: Adding Shared Sense-Making and Collaborative Engagement for Professional Renewal. Stud. Contin. Educ. 2010, 32, 201–215. [Google Scholar] [CrossRef]
- Buchanan, D.; Dawson, P. Discourse and Audience: Organizational Change as Multi-Story Process. J. Manag. Stud. 2007, 44, 669–686. [Google Scholar] [CrossRef]
- van Woerkom, M. The Concept of Critical Reflection and Its Implications for Human Resource Development. Ph.D. Thesis, Twente University, Enschede, The Netherlands, 2003. [Google Scholar]
- van Woerkom, M. Critical Reflection and Related Higher-Level Conceptualizations of Learning: Realistic or Idealistic? Hum. Resour. Dev. Rev. 2008, 7, 3–12. [Google Scholar] [CrossRef]
- van Woerkom, M.; Croon, M. Operationalising Critically Reflective Work Behaviour. Pers. Rev. 2008, 37, 317–331. [Google Scholar] [CrossRef]
- de Groot, E.; van den Berg, B.A.M.; Endedijk, M.D.; Beukelen, P.; Simons, P.R.J. Critically Reflective Work Behaviour Within Autonomous Professionals′ Learning Communities. Vocat. Learn. 2011, 1, 41–62. [Google Scholar] [CrossRef] [Green Version]
- Janis, I.L. Groupthink: Psychological Studies of Policy Decisions and Fiascoes, 2nd ed.; Houghton Mifflin: Boston, MA, USA, 1982. [Google Scholar]
- Klocke, U. How to Improve Decision Making in Small Groups: Effects of Dissent and Training Interventions. Small Group Res. 2007, 38, 437–468. [Google Scholar] [CrossRef]
- Gartmeier, M.; Bauer, J.; Gruber, H.; Heid, H. Negative Knowledge: Understanding Professional Learning and Expertise. Vocat. Learn. 2008, 1, 87–103. [Google Scholar] [CrossRef] [Green Version]
- Ashford, S.J.; Blatt, R.; VandeWalle, D. Reflections on the Looking Glass: A Review of Research on Feedback-Seeking Behavior in Organizations. J. Manag. 2003, 29, 773–799. [Google Scholar] [CrossRef]
- Hattie, J.; Timperley, H. The Power of Feedback. Rev. Educ. Res. 2007, 77, 81–112. [Google Scholar] [CrossRef]
- Kamin, C.; O′Sullivan, P.; Deterding, R.; Younger, M. A Comparison of Critical Thinking in Groups of Third-Year Medical Students in Text, Video, and Virtual PBL Case Modalities. Acad. Med. 2003, 78, 204–211. [Google Scholar] [CrossRef] [PubMed]
- Walker, B.; Salt, D. Resilience Practice: Building Capacity to Absorb Disturbance and Maintain Function; Island Press: Washington, DC, USA, 2012. [Google Scholar]
- Njihia, J.M.; Merali, Y. The Broader Context for ICT4D Projects: A Morphogenetic Analysis. MIS Q. 2013, 37, 881–905. [Google Scholar] [CrossRef]
- Krishna, S.; Walsham, G. Implementing Public Information Systems in Developing Countries: Learning from a Success Story. Inf. Technol. Dev. 2005, 11, 123–140. [Google Scholar] [CrossRef]
- Lu, C.; Tsai, C.-C.; Wu, D. The Role of ICT Infrastructure in Its Application to Classrooms: A Large Scale Survey for Middle and Primary Schools in China. J. Educ. Technol. Soc. 2015, 18, 249–261. [Google Scholar]
- Gichoya, D. Factors Affecting the Successful Implementation of ICT Projects in Government. Electron. J. E Gov. 2005, 3, 175–184. [Google Scholar]
- Heinz, J. Digital Skills and the Influence of Students′ Socio-Economic Background. An Exploratory Study in German Elementary Schools. Ital. J. Sociol. Educ. 2016, 8, 186–212. [Google Scholar] [CrossRef]
- Parmar, V. A Multidisciplinary Approach to ICT Development. Inf. Technol. Int. Dev. 2009, 5, 89–96. [Google Scholar]
- Verdegem, P.; De Marez, L. Rethinking Determinants of ICT Acceptance: Towards an Integrated and Comprehensive Overview. Technovation 2011, 31, 411–423. [Google Scholar] [CrossRef] [Green Version]
- Fraser, E.D.G.; Dougill, A.J.; Mabee, W.E.; Reed, M.; McAlpine, P. Bottom up and Top down: Analysis of Participatory Processes for Sustainability Indicator Identification as a Pathway to Community Empowerment and Sustainable Environmental Management. J. Environ. Manag. 2006, 78, 114–127. [Google Scholar] [CrossRef] [PubMed]
- Khadka, C.; Vacik, H. Comparing a Top-down and Bottom-up Approach in the Identification of Criteria and Indicators for Sustainable Community Forest Management in Nepal. For. Int. J. For. Res. 2012, 85, 145–158. [Google Scholar] [CrossRef] [Green Version]
- Reed, M.S.; Fraser, E.D.G.; Dougill, A.J. An Adaptive Learning Process for Developing and Applying Sustainability Indicators with Local Communities. Ecol. Econ. 2006, 59, 406–418. [Google Scholar] [CrossRef]
- Magee, L.; Scerri, A.; James, P. Measuring Social Sustainability: A Community-Centred Approach. Appl. Res. Qual. Life 2012, 7, 239–261. [Google Scholar] [CrossRef]
- Grunfeld, H. Framework for Evaluating Contributions of ICT to Capabilities, Empowerment and Sustainability in Disadvantaged Communities; IEEE: Chennai, India, 2007; p. 21. [Google Scholar]
- Leech, N.L.; Onwuegbuzie, A.J. A Typology of Mixed Methods Research Designs. Qual. Quant. 2009, 43, 265–275. [Google Scholar] [CrossRef]
- Greene, J.C.; Caracelli, V.J. Advances in Mixed-Method Evaluation: The Challenges and Benefits of Integrating Diverse Paradigms; Jossey-Bass: San Francisco, CA, USA, 1997. [Google Scholar]
- Heeks, R.; Alemayehu, M. Impact Assessment of ICT-for-Development Projects: A Compendium of Approaches; SSRN Scholarly Paper ID 3477380; Social Science Research Network: Rochester, NY, USA, 2009. [Google Scholar] [CrossRef]
- Weber, E.; Khademian, A.M. Managing Collaborative Processes: Common Practices, Uncommon Circumstances. Adm. Soc. 2008, 40, 431–464. [Google Scholar] [CrossRef]
- Hanafizadeh, P.; Hanafizadeh, M.R.; Khodabakhshi, M. Taxonomy of E-Readiness Assessment Measures. Int. J. Inf. Manag. 2009, 29, 189–195. [Google Scholar] [CrossRef]
- Luederitz, C.; Schäpke, N.; Wiek, A.; Lang, D.J.; Bergmann, M.; Bos, J.J.; Burch, S.; Davies, A.; Evans, J.; König, A.; et al. Learning through Evaluation—A Tentative Evaluative Scheme for Sustainability Transition Experiments. J. Clean. Prod. 2017, 169, 61–76. [Google Scholar] [CrossRef] [Green Version]
- Taanman, M. Looking for Transitions. Ph.D. Thesis, Erasmus University Rotterdam, Rotterdam, The Netherlands, 2014. [Google Scholar]
CRWB Dimension | Key Principles |
---|---|
Challenging groupthink | Request contrary ideas and criticize espoused theories within a group when alternative courses of action [59] appear. Consider that the risk of groupthink increases when the group is striving for consensus and unanimity [56,60]. |
Critical opinion sharing | Emphasize on constructive challenges intended to improve and propose alternatives rather than merely criticize [57]. |
Openness about mistakes | Push collective reflections on mistakes to correct false assumptions and to explore alternatives. Stick the reflection around mistakes not only as an individual activity rather as an essential social activity [55]. This is a special form of experiential learning that is highly relevant for professional learning [61]. |
Asking for feedback | Ask for feedback, especially when people work alone or in small practices [62]; earning communities could add opportunities to ask for feedback as a prerequisite for learning to occur [63]. |
Experimentation | It is a broader concept for reflection-in-action [14] and it will probably not take place during the meetings of learning communities but in the daily work setting. Therefore, the approach to action within an experimental perspective is crucial to be discussed when exploring and imagining alternatives [64]. |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Concilio, G.; Karimi, M.; Rössl, L. Complex Projects and Transition-Driven Evaluation: The Case of the easyRights European Project. Sustainability 2021, 13, 2343. https://doi.org/10.3390/su13042343
Concilio G, Karimi M, Rössl L. Complex Projects and Transition-Driven Evaluation: The Case of the easyRights European Project. Sustainability. 2021; 13(4):2343. https://doi.org/10.3390/su13042343
Chicago/Turabian StyleConcilio, Grazia, Maryam Karimi, and Lydia Rössl. 2021. "Complex Projects and Transition-Driven Evaluation: The Case of the easyRights European Project" Sustainability 13, no. 4: 2343. https://doi.org/10.3390/su13042343