Real-Time Evaluation of City–University Partnerships for Sustainability and Resilience
2. Materials and Methods
3.1. Indicators and Measures: What to Evaluate and Why
- Foundation—Measures CUP participants’ perceptions of interest, competency, and capacity related to project undertaking. Seeks to understand feelings towards own organization and partner organization.
- Actions—Assesses perceived ability of all partners to plan and implement project-related change interventions in a co-created and co-managed way.
- Impact—Evaluates the perceived achievement of project goals and identification of opportunities for future work.
- Interpersonal Context—Measures participant perceptions of collaborative history between institutions, interest to engage, demonstrated motivation to engage, and mutual understanding of need. Seeks to understand perceptions of both own organization and partner organization.
- Empowering Supports—Assesses perceived and/or actual formalization of a partnership, mechanisms of the partnership, and resources committed on all sides of the partnership.
3.2. Actors: Who Evaluates and Who Is Evaluated
3.3. Tools: How to Collect Data and Disseminate Results
3.4. Timing: When and How Often to Evaluate
3.5. Knowledge Integration: Using Evaluative Results in Real-Time
3.6. Implementation: Quick Guide to the FAIICES Evaluation Scheme
- Define your city–university partnership—Who is involved, what are your goals, why do you want to undertake this collaborative work?
- Choose an evaluator—Determine whether you want to collaboratively conduct the evaluation, or if you want to identify a specific person or people on your team or entity outside your team to undertake the evaluator role.
- Pick your evaluation participants—Choose at least one central figure from each partner institution to participate in the evaluation. These people should understand both the relational and outcomes-oriented sides of the partnership. They will be the subjects of data collection.
- Determine data collection methods—Decide whether open-ended surveys, Likert scales, informal interviews, or focus group sessions will be best for your participants (and feel free to get creative or adjust over time). Develop questions and prompts to explore participants’ perceptions of project foundation, actions, and impact as well as the partnership’s interpersonal context and empowering supports (see Section 3.1). Example open-ended informal interview questions and guidelines and example open-ended and Likert style survey questions are available in Appendix A.
- Conduct evaluation—Choose an appropriate time to conduct your evaluation, usually just before or after a key event or milestone (see Section 3.4). Get survey/interview responses from your key informants on all sides of the partnership.
- Analyze and compile data—Data analysis techniques will vary depending on the data collection methods used. Therefore, either quantitatively, qualitatively, and/or subjectively compile data to show institution-specific and combined responses for each FAIICES category; depict in a visual format if possible (see Figure 4).
- Disseminate and discuss—Soon after results have been compiled, schedule a time to collectively examine results. At minimum, the people who participated in the evaluation should be present, but this can also be expanded to include the larger CUP team. As a group, (typically led by the evaluator) go through the results, question them, add context, change or reinforce the findings.
- Integrate results into CUP administration—Have the management team think about any opportunities, challenges, or interesting findings that were exposed by the analysis. Question whether these findings indicate that a change in CUP typology, strategy, or goals is needed. Pay specific attention to places where modifications could lead to a better partnership trajectory, or tangible impacts. Finally, decide if and how to respond to these findings, and adjust CUP practices accordingly.
- Repeat FAIICES process—Follow the same instructions at the next appropriate evaluation time; you can then also explore how results change over time for deeper understanding of CUP evolution.
Conflicts of Interest
- Please describe the approach you are using for your collaborative project. Has your approach changed?
- Where would you like to be a year from now? Why? What do you need to get there?
- What are the impacts you envision from your project? From your partnership?
- Who do you work with at the city/university?
- What challenges are you currently facing? What opportunities do you see?
- What have you learned from using the real-time evaluation tool so far? What has been most helpful or hurtful and why?
- What is your relationship to this project?
- What is the goal of this project?
- What are the primary actions being taken to support these goals?
- State the primary individuals and organizations involved in this project. Who are the leads?
- Please describe where you are currently at within the project timeline (i.e., phase 1 of a 3 phase project, or month 6 out of a yearlong project).
- Will this project have permanent sustainability impacts that endure after the project has been completed? Please explain.
- At the university, are there a variety of academic positions (including students, researchers, and faculty) that are interested in the topic of this project? Please explain.
- At the city are there a variety of staff interested in the topic of this project? Please explain.
- At the university, how would you describe the level of understanding of the project topic? Do they have the skills and abilities needed to complete this project?
- At the city, how would you describe the level of understanding of the project topic? Do they have the skills and abilities needed to complete this project?
- Does the city have all of the resources (time, money, personnel, etc.) needed to undertake this project? Please explain.
- Does the university have all of the resources (time, money, personnel, etc.) needed to undertake this project? Please explain.
- Does the university have the ability to engage students in this work and/or provide them with related research opportunities? Please explain.
- Does the university have experience working as a convener (i.e., bringing together multiple stakeholders)? Please explain.
- Please describe the level of trust between the city and university regarding this project.
- Please describe the level of communication between the city and university regarding this project.
- Please describe the level of commitment to this project. Are both sides of the partnership fully dedicated?
- Have the roles and responsibilities regarding project scoping and management been well defined, agreed upon, and co-created by both sides of the partnership? Please explain.
- Have the roles and responsibilities regarding fundraising and communications been well defined, agreed upon, and co-created by both sides of the partnership? Please explain.
- Have the roles and responsibilities regarding scheduling, meeting, and planning been well defined, agreed upon, and co-created by both sides of the partnership? Please explain.
- A reference document that memorializes the partnership has been created and agreed upon by both sides of the partnership.
- Before this project began, what actions had been taken by the city to work towards the topic of this project? i.e., City council announced that they would make a climate action plan.
- Since this project began, what actions have been taken by the city to work towards the goal of this project? i.e., City officers have attended 2 workshops to start visioning the climate action process.
- Before this project began, what actions had been taken by the university to work towards the topic of this project? i.e., multiple publications on climate mitigation strategies has been produced.
- Since this project began, what actions have been taken by the university to work towards the goal of this project? i.e., University hired students to coordinate and facilitate climate action planning workshops.
- Is the partnership structure being used to co-develop and design project activities? Please explain.
- Based on your own personal understanding and assessment of the project, do you feel that the goals of this project have been achieved? Please explain.
- Do you envision future projects that build off this project and can utilize this partnership? Please explain.
- Do both sides have a desire to be partners with each other? Please explain.
- What drives the participation in the partnership? What do the partners hope to gain from partnering?
- Do both sides of the partnership have enough motivation to enable dedication to the partnership? Please explain.
- Are both sides of the partnership willing to do what it takes to actively engage in the partnership? Please explain.
- Please rate your satisfaction with the level of motivation to partner and willingness to engage in partnership:
- Have you and your partner completed projects together in the past? Please explain.
- Were you satisfied with the outcomes of the past projects and your experience with the partner? Please explain.
- Are both sides of the partnership committing resources (time, money, personnel, etc.) to the development of the partnership itself? Please explain.
- Have roles and responsibilities in the partnership been outlined and agreed upon? Please explain.
- Are there documents that specifically state the goals and/or purpose of the partnership? Please explain.
- Would you describe both sides of the partnership as feeling empowered and valued in the partnership? Please explain.
- Do the partners have an understanding of each others needs? Please explain.
- Do the partners have an understanding of each others mission and priorities? Please explain.
- Does the partnership influence the internal strategies at both organizations? Please explain.
- Have the partners aligned their missions, in the context of the partnership? Please explain.
- Please rate your satisfaction with the sustainability impacts this project aims to produce:
- Please rate your satisfaction with the overall amount of interest in the topic of this project:
- Please rate your satisfaction with the level of capacity for this project:
- Please rate your satisfaction with the level of co-management for this project:
- Please rate your satisfaction with the actions that have been taken by this project:
- Please rate your current satisfaction with the outcomes and impacts that have been achieved by this project:
- Overall, rate your current level of satisfaction with the progress and functioning of the project:
- Please rate your satisfaction with the history of collaboration with your partner:
- Please rate your level of satisfaction with the resources that have been committed to the partnership:
- Please rate your satisfaction with the level of mutual understanding in the partnership:
- Overall, rate your current level of satisfaction with the progress and functioning of the partnership:
- Please rate your level of satisfaction with the structure of the partnership overall:
- United Nations. Sustainable Development Goals; Sustainable Development Knowledge Platform. 2020. Available online: https://sustainabledevelopment.un.org/?menu=1300 (accessed on 14 November 2019).
- C40. 2020. Available online: https://www.c40.org/why_cities (accessed on 29 January 2020).
- Acuto, M.; Parnell, S.; Seto, K.C. Building a global urban science. Nat. Sustain. 2018, 1, 2–4. [Google Scholar] [CrossRef]
- Nauslar, N.J.; Abatzoglou, J.T.; Marsh, P.T. The 2017 North Bay and Southern California Fires: A Case Study. Fire 2018, 1, 18. [Google Scholar] [CrossRef][Green Version]
- Nolan, R.; Boer, M.M.; Collins, L.; de Dios, V.R.; Clarke, H.; Jenkins, M.; Kenny, B.; Bradstock, R.A. Causes and consequences of eastern Australia’s 2019–20 season of mega-fires. Glob. Chang. Biol. 2020, 26, 1039–1041. [Google Scholar] [CrossRef] [PubMed][Green Version]
- Pickrell, J. Will Australia’s Forests Bounce Back after Devastating Fires? Science News. 11 February 2020. Available online: https://www.sciencenews.org/article/australia-forest-ecosystem-bounce-back-after-devastating-fires (accessed on 13 February 2020).
- Wolfram, M.; Borgström, S.; Farrelly, M. Urban transformative capacity: From concept to practice. Ambio 2019, 48, 437–448. [Google Scholar] [CrossRef] [PubMed][Green Version]
- Loorbach, D. Transition Management for Sustainable Development: A Prescriptive, Complexity-Based Governance Framework. Governance 2010, 23, 161–183. [Google Scholar] [CrossRef]
- Olsson, P.; Galaz, V.; Boonstra, W.J. Sustainability transformations: A resilience perspective. Ecol. Soc. 2014, 19. [Google Scholar] [CrossRef][Green Version]
- Keeler, L.W.; Beaudoin, F.; Wiek, A.; John, B.; Lerner, A.M.; Beecroft, R.; Tamm, K.; Seebacher, A.; Lang, D.J.; Kay, B.; et al. Building actor-centric transformative capacity through city-university partnerships. Ambio 2018, 48, 529–538. [Google Scholar] [CrossRef]
- Polk, M. Transdisciplinary co-production: Designing and testing a transdisciplinary research framework for societal problem solving. Futures 2015, 65, 110–122. [Google Scholar] [CrossRef]
- Lozano, R. Collaboration as a pathway for sustainability. Sustain. Dev. 2007, 15, 370–381. [Google Scholar] [CrossRef]
- Caughman, L. Collaboration and Evaluation in Urban Sustainability and Resilience Transformations: The Keys to a Just Transition? Ph.D. Thesis, Portland State University, Portland, OR, USA, 2020. [Google Scholar] [CrossRef]
- Ramaswami, A.; Bettencourt, L.; Clarens, A.; Das, S.; Fitzgerald, G.; Irwin, E.; Pataki, D.; Pincetl, S.; Seto, K.; Waddell, S. Sustainable Urban Systems: Articulating a Long-Term Convergence Research Agenda; National Science Foundation: Alexandria, VA, USA, 2018. [Google Scholar]
- Trencher, G.; Bai, X.; Evans, J.; McCormick, K.; Yarime, M. University partnerships for co-designing and co-producing urban sustainability. Glob. Environ. Chang. 2014, 28, 153–165. [Google Scholar] [CrossRef]
- Keeler, L.W.; Beaudoin, F.D.; Lerner, A.M.; John, B.; Beecroft, R.; Tamm, K.; Wiek, A.; Lang, D.J. Transferring Sustainability Solutions across Contexts through City—University Partnerships. Sustainability 2018, 10, 2966. [Google Scholar] [CrossRef][Green Version]
- Birch, E.L.; Perry, D.C.; Taylor, H.L., Jr. Universities as Anchor Institutions. J. High. Ed Outreach Engag. 2013, 17, 7–15. [Google Scholar]
- Harris, M.; Holley, K. Universities as Anchor Institutions: Economic and Social Potential for Urban Development. In Higher Education: Handbook of Theory and Research; Paulsen, M., Ed.; Springer: Cham, Switzerland, 2016; Volume 31. [Google Scholar] [CrossRef]
- Durst, S.; Poutanen, P. Success factors of innovation ecosystems—Initial insights from a literature review. In Proceedings of the CO-CREATE 2013: The Boundary-Crossing Conference on Co-Design in Innovation, Helsinki, Finland, 16–19 June 2013; pp. 27–38. [Google Scholar]
- Mattila, P.; Eisenbart, B.; Kocsis, A.; Ranscombe, C.; Tuulos, T. Transformation is a game we can’t play alone: Diversity and co-creation as key to thriving innovation ecosystems. In Research into Design for a Connected World; Springer: Singapore, 2019; Volume 135, pp. 151–162. [Google Scholar] [CrossRef]
- Keeler, L.W.; Wiek, A.; Lang, D.J.; Yokohari, M.; van Breda, J.; Olsson, L.; Ness, B.; Morató, J.; Segalàs, J.; Martens, P.; et al. Utilizing international networks for accelerating research and learning in transformational sustainability science. Sustain. Sci. 2016, 11, 749–762. [Google Scholar] [CrossRef][Green Version]
- EPIC-N. Who’s in the Network. 2020. Available online: https://www.epicn.org/whos-in-the-network/ (accessed on 13 February 2020).
- Margerum, R.D. A Typology of Collaboration Efforts in Environmental Management. Environ. Manag. 2008, 41, 487–500. [Google Scholar] [CrossRef]
- Teitel, L. School/Uníversíty Collaboratíon: The Power of Transformative Partnerships. Child. Educ. 2008, 85, 75–80. [Google Scholar] [CrossRef]
- Butcher, J.; Bezzina, M.; Moran, W. Transformational Partnerships: A New Agenda for Higher Education. Altern. High. Educ. 2010, 36, 29–40. [Google Scholar] [CrossRef]
- Kula-Semos, M. Seeking Transformative Partnerships: Schools, University and the Practicum in Papua New Guinea. Ph.D. Thesis, James Cook University, Douglas, QLD, Australia, 2009. Available online: https://researchonline.jcu.edu.au/15463/ (accessed on 29 January 2020).
- Swartz, A.L.; Triscari, J.S. A Model of Transformative Collaboration. Adult Educ. Q. 2010, 61, 324–340. [Google Scholar] [CrossRef]
- Warren, J.D. The North Carolina Policy Collaboratory: A Novel and Transformative Partnership for Decision-Relevant Science; Fall Meeting; Abstract #PA43A-08; American Geophysical Union: Washington, DC, USA, 2018; Available online: http://adsabs.harvard.edu/abs/2018AGUFMPA43A..08W (accessed on 29 January 2020).
- Salimova, T.; Vatolkina, N.; Makolov, V. Strategic Partnership: Potential for Ensuring the University Sustainable Development. Qual. Innov. Prosper. 2014, 18, 107–124. [Google Scholar] [CrossRef][Green Version]
- Luederitz, C.; Schäpke, N.; Wiek, A.; Lang, D.J.; Bergmann, M.; Bos, J.J.; Burch, S.; Davies, A.; Evans, J.; König, A.; et al. Learning through evaluation—A tentative evaluative scheme for sustainability transition experiments. J. Clean. Prod. 2017, 169, 61–76. [Google Scholar] [CrossRef][Green Version]
- Reed, M.S.; Fraser, E.D.; Dougill, A.J. An adaptive learning process for developing and applying sustainability indicators with local communities. Ecol. Econ. 2006, 59, 406–418. [Google Scholar] [CrossRef]
- Greenwald, H.P.; Zukoski, A.P. Assessing Collaboration. Am. J. Eval. 2018, 39, 322–335. [Google Scholar] [CrossRef]
- Marek, L.I.; Brock, D.J.P.; Savla, J. Evaluating Collaboration for Effectiveness. Am. J. Eval. 2014, 36, 67–85. [Google Scholar] [CrossRef]
- Woodland, R.H.; Hutton, M.S. Evaluating Organizational Collaborations. Am. J. Evaluation 2012, 33, 366–383. [Google Scholar] [CrossRef]
- Kitzinger, J. Qualitative Research: Introducing focus groups. BMJ 1995, 311, 299–302. [Google Scholar] [CrossRef] [PubMed][Green Version]
- Massey, A.; Wallace, W. Focus groups as a knowledge elicitation technique: An exploratory study. IEEE Trans. Knowl. Data Eng. 1991, 3, 193–200. [Google Scholar] [CrossRef]
- Tremblay, M.C.; Hevner, A.R.; Berndt, D.J. The Use of Focus Groups in Design Science Research. In Design Research in Information Systems; Springer: Boston, MA, USA, 2010; pp. 121–143. [Google Scholar] [CrossRef]
- Rodríguez-Campos, L. Advances in collaborative evaluation. Eval. Program Plan 2012, 35, 523–528. [Google Scholar] [CrossRef] [PubMed]
- Whitmore, E.E. Understanding and Practicing Participatory Evaluation. New Dir. Eval. 1998, 80, 1–104. Available online: https://eric.ed.gov/?id=EJ580835 (accessed on 29 January 2020). [CrossRef]
- Brown, L.D.; Feinberg, M.E.; Greenberg, M.T. Measuring coalition functioning: Refining constructs through factor analysis. Health Educ. Behav. 2011, 39, 486–497. [Google Scholar] [CrossRef]
- Seitanidi, M.M.; Koufopoulos, D.N.; Palmer, P. Partnership Formation for Change: Indicators for Transformative Potential in Cross Sector Social Partnerships. J. Bus. Ethics 2010, 94, 139–161. [Google Scholar] [CrossRef]
- Ferraris, A.; Santoro, G.; Papa, A. The cities of the future: Hybrid alliances for open innovation projects. Futures 2018, 103, 51–60. [Google Scholar] [CrossRef]
- Faehnle, M.; Tyrväinen, L. A framework for evaluating and designing collaborative planning. Land Use Policy 2013, 34, 332–341. [Google Scholar] [CrossRef]
- Guyadeen, D.; Seasons, M. Evaluation Theory and Practice: Comparing Program Evaluation and Evaluation in Planning. J. Plan. Educ. Res. 2016, 38, 98–110. [Google Scholar] [CrossRef]
- Plummer, R.; Armitage, D. A resilience-based framework for evaluating adaptive co-management: Linking ecology, economics and society in a complex world. Ecol. Econ. 2007, 61, 62–74. [Google Scholar] [CrossRef]
|Routine||Transactional, consultant-based or fee-for-service; Loose exchanges; One-sided deliverables; Limited joint efforts; One-off projects; Often based on individual (rather than institutional) relationships||Straight-forward problems|
i.e., community organization manager working with a researcher to develop maps of street trees
|Strategic||Loose partnership; Shared vision and desire to co-create; Often externally focused||Complicated problems|
i.e., Business organization and city department collaborating on the development and implementation of initiatives for minority-owned businesses
|Transformative||Formalized partnerships; Deep cross-institutional learning and mission alignment; Focused on internal and external systemic change||Complex problems|
i.e., City, university, and community coming together to develop holistic equity-focused climate adaptation plan
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Caughman, L.; Withycombe Keeler, L.; Beaudoin, F. Real-Time Evaluation of City–University Partnerships for Sustainability and Resilience. Sustainability 2020, 12, 8796. https://doi.org/10.3390/su12218796
Caughman L, Withycombe Keeler L, Beaudoin F. Real-Time Evaluation of City–University Partnerships for Sustainability and Resilience. Sustainability. 2020; 12(21):8796. https://doi.org/10.3390/su12218796Chicago/Turabian Style
Caughman, Liliana, Lauren Withycombe Keeler, and Fletcher Beaudoin. 2020. "Real-Time Evaluation of City–University Partnerships for Sustainability and Resilience" Sustainability 12, no. 21: 8796. https://doi.org/10.3390/su12218796