Comparing Societal Impact Planning and Evaluation Approaches across Four Urban Living Labs (in Food-Energy-Water Systems)

Abstract

Achieving societal impact, as opposed to academic impact, is a growing area of focus for the research community globally. Central to this changing mission is the focus on multiple interconnected complex systems and the need for research that is not just interdisciplinary, but also transdisciplinary and grounded in stakeholder co-production. This document compares multiple approaches to impact planning and evaluation across four newly formed urban living labs in Sao Paolo (Brazil), Western Cape (South Africa), Bristol (UK) and Rotterdam (Netherlands), each of which sought to address societal issues linked to the food-energy-water nexus. A comparison matrix and a disaggregated impact table are derived from a comprehensive review of key definitions. These new tools were completed by each ULL alongside a post hoc pathway to impact statements. Comparisons are presented and discussed, the strengths and weaknesses of this approach are considered and opportunities for improvement in societal impact planning and evaluation are provided. Our main findings include the importance of establishing clear shared definitions while accepting plural understandings, the need to acknowledge resource as a critical factor in impact delivery and the headline need for far greater focus in this area from both funders and research groups.

1. Introduction

Achieving societal impact, as opposed to academic impact [1], is a growing area of focus for the research community, especially in areas of global challenge such as the climate, biodiversity and inequality crises. Central to this changing mission is the need for research that engages with complex “systems of systems” [2,3] and is both interdisciplinary and grounded in stakeholder co-production [4].

The Sustainable Urbanisation Global Initiative (SUGI)/Food-Water-Energy Nexus is a call that was jointly established by the Belmont Forum and the Joint Programming Initiative (JPI) Urban Europe [5]. The cooperation was established “in order to bring together the fragmented research and expertise across the globe to find innovative new solutions to the Food-Water-Energy Nexus challenge”; the overarching goal is “to increase…quality of life…to move to action…”. The call states a range of areas where it is seeking a societal impact, many of which contribute to the various concepts of urban health, including Ecological Public Health and Planetary Health, the United Nations 2030 Agenda for Sustainable Development and the associated Sustainable Development Goals (SDG)—

Table 1. Societal impacts sought by the SUGI programme.

Security and social justice—increasing regional stress on urban food-water-energy Reciprocal and dynamic processes of urbanisation Physical movements of populations Build-up of city territories Transformation of economic structures Extension of suburban sprawl and re-urbanisation Material and energy use Land-use transformations

Resource-intensive behaviours and consumption Impacts on ecosystem services Changes driving social/cultural inequities Balancing trade-offs Resilience Climate compatibility Sustainable Consumption and Production (SCP) Comprehensive spatial perspectives Multi-level governance strategies Synergies between sectors (“silos”)

.

This paper describes the approaches to impact planning and evaluation taken by the WASTE FEW-ULL (pronounced “fuel”) consortium [6], which involved four urban living labs in Bristol (UK), Rotterdam (Netherlands), Western Cape (South Africa) and Sao Paolo (Brazil). Societal impact was therefore a core focus in the conceptualisation, with aims “to map and substantially reduce waste (resource inefficiencies) in the urban food-energy-water (FEW) nexus in city-regions across three continents: Europe, Africa and South America” and with the group committed to “co-creating and testing newly integrated problem-solving methods appropriate to each context” [6]. Given the focus on urban system inefficiencies with regards to socio-ecological outcomes, these align squarely with SDGs 3 (Good Health and Wellbeing) and 11 (Sustainable Cities and Communities).

2. Materials and Methods

The project had five primary objectives and six work packages (

Table 2. Objectives, work packages, impact objectives and impact deliverables of the WASTE FEW ULL project.

Primary Objectives	Work Packages
Map resource flows Identify critical dysfunctional linear pathways Agree on the response most appropriate to the local context Model the market and non-market economic value of each intervention Engage with decision-makers to close each loop	Stakeholder analysis and engagement Systems mapping Valuation of externalities Impact planning and monitoring Creation of a replicable “toolkit” Dissemination through mixed media
Impact Objectives	Impact Deliverables
Agree on likely pathway(s) to impact in each context Provide evidence of impact	An impact plan A report on economic valuation impacts An impact report

), with impact planning and monitoring intended to be integrated throughout the project design and operationalisation. There were two objectives on impact, specifically (1) to agree on likely pathways to impact in each context and (2) to provide evidence of impact. Alongside those objectives on impact, there were three associated deliverables: (a) an impact plan, (b) a report on economic valuation impacts and (c) an impact report. While the impact work was nominally coordinated centrally, each ULL was responsible for their own impact strategies, given the unique and very different contexts and focus areas of each. A theory of change (or multiple theories of change) was not defined in the project development stage because co-production with ULL stakeholders was central to the whole approach, with each drawing on a range of participatory approaches to research co-production [7].

The original proposal set out to use Fast Track Impact [8], a framework developed by a research team in the UK for impact planning that uses simple, open source templates for (i) stakeholder analysis; (ii) impact planning; (iii) impact monitoring. The templates and guidance prompt researchers to think about (a) who they are engaging with and why them in particular (e.g., their level of interest in the research and their perceived influence over the outcomes); (b) what specific activities they are going to undertake, why those activities in particular and to what end; (c) how impact will be measured. The interest in this approach varied across the ULLs. It was championed by the Bristol ULL, where there was a wide range of research and practitioner partners involved, a strong desire to see societal impact within the award period and an appreciation for a more nuanced stakeholder analysis approach. It was supported too by the Cape Town ULL, where it was seen to be applicable due to the clear focus there on job creation and material changes to the Water Hub infrastructure. The Rotterdam ULL had some reservations, given their primary academic focus and limitations in terms of resources (one PhD) and directly involved stakeholders. There was some confusion at first from the Sao Paolo ULL, who queried the fundamental focus on societal impact, especially during award period, though broadly, the approaches were very similar in practice.

Each ULL therefore decided to develop their own approach to impact planning and monitoring, given that it would enable them to respond to their own unique contexts. An impact group was then formed, made up primarily of the group research and ULL leads and chaired by the lead author of this paper.

A literature review of definitions (Section 2.1 and Section 2.2 below) resulted from an investigation into how to undertake effective impact evaluation, which resulted in turn in an impact comparison matrix (Supplementary Material File S1). This considered both the characteristics of each ULLs in some detail, as well as the range of societal impacts that we were interested in. This first tool provided us with a useful framework for thinking about how we might compare across each ULL. Each ULL’s summary reports, which were requested to focus explicitly on societal impact, provided and consolidated essential context within which to consider pathways to impact.

Finally, a separate table was developed focusing solely on the definitions of impact, and this was used to bring together all those impacts reported from each ULL and broken down by their category. A summary table is given below (Last Table), and a disaggregated breakdown of impacts is given in Supplementary Material File S2.

2.1. Defining “Societal Impact”

In the UK, the latest formal definition of “impact” given by the recently combined research councils, UKRI, is “the demonstrable contribution that excellent research makes to society and the economy” [1]. This is then split into two parts: (i) academic impact; (ii) economic and societal impacts. Though arguably well understood, this differentiation is worth underlining, as it links to different priorities between practitioner-stakeholders and academic researchers, i.e., universities prioritise and incentivise high-quality academic outputs, while stakeholders engaged in the research want to see changes in the real world, and the two are not necessarily mutually inclusive [4]. There are many types of societal impacts, and interpretation depends on life experience and interest. For example, for those interested in inclusivity, it might mean opportunities for education and training; for those working in public administration, it may mean changes in policy and decisions made; for those working in public health, it may mean the containment of infectious disease or community health and wellbeing; for those in inequality and sustainability, it may mean the sustained functioning of interrelated socio-ecological systems. Defining what is meant by societal impact is therefore an important first step.

Reed (2016) described five different types of impact [8], without differentiating clearly between real world practical and academic, which nonetheless illustrate the breadth of potential impact (

Table 3. Types of impacts and real world changes (Reed, 2016).

Types of Impact	Real World Changes
Instrumental: e.g., actual changes in policy or practice Conceptual: e.g., broad new understanding/awareness-raising Capacity-building: e.g., training of students or professionals, CPD Attitudinal or cultural: e.g., increased willingness to engage in new collaborations Enduring connectivity: e.g., follow-on interactions such as joint proposals	Understanding and awareness Attitudes Economy Environment Health and wellbeing Policy Other forms of decision-making and behaviour change Culture Other social Capacity or preparedness

). He also described a variety of practical quantifiable changes in the real world that one may be seeking to influence.

The US Agency for Healthcare Research and Quality (1999) set out four levels of impact in their Hierarchy of Research Impact (Figure 1) [9], a report that was written up into a paper by the lead authors, Stryer et al. (2000) [10]. Though conceptualised within the context of healthcare provision specifically, it is arguably universally applicable, as it helpfully simplifies what otherwise becomes complex networks of interaction between research activity and the actors and agents involved in making use of the evidence in the “real world”. As an alternative from the world of public policy, the London School of Economics (2011, Figure 1.1, p.15) provided their own graphic visualising the routes from primary impacts to changes in societal outcomes [11]. In a previous paper, we contrasted an idealised UK Research Council pathway to impact visualisation with our interpretation of misguided pathways to impact in urban development research [12] (Black et al., 2019, Figure 3).

By using Reed (2016) as a starting point, then comparing UKRI’s (2020) and Stryer et al.’s (1999) definitions (

Table 4. Comparing definitions of impact suggests possible misunderstandings between what constitutes academic and societal impact.

Reed (2016) [8]		UKRI (2020) [1]	Stryer et al. (1999) [10]
Categories	Examples Impacts	Academic/Societal	Hierarchy of Research Impact
Instrumental	“Actual” changes in policy or practice	Societal	Policies/Practice
Conceptual	Broad new understanding/awareness-raising	Academic/societal	N/A
Capacity-building	Training of students or professionals, CPD	Academic/societal	Further Research
Attitudinal or cultural	Increased willingness in general to engage in new collaborations	Academic/societal	Further Research
Attitudinal or cultural	Willingness to change (e.g., agriculture practices/drinking or eating habits)	Academic/societal	Further Research
Enduring connectivity	Follow-on interactions (e.g., joint proposals, reciprocal visits, workshops, relationships)	Academic	Further Research

), it is tempting to infer (a) that there is considerable variation in understandings, with (b) an apparently dominant focus on further research rather than policies and practice, albeit with (c) the significant caveat that there is considerable uncertainty as to what constitutes societal impact and what constitutes academic impact or both.

Even with the detailed definitions above being clarified, understanding may still be missing. For example, two researchers may both be thinking of “instrumental” impact, but one may be thinking of changes to public sector policies and another of changes to private sector practice (i.e., sector-specific impact) or executive-level changes, as opposed to operational-level practice (i.e., level of organisational hierarchy). There may also be a lack of clarity around evidence of impact. One may be thinking of the provision of an email from a stakeholder stating their intention (possibly slipping in to “warm words” and platitudes), while another may be thinking of actual ratified changes to local government policy and procedures (i.e., “tangibility” of impact). There is the question, too, of ongoing monitoring to understand the full extent of potential impact, which needs to be developed into the legacy aspects of the programme.

Clearly defining the types of societal impacts, including at a granular level, therefore seems crucial. That said, there is at the same time an emerging recognition from the transdisciplinary literature, which suggests that plural understandings are in fact an inevitable (and essential) prerequisite to innovation and scientific progress [13]. The process of creating shared understandings relies on tension for impetus, the resolution of which creates the value [14]. A quote often attributed to Albert Einstein is “everything should be made as simple as possible, but not simpler”. An alternative that perhaps captures the balance that is required between clarity of communication on the one hand and openness to change and innovation on the other might be “definitions should be made as clear as is appropriate, but not clearer”.

2.2. Defining “Urban Living Labs”

A core mechanism promoted within the SUGI call was the concept of the “urban living lab” (ULL), a term that the WASTE FEW ULL Consortium adopted for their four case study areas. Given that there are considerable variations between ULLs—e.g., level of resource, focus areas, types of activity, missions—the nature and characterisation of the ULL has profound implications for the understanding of the potential (or not) for achieving societal outcomes. As a growing phenomenon, much has been written on ULLs in recent years [15,16,17], including a number of different characterisations [18,19,20,21]. The focus of this paper is not to review, critique or examine that literature. However, in order to compare the four WASTE FEW ULLs in terms of their approach to the planning and evaluation of societal impact, a starting point is how to characterise each ULL.

The original call documentation (Belmont/JPI Europe, 2018) is based ostensibly on the learnings presented in Marvin et al. (2018) and the “Urban Living Lab Handbook” briefing document, both of which were funded as part of the Urban-Europe Research programme, Governance of Urban Sustainability Transitions (GUST) [5,16,22]. These texts set out a rich and detailed characterisation of the many and varied ULLs that have been emerging over recent years, identifying main observed defining aspects, including:

• Characteristics: geography, experimentation, participation, leadership, evaluation.
• Types: strategic, civic, grassroots/organic.
• Domains: community/local sustainability, ICT, mobility and energy, social interaction, spatial/area development.
• Role of municipalities: promoter, enabler, partner.
• Stages: design, operation, evaluation.

Adjustment is needed to avoid duplication and to focus more specifically on societal impact. As noted above, there are a number of questions related specifically or directly to societal impact, for example, in the different working stages of the ULL, namely: the design stage (kind of impacts or benefits), operation stage (deliverables, expected/intended outcomes) and evaluation stage (purpose/main question of evaluation, data needs). A further characterisation relating to capacities—or “potential to effect change”, which is “shaped by their disposition”—is included below [22]:

• Trial: testing of products, technologies or processes under “real world conditions”.
• Enclave: innovation under protected conditions (through “spatial segregation”).
• Demonstration: demonstrating what “the urban could resemble”.
• Platform: “making coincidental” different interests, fostering new “urban configurations”.

This focus on typology of purpose is useful, as it starts to focus on our interest in societal outcomes. This characterisation was developed further by von Wirth et al. (2019) into six strategies and practices, which are closely linked to the three areas of impacts set out above in Reed (2016) and are as follows [8,21]:

• Embedding: transformative place-making, activating network partners (e.g., “attitudinal/cultural” impact).
• Scaling: replication of lab structure, education and training (e.g., “capacity-building”).
• Translating: stimulating entrepreneurial growth (e.g., “instrumental”); narratives of impact (“attitudinal/cultural”).

In addition, although Menny et al. (2019) focus predominantly on assessing the role of users in co-creation using traditional characterisations of stakeholder inclusion, they also focus on “outcomes” (and their relation to sustainability challenges) and the “transformative potential of urban living labs for sustainability” [19]. They also use four other categories—co-creation, consultation, information, non-participation—rather than the usual five or the original eight from the “Ladder of Citizen Participation” [23]. Menny et al. (2019) use the following three themes and qualitative assessments of change [19]:

• Change processes: catalysed change, little change.
• Sustainable innovations: advanced at a small scale; promising at a large scale; advanced at a large scale; feasible at a small scale.
• Societal challenges: addressed; not addressed; addressed selectively; not fully addressed.

By combining these characterisations and reducing them to their main component parts focusing on societal outcomes, we can arrive at a typological breakdown that might suit our purposes for comparing impact approaches across the four ULLs.

3. Results

The following sections provide post hoc reports of each ULL’s approach to impact planning and evaluation. The specific impacts from each ULL are set out in Supplementary Material File S2 and disaggregated into different areas of impact, and evidence for each is shown. A shortened version of the recorded impacts is presented below (Last Table) alongside a summary description.

3.1. Sao Paulo, Brazil—In Natura Lab

The main objectives of the Sao Paolo ULL initiative were:

• To map and model existing and future conditions of five food production systems within the urban green belt in the State of São Paulo, Brazil, at the Atlantic Rainforest border.
• To propose, from a policy foresight approach, the selection and adoption of technological and organisational innovation, looking for the transition to a more sustainable scenario.

To address these objectives, two procedures were developed:

• Developing, validating and starting to disseminate a decision-making tool, drawing on a set of newly developed FEW governance indicators (Water Footprint, Eutrophication, Carbon Footprint, Rural Property Net Profit, Social Development Land Index, Ecosystems Services).
• Forming a network of stakeholders and shareholders (20 institutional actors from public, private and third sector) for the decision-making tool development and diffusion phases, using systems dynamics modelling and the Delphi foresight technique—Figure 2 [24,25,26].

Climate change and the governance of natural common-pool resources of the FEW nexus—the “Commons” [27]—are the central sustainability challenges. The approach for this initiative draws on public-choice economic theory [28], in which sustainable ecosystem services depend on collective future design and implementation. It is a long-term challenge, only possible (in this case) with the participation of more than 20 institutional stakeholders: three municipal governments, NGOs, the Environmental Public Ministry, Sao Paulo State Environmental Regulatory Agency, Sao Paulo State Environment Secretary, five agricultural cooperatives at the border of the Atlantic Rain Forest, the São Paulo State Forest Institute, Community-Supported Agriculture Networks and the Sao Paulo State Secretary of Agriculture (the AgroSP platform of agri-trade, in which the SP in Natura tool will be made available to user communities). The main operational challenges anticipated were not the development of the tool but the market barriers, regulatory demands, intellectual property and appropriability dynamics, market displacement and the policy and decision-making processes.

The Lab is in the aggregation phase of Delphi responses (using Promethée method for decision-tree results analysis). It has produced a validated, multi-criteria tool of 13 indicators on sustainable agriculture dynamics—SPIN (Sustainability Policy Innovation Network)—which is intended to be used as an accountability tool in sustainable production practices (Figure 3). It is an actor’s network “glue”, supporting decision-making based on these indicators. The tool appears to make transition processes more efficient, creating accountability and a higher level of expected future collective design.

We accept that commons governance is an essentially innovative process, from the perspective of demands for technological, organisational, financial and human behaviour innovations. It encompasses changes at the local, regional, national and international levels, almost in a process of entropy and reorganisation, so that the latter is more sustainable than the initial conditions. However, the promotion of this profound process of change towards sustainability is not linear: all levels of human organisation must be seen as a complex system, and its multiple institutions must be reoriented so that an evolutionary process of transition towards sustainability is taken forward. The SPIN tool is for planning and implementing this transition process, enabling the creation of learning communities who can maximise sustainability across FEW systems. A fuller description can be found in the associated paper [29].

3.2. Cape Town, South Africa—Water Hub

The initial reason for establishing the Water Hub ULL in Franschhoek, South Africa, was to determine the performance of nature-based solutions (NbS) in the treatment of contaminated surface water. The ULL is situated in a region that experiences intervals of drought and the worst meteorological drought on record between 2016 and 2018 that led to the situation in which Cape Town almost ran out of water, the so-called “Day Zero” scenario. The ULL site was also chosen because it is in an area of abject urban poverty where nearly 6000 people live in a densely populated, underserviced settlement and one of the main sources of contamination of a nearby river. The impact of this ULL study is set in three broad contextual issues, with the grand challenges being climate-constrained water resources; socio-economic hardship, vulnerability and poverty; and the pollution of downstream surface water that compromises ecological systems and services.

The initial study on understanding water treatment processes consumed at least 60% of the time devoted to this project. The challenge focused on understanding how constructed wetlands, which in this case were in the form of large biofiltration cells, could be managed without the addition of chemicals or non-renewable forms of energy. The water quality was sampled regularly, which eventually resulted in the development of continuous monitoring using sensors, loggers and the internet to monitor water and soil conditions transmitted via the internet. This research phase was focused on the assessment of water quality to determine the risk of using treated water for irrigating edible crops and potential impact on the soil as a necessity point of departure before engaging local community growers in the initiative. An overview of the research journey is captured in the image below (Figure 4).

Four impacts were featured in the first phase of the study in NbS treatment of water:

• Scientific evidence that NbS processes were capable of treating water to comply with South African and World Health Organisation (WHO) irrigation standards.
• Reduction in nutrient concentration to levels that could be safely applied to food gardens and soil and used to enhance growth without harming crops or soil.
• Emerging evidence of the entrapment/containment of drugs, pharmaceutical compounds and pathogens within the NbS biofiltration cells.
• Development of the sensors and loggers using the Internet-of-Things to monitor water quality and soil in near-real time.

Circularity of resources involves the community in removing invasive non-indigenous vegetation and transforming the cuttings into biochar, which acts as water filtration media and a denitrification resource. Once clogging occurs over time in the capillaries of the biochar, the media will be removed and inserted into the soil, thereby raising the carbon content, improving infiltration and retaining nutrients at the root level.

Laboratory tests were conducted to assess the health and safety of vegetables and soil. Results from three successive harvests of bulbous and leafy plants indicated that the quality of the vegetables complied with measurable agricultural standards and compared favourably with the quality of vegetables from local supermarkets. After two years of irrigation with treated water and the regeneration of soil from active composting, no changes in soil moisture and salinity were observed. Experimental plots that were given biochar showed an increase in carbon content, small improvements in the percentage of water and mass of the crop and an increase in Mg, Fe and K compared to local supermarket stock.

All energy requirements on site are generated from renewable energy sources. An assessment of energy demand required by the site manager, security system, water pumps and site maintenance showed that that initial investment of approximately €2000 would yield a return on the investment within 4 years and cumulative savings of €33,000 over 25 years.

The project is in the early stage of reaching a broader social impact, which is focused on job creation, capacity building and securing livelihoods. Prior to the commencement of the ULL project, a site manager, who previously lived in the informal settlement, was appointed to assist with rudimentary maintenance on the site. Through training and support, he has become a skilled research assistant from the informal settlement. Three young women from the informal settlement, who were 3rd/4th year agricultural studies students from a local technical university, were engaged to expand the experimental gardens while gaining valuable skills in urban agriculture. The ULL has an “open for business” policy and invitation to community representatives and entrepreneurs. Towards the end of the project, one business involved a small community group to participate in making biochar. The community struggled to find productive ways of managing land that was heavily invested by invasive trees. Subsequently, about 400 m³ of biochar has been produced and used as a water filtration media. Once the biochar becomes clogged, following a risk analysis, it will be removed and placed in the soil as a carbon source. This is another example of a circular system in practice. It has created employment for 10 people, with additional benefits of cleaning contaminated water and returning carbon and captured nutrients to the soil to improve the yield and quality of crops.

The social impact is relatively small, but the project has made progress in developing a vision for a social enterprise, which has the potential to involve 15 trained growers by 2023 and an opportunity to sell garden produce to the formal town nearby and, in return, to create a conduit for receiving food and organic waste that will be used to increase composting processes and for regenerating soil at the ULL.

3.3. Bristol, UK—Nutrient Recovery

As with the WASTE FEW ULL project as a whole, the Bristol ULL started with a very broad mandate: to reduce inefficiencies in Bristol’s food, energy and water systems. A core aim of the WASTE FEW ULL project was to identify and put a value on the external costs (i.e., the “inefficiencies”) in Bristol’s food-energy-water systems. In the Bristol ULL, the valuation had two main components: a non-market, socio-environmental valuation and a market macro-valuation.

Co-production was also critical to the Bristol ULL, which was focused primarily on three core partners: Wessex Water, the city (and region’s) waste/water utility company; the Centre for Sustainable Energy, a leading charity on energy and social justice; and the Bristol Food Network, a local Community Interest Company expert in local food production. The project started by considering the whole city/city region, as well as its food (including global supply chains), energy (national, regional and local) and water (catchment area)—Figure 5.

Three workshops were run in the first year of the project—one in each sector area (food, energy, water). They were intentionally small and were targeted towards individuals identified by the stakeholder partners. They provided rich data sets that expanded substantially during discussions with core partners, identifying and agreeing with the wider group’s key areas of focus, barriers and potential opportunities.

The direction of the Bristol ULL was therefore co-created with the stakeholder partners, and they provided the main direction of the research throughout the programme. Over time, the focus narrowed to the food system and residential food waste specifically, which brought in two new main stakeholders central to this challenge: Bristol Waste, the Council-owned waste management company, and Resource Futures, a national NGO based in Bristol. These partners helped to:

• Map out in some detail the key challenges (nutrient waste, contamination, associated energy wastage) and “problem spaces” (the actors, decision-makers, processes) and the different needs and characteristics of the different FEW sectors (e.g., food sector far more complex than larger water and energy infrastructure);
• Focus on food waste and nutrient recovery—Figure 6;
• Target and secure buy-in from key stakeholders (including two organisations that became key partners—the city’s waste management company and a national waste consultancy);
• Validate the optimal focus areas for the economic valuation and hypothesised pathways to impact.

The group then narrowed in further to the main areas of focus selected, all of which relate to nutrients flows through the Avonmouth plant and formed the basis of the work as follows:

• Residential food waste: The average UK family throws away £60 of food each month [30,31,32], which for Bristol would equate to c.£36,000,000/yr [33].
• Plastics in commercial food waste: At the Avonmouth plant alone, c.1000 tonnes of plastic contaminate 33,000 tonnes of Bristol’s commercial food waste each year.
• Phosphorous: This is a global food security issue and potentially more of a threat than climate change yet largely unknown outside of agriculture/environmental sectors; the UK is 100% reliant on foreign imports of phosphorus.

As the work developed, the valuation focused primarily on the food waste challenge, with some consideration for plastics. Feedback from stakeholders suggests that the socio-environmental valuation provides useful evidence to support the waste hierarchy: specifically, to substantially reduce the main socio-environmental impacts of food waste (GHG emissions, eutrophication and air pollution) from the food system, the City needs to focus its efforts on reducing upstream consumption linked to food waste [34]. These findings were used by the Going for Gold team, which enabled them to secure Gold Sustainable Food City status for Bristol (Table 2).

Understanding these socio-environmental impacts alone provided only a partial picture of the costs and benefits, however. As the x-curve illustrates [35], there are inevitable winners and losers in any transition. Further workshops were run to consider the valuation work on its own and to help develop the approach. A main goal of these was to show clearly that Bristol’s One City Plan was only considering the scaling up of sustainable practice; it was not considering the impact on industries that would need to contract.

Using the same scenarios on food waste reduction as the non-market valuation, the newly integrated macro-valuation enabled stakeholders to start to consider some of the major barriers to change, most notably, the potential risk to the food industry from a targeted reduction in consumption: i.e., if food consumption needs to be reduced in order to reduce food waste, there is a clear knock-on effect on jobs. However, the modelling also showed that it does not have to lead to job losses: it depends on how the money saved on food waste is redistributed in the system (i.e., there may be associated increases in, say, travel or clothing with the money saved on food).

A follow-up presentation and meetings with the City’s Food Waste Action Group [36], which includes the main actors from the public and third sectors, suggests there is significant interest in this approach, and discussions about how to develop it further are ongoing. These economic findings are presented in a separate paper (submitted at time of writing) alongside the systems dynamics modelling work in an Urban Europe blog article [37,38]. All other impacts are listed in Table 2.

3.4. Rotterdam, Netherlands—Circular Economy Start-Ups

The Rotterdam ULL focused its activity on BlueCity Lab (BCL), which had been established over four years prior to the start of the WASTE FEW ULL project. BCL is located at Maasboulevard in the city of Rotterdam in the Netherlands. BCL was found in 2015 by a group of sustainability and circular economy start-up pioneers and activists at an iconic site in a former indoor swimming pool. This site had been left empty for several years, and these pioneers set up their “meanwhile use” operations there as a means of experimenting with circular economy material flows and alternative, sustainable business models. The guiding idea of experimenting in BCL has been to become a platform for the circular economy. Today, BCL has become an acknowledged platform for innovative approaches to enhanced material flows by providing spaces for co-creation and coworking, hosting (waste free) events and meetings as well as offering lectures, hack parties and spaces for experimentation, including wet and dry laboratories [21]. A philanthropist investor bought the building, offering it for experimental use via the “bottom-up” BCL managing process. Embedded in this multi-functional context, entrepreneurs were invited to develop business cases with closed-loop material flows. Academia and knowledge institutions from Rotterdam and beyond are regularly involved in lab activities in different roles. For example, students at the University of Applied Sciences Rotterdam have regular classes on site and interact as learning partners with start-ups and lab people regularly.

Within the WASTE FEW ULL project, one focus of co-creating impact was the identification of actual impact-oriented actor coalitions closely connected to BCL. The niche experiments at BCL in the field of circular food services (e.g., catering) were examined with a focus on how it interacted with incumbent actors from the food service industry and with larger public catering clients. By conducting a thorough stakeholder analysis and a series of in-person interviews, data were collected that allowed us to map the processes of diffusion, the inherent power dynamics and the connecting mechanisms between circular niche and current regime actors [39].

The findings from research in the context of BCL show various mechanisms through which niche and regime actors connect in order to create impact together, including actions taken to facilitate the diffusion of circular catering. These findings have implications for the future impact planning of co-creative work at sites such as ULL, as the transdisciplinary research offers empirical insights into how to increase and scale cleaner production practices towards a circular economy through circular start-ups. The findings were summarised into 15 observed principles for connecting and integrating niche innovations to incumbent practices, though the immediate social and economic impacts of these transdisciplinary research findings cannot be defined clearly.

Rotterdam hosted the WASTE FEW ULL International Consortium gathering on-site at BCL, involving managers and entrepreneurs from the ULL. In addition to the academic impact of the Rotterdam ULL scholars in the form of three academic papers, the outputs included the development and validation of a new tool for policy and practice: the “Circular Decision-Making Tree” (CDMT) [40]. A lack of decision-support was identified as an important reason why the transition towards a circular economy is not proceeding as quickly as hoped in the management processes of companies and investors and among policymakers. A practice-oriented framework that supports decision-making could thus be a key enabler of such transition dynamics. The tool was co-designed to enable a decision-making logic that is rooted in circularity in order to help navigate trade-offs and make decisions while considering the quality of innovation circularity and its respective diffusion potential.

The CDMT synthesised insights from existing frameworks and integrated these with novel perspectives of transition theory and the circular economy. The framework was then developed, tested and refined together with diverse stakeholders in a series of international workshops. These workshops reflected the internal logics and applicability of the framework with a focus on usability in four application contexts (Netherlands, Brazil, UK and South Africa). Feedback from a total of n = 50 stakeholders from policy, practice and academia was integrated, and the application (as well as societal impact) potentials were critically discussed. The final version of the CDMT tool was then published and further disseminated through keynotes and conference presentations [40].

3.5. Main Impacts Recorded by each ULL

Table 5. Reduced table listing reported impacts for each ULL. See Supplementary Material File S2 for disaggregated impacts and associated evidence.

Western Cape	Sao Paulo	Rotterdam	Bristol
Instrumental change to “practice“: Provision of small-scale, appropriate tech. food growing/water purification “infrastructure”: operated by local community members, it uses integrated nature-based processes for the treatment of water to a quality to support food gardens and renewable energy to power the pumps that are required to abstract and distribute water around the site. Research “practice”: The site offers opportunities for multi-disciplinary exchanges among researchers and attracts an interdisciplinary interest who are raising new research questions to address the grand challenges on the site and in the context. Improved understanding of FEW in practice. The concept of the nexus and the integration of each component has shifted from the conceptual into a field scale application. New questions have arisen about the risk of using treated water which has resulted in studies on potential health risk in food (vegetables and fish), water and soil. As a direct result, studies have shifted to identify contaminants of emerging concern in surface and ground water. Researchers and students have learnt new methods and the application of science to a large-scale project. Community members have learnt about food gardening and how to approach this activity in a challenging socio-economic setting. Limited engagement with local authorities who could be in a stronger position to affect a transition towards a more just society and environment. Engagement with local community workers and leadership in the informal settlements. No change in attitudes to food and diet as yet, other than conversations about food and diet. Strengthened relationships with research institutes local and international; building on knowledge gained to write new proposals with partners; ongoing activity with leaders and members of the informal settlement. New research funding achieved through this FEW project Growing interest from community members with more meetings and discussions about future developments at the ULL. Interest from other local authorities in upscaling the work to their municipalities. Funding support from provincial government to develop the ULL further.	The SPIN tool—Sustainability Policy Innovation Network—is being adopted as a decision-making support for policy-makers, producers and academics involved in the Atlantic Rainforest food production areas (e.g., “Conexão Mata Atlântica“, the Interamerican Bank of Development, and “Brasil Orgâncio”, an agroecological business association, among others). A Delphi cycle of foresight is applied in order to have policy strategies for the transition. The SPIN tool is enabling increased understanding and awareness raising in the food production community. The diffusion of the SPIN tool is the capacity building path. It is not possible to measure such an increase in capacity. However, the transition community in the Sao Paolo region has yet an astonishing willingness to participate. SPIN will be a more rational support for the network enforcement, once it has a sustainable governance. SPIN will be used as a guide for a more oriented and systematic ensemble of practices in São Paulo State around agricultural practices/policies. As a result an extensive network of food production agents has been formed. Details of ongoing interactions yet to be determined.	Established new understanding of “Waste Resource Paradox” phenomenon, which may hinder further acceleration towards circular economy. Awareness was raised among BlueCity Lab management and start-up entrepreneurs to sufficiently record and monitor material flow and resource flows in the building as well as into and out from the building. Several workshops with civil servants and consultants on Circular Economy in the Netherlands aimed at capacity building based on the novel Circular Decision-Making Tree heuristic. The findings from the PhD were presented to fellow PhD students and industry professionals in during further disseminations events. Existing networks between DRIFT researchers and policy makers, civil servants as well as industry and civil society representatives were strengthened. Not yet decided, to be on the agenda for the final project meeting in March.	Helped secure successful bid for “Going for Gold“ City Consortium: The WASTE FEW ULL Project contributed “most definitely” to Bristol’s successful Going for Gold bid, which saw Bristol become only the second city in the UK to be awarded “Gold Sustainable Food City” [36,41]. The WASTE FEW ULL project was “extremely helpful in bringing key actors together, digging deeper on specific areas of expertise, connecting practitioners with academics, creating a space where we can dig in together, and testing One City Plan targets”. (Bristol Food Network) The bid refers specifically to the micro-valuation generating through WASTE FEW ULL, which was “very helpful” and “gave confidence” (in supporting so clearly the waste hierarchy) Augmented waste campaign: Bristol Waste will be using the micro-valuation work in its campaign this year to underpin waste hierarchy (2022). Can “help evidence need for waste hierarchy when talking to local authorities” “people often good at recycling, but they miss the broader picture” (e.g., reduction before recycling) (Resource Futures) “Localised, factual data” (Bristol Waste) New framework being used: “The Schumacher Institute’s new ‘Hexalemma’ framework—see ‘Conceptual’ section below—is now being used to look at energy transition involving a major UK Distribution Systems Operator (DSO).” (Schumacher Institute) Macro-economic valuation flags key strategic barriers to achieving targets: The newly integrated conceptual macro-economic/scenario model has raised considerable awareness of potential macro-economic implications of food waste targets across key members of the Food Waste Action Group, especially with regards who might lose out and the implications for the One City Plan. It “helps shape directions of solutions…walk you through that complexity…see ‘shapes’” (Resource Futures). It is “fascinating…makes you think about things differently…needs a bigger conversation” “this is the best data we’ve got” (Bristol Waste); “very helpful for us taking a look at the One City Plan goals” (Bristol Food Network). New conceptual framework developed: “A new framework emerged from the project, developed/led by the Schumacher Institute, to explore dilemmas in policymaking—the ‘Hexalemma’—with six components considered in decision-making for complex systems: resilience, sustainability, efficiency, reliability, safety, and equity.” (Schumacher Institute) Non-market valuation provides important evidence The non-market valuation has provided Resource Futures, Bristol Waste and Bristol Food Network with important evidence underpinning the waste hierarchy, which they will use in ongoing campaigns. Macro valuation flags key strategic barriers to achieving targets See above on “Conceptual” Key partner supported “The project added to the capacity, credibility and profile of The Schumacher Institute. As an independent think tank its reputation has been greatly enhanced by participating. It demonstrated the value of systems approaches to complex interactions of infrastructure and people.” (Schumacher Institute) Enabled and forged much closer links between key actors: The project forged close ties among the ULL participants, which means that future work by the Schumacher Institute in the city of Bristol on plans and initiatives involving the FEW nexus will be smoother and more likely to succeed than otherwise. (Schumacher Institute) “Very useful in bringing key actors together, which wouldn’t have happened otherwise…I don’t think you can ever underestimate the power of getting people together…it’s been empowering with key people on the Food Waste Action Group” (Bristol Food Network) A dedicated presentation and workshop with the City’s Food Waste Action Group, with dialogue ongoing and invitations to explore future collaborations.

sets out the reported impacts, aggregated under each ULL (see Supplementary Material File S2 for impacts disaggregated into key impact headings and with available supporting evidence). By combining them like this, it is easier to see the overall impact of each ULL, how the impacts vary and what similarities there are.

For example, in the Western Cape, the focus was on the provision of small-scale but tangible changes to local infrastructure (which we bundle under the heading “instrumental impacts”), around which less tangible (though not necessarily less important) impacts are reported (“conceptual”, “capacity-building”, “attitudinal or cultural”). In Sao Paolo, the focus was on the development and application of the “SPIN” tool, which was reported to have been adopted already by significant agencies in the region and, as with the Western Cape, conceptual, capacity-building, attitudinal and cultural and enduring connectivity impacts were reported as a result. The Rotterdam case study was quite different in that it was primarily focused on a PhD project, so there were relatively fewer resources, and there were also issues of engagement due in part to COVID-19. While a “Circular Decision-Making Tree” was developed as a core output of the PhD, which was tested amongst researchers and stakeholders internationally, the primary impacts from the PhD, as may be expected, were less instrumental and more focused on awareness-raising and network development. In the case of the Bristol ULL, there were a number of different instrumental impacts, mainly resulting from the two economic valuations. The early, socio-environmental valuation was reported to have helped the Bristol Food Network (and Bristol City Council) secure Bristol as a “Gold Sustainable Food City” and was also reported to have been useful in Bristol Waste and Resource Futures’ campaigns. A further instrumental impact was the application of the Schumacher Institute’s “Hexalemma” framework examining energy transition with a Distributor Systems Operator (a digitally enabled, “bottom-up” energy distribution system). It also increased conceptual understandings through the exploration of five critical challenge areas, which, in addition to efficiency, are resilience, reliability, safety and equity. The macro-economic valuation also enabled conceptual impacts by flagging potential systemic barriers to the City’s Zero Food Waste 2049 target. As with the other ULLs, it was also reported that the ULL as a whole enabled and forged much closer links between key actors (“enduring connectivity”).

A notable difference between the four ULLs was the evidence available to support the reported impacts claimed. This is not to suggest that the impacts are not legitimate in each case but rather that the methods for meaningful impact evaluation—whether through lack of time or insufficient foresight and planning—were unavailable. Photographs provided clear evidence of infrastructural (“instrumental”) changes in South Africa, and both there and in Rotterdam, there was clear evidence of capacity-building impacts via Master’s and PhD projects. In South Africa, too, the project was reported to have led to the winning of further research funding. In Bristol, testimonials were sought and provided from key stakeholders by way of evidence of impact and appear to strongly support the value of the work undertaken, but there are clear potential conflicts of interest, given that the stakeholders providing the testimonials were part of the ULL project, and they are being reported by the co-lead(s) of the ULL (and the lead author of this paper).

4. Discussion

The funding body expressed interest in a wide range of societal impacts, which could all broadly contribute to Ecological Public Health or Planetary Health. Our approach to impact planning and evaluation was described only in broad terms in the application. Central concepts and principles were presented at the first and second annual international conferences in Bath (UK) and Rotterdam (Netherlands); however, we were clear up to a point in the application, given that we stated explicitly that we were interested in “reducing inefficiencies in cities’ food-energy-water systems” (and wider socio-ecological functioning). We also made clear that we were aiming to “develop and test non-conventional economic valuation approaches that might enable consideration of social impacts” of urban projects and plans on community health and wellbeing and to “integrate them with a range of systems approaches to avoid narrow cause and effect”.

There were marked differences across the different ULLs in a range of areas, including resources, stakeholders, focus areas and methods used. For example, the Bristol ULL’s researcher resource was spread thinly across a wide range of disciplines, which on the one hand enabled a broader investigation of knowledge domains but on the other hand made coordination and linear efficiency more of a challenge. The Rotterdam ULL resource supporting an individual PhD allowed for a clear focus on development in a specific research area but limited practitioner engagement. The Western Cape and the Sao Paolo ULLs were arguably more similar in terms of available resources, and both were working closely with external stakeholders in the field, but the methods were quite different and the former focused on the development and testing of a field site and its equipment, while the latter on a decision and indicator tool. These differences presented both challenges and opportunities for the consortium: on the one hand, it was not possible to co-create an overarching impact strategy for the whole consortium, and on the other hand, there were useful lessons that each could learn from the other. For example, the Bristol ULL used as a key concept in its newly integrated macro-economic scenario development the “x-curve” developed by DRIFT in the Netherlands ULL [35]. Additionally, the Water Hub in Cape Town found the focus and structure of the Bristol ULL societal impact approach useful to help better frame the practitioner- and citizen-led work that it engages with.

In developing impact outputs together, including this paper, it has been possible to think more deeply and at a greater level of detail on the range of areas relevant to impact planning and evaluation via the ULL approach, including the consideration of issues such as resourcing, approaches, stakeholder engagement and evaluation. For example, a perceived strength of Reed (2016) is the key focus on stakeholder analysis and, specifically, the associated attention to the level of interest and influence of each stakeholder identified [8]. While the value of this kind of subjective or arbitrary assessment is reasonably contested, we suggest it remains a strength, given that in our experience, it is a consideration that often appears to be missing in academic stakeholder engagement [42]. An arguable weakness in impact planning more generally is the lack of consideration for who exactly is undertaking the stakeholder analysis and impact planning itself, given that this substantially influences who is engaged with and the information that flows from that exercise [12,43].

Reviewing the resource available to each ULL, their different contexts and the impacts reported appears to bear out to some extent the earlier observation that “further research” [9]—the awareness-raising, conceptual, attitudinal or cultural—is a far more common impact than changes to real world policy or practice. This is arguably to be expected given the nature of the scientific enterprise [44], yet it also raises the question, given the increasing focus on the need for science to achieve societal impact, as to how this is to best be achieved [5].

A clear observation from our comparison appears to be that future (ULL) projects may benefit from strengthening the focus on impact evaluation, and it being built in earlier to the process may ensure sufficient time and resource.

Though not discussed in much detail in the group, there are clear issues of potential bias, for example in the self-reporting and evaluation. Part of the planning would likely also therefore benefit from the consideration of the issues of potential bias and conflicts of interest and how those might best be resolved, drawing on, for example, Cochrane’s best practices in bias identification [45]. Reed (2021) provides a helpful typology of research impact evaluation designs [46] and discusses in some detail issues of bias, while the journal, Research Evaluation, offers a treasure trove of learning from which to develop this evaluation [47]. In some ways, therefore, it is perhaps surprising that there was not more emphasis on this area, particularly given the relative novelty of the ULL approach and the clear focus on societal impacts.

5. Conclusions

Achieving a real world (societal or planetary health) impact is a growing area of focus for the research community, and its planning and evaluation within new research approaches, and especially ULLs, appears in need of scrutiny. This document starts by exploring what we mean by “impact” and “urban living lab”, revealing a plurality of definitions that cause considerable confusion but may also allow space for innovation. It then sets out the approaches to impact planning and evaluation used by the four ULLs in the WASTE FEW ULL consortium in such a way that comparison can be made, including differences in capacity and context. The reported impacts varied considerably across the four ULLs, and understandably, there was a tendency towards further research rather than actual changes in policy and practice. There were also clear gaps in the evidence of impacts, alongside issues of potential bias, suggesting that a more proactive approach to impact planning and evaluation should be made going forward. That said, a societal impact does appear to have been achieved in a number of ways and contexts, and those involved have a greater shared understanding of the language of impact and the ULLs and have further refined their approaches as a result.