Sustainability Transitions and the Contribution of Living Labs: A Framework to Assess Collective Capabilities and Contextual Performance

Abstract

Living labs are understood as collaborative platforms in which actors from research, government and business, and citizens, work together to address complex societal challenges. They are increasingly seen as an instrument to support sustainability transitions, such as transitions to a circular bio-based climate smart society. Living labs can create spaces for joint experimenting and learning by exploring the barriers and possibilities for transition and co-creating appropriate and viable solutions. These high expectations for and increased interest in living labs has sparked a keen interest in methods for assessing the performance of living labs. However, there is not yet an evaluation method or framework that is generally accepted and used. The few existing methods and frameworks mostly focus on the functioning of the living lab itself, and not on its wider impacts. Building on existing approaches and informed by the experiences in three living labs, we developed an assessment framework that enables the capturing of the dynamic role and contribution of living labs. This paper describes the framework and how it was developed. The paper contributes to the development of appropriate ways of assessing the functioning of living labs and the ways in which they contribute to sustainability transitions.

1. Introduction

1.1. Context

Over the past two decades, the living labs approach has been applied in a variety of contexts and sectors. Originally it focused on technological innovation, but later its application expanded to broader social challenges in areas such as eHealth, smart cities, public sector innovation, university campuses, and rural development [1,2]. In recent years, living labs have increasingly been embraced as a key instrument in innovation policies, particularly in the context of the aspired sustainability transitions, such as in agriculture and food systems [2]. In the coming decade, agro-food systems will need to be transformed in order to be able to adapt to expected climate change [3], reduce pollution and soil degradation, and restore biodiversity [4,5]. Developing a sustainable and circular agro-food system requires co-operation between multiple actors and the testing out of new farming practices and new production methods. However, changing the agro-food system also requires more fundamental shifts in our perception of food production, the role of food in our society, the way cost price is determined, and the roles and responsibilities of the actors in the food chain. Living labs are thought to be capable of addressing these challenges, since they enable the involvement of multiple and diverse actors, create opportunities for overcoming fragmented decision-making powers, and bring to the table the multiple values to be taken into account. By fostering experimentation and the development of knowledge in real-life settings in a participatory and co-creative way [6], they offer opportunities for finding appropriate and viable ways forward. Based on multiple case studies in different domains, Compagnucci et al., (2020) conclude that living labs are able to create synergies between different stakeholders (particularly from the fields of science, business and policy) and to create a diversity of potential benefits in terms of good practices, knowledge production, knowledge transfer, local development, transdisciplinarity, user engagement and creativity [7]. The EU is among those who consider living labs to be an important instrument of innovation policies, and actively supports the development of the European Network of Living Labs (ENoLL) and their application in new programs related to agroecology living labs and research infrastructures [8,9].

Various definitions of living labs have been proposed, each emphasizing certain aspects of living labs [10,11]. No single definition has become particularly dominant or gained widespread acceptance [2]. Nevertheless, common characteristics can be noted. Living labs are generally considered to be collaborative platforms for various actors from research, government, and business, and citizens, to work together to address complex challenges. Living labs are seen as spaces for joint experimentation and learning by exploring the barriers and possibilities, with preferably high levels of co-creation and experiments [10]. Living labs in the context of agro-food systems generally seek to contribute to transitions to more sustainable agri-food systems. This may involve different sustainable transition pathways with specific transition challenges that living labs can address [12].

Many initiatives are called living labs, but in this paper we consider living labs to be iterative collaborative platforms involving co-creation and experimentation by various actors from research, government and business, and citizens, working together to address transition challenges in a real life context.

1.2. Research Question and Research Approach

Given the widespread occurrence of living labs, and the high expectations that many have regarding their potential contribution in relation to sustainable transitions, one would expect that evaluation of living labs is common practice. Not only would it be relevant to find out what exactly living labs contribute to sustainability transitions, but also how they do so, and what factors determine whether or not living labs can make such contributions. However, there is not yet an evaluation method or framework that is generally accepted and used. The few existing methods and frameworks mostly focus on the functioning of the living lab itself and not its wider impacts [8]. Although the first is important to evaluate the dynamics of the living lab, the second is important to be able to evaluate to what extent they contribute to broader societal goals (Ibid.). This is particularly relevant because some authors suggest that living labs are in many cases isolated events and will not result in further changes in society without government support [13]. One of the issues complicating the development of an assessment method is the diversity in living labs, as they can be quite different in their specific setup, goals, the scale at which operate, the relative complexity of the issues at hand and the context in which they operate. Despite the diversity among living labs, the need for assessing whether a particular living lab is living up to expectations or not is often expressed by living lab participants.

This paper addresses this need for a systematic approach to the evaluation of living labs, one that supports the assessment of both their functioning and their contribution to sustainability transitions. Being involved in three living labs in the Netherlands in the field of agro-food systems, we also found ourselves in need of such an approach. The three living labs focus on three different topics: (1) development of a sustainable French fries chain, (2) sustainable cheese production and the avoidance of soil subsidence, and (3) future-proofing agriculture in the Netherlands. The first two living labs are primarily focused on a specific region and a particular food chain (dairy farming and cheese production, and field crops and French fries) while the latter is addressing agriculture in the whole of the Netherlands. These living labs were used as a reference point in developing a systematic assessment approach for living labs. As the living labs are still ongoing, the assessment approach should be suited for retrospectively reviewing our achievements (e.g., ex post) as well as proactively monitoring to change and adapt the living labs whilst they are unfolding (e.g., ex durante). These desired features were taken on board in the development of the approach.

The question this article addresses is: what key aspects and dynamics of living labs need to be assessed to understand how a living lab is functioning and contributing to relevant wider transitions to sustainability?

Two methodological avenues interactively informed the development of an answer to this question. First, we explored existing frameworks or approaches for evaluating policies or projects, and assessed the aspects and dynamics included and their suitability for our living labs. As this provided insufficient guidance, we reviewed the following four different strands of the literature for inspiration: (i) a results-based monitoring and evaluation approaches, (ii) reflexive evaluation approaches, (iii) innovation experiments/living labs literature, and (iv) capacity development approaches (see Section 2 for a more elaborate explanation).

Second, we explored the evaluation needs in the on-the-ground realities in the three living lab cases in the Netherlands. As participants in the living labs, we facilitated several workshops with the other participants to gather their needs for evaluation. This led to insights which we used in developing the framework. This secondary exploration enriched our understanding of who needs to be able to use the framework and for what purpose. In an iterative process in which the insights from the literature and the needs of the living lab were combined, the framework for assessing the living lab which is described in this article was developed.

1.3. Outline of the Article

In Section 2, we present literature building blocks for the development of an appropriate assessment framework. In Section 3 we outline the needs of the three living labs cases. In Section 4, we present the framework that we composed from the literature review and the experience of the three cases. In Section 5, we discuss the strengths and weaknesses of the assessment framework. In Section 6 we briefly reflect on what this paper contributes to living labs as a field of study and practice.

2. Evaluating Living Labs: Building Blocks from the Literature

This section discusses five complementary angles and explains how they informed our assessment framework:

• A living lab as a change initiative leading to tangible results
• A living lab as a transition experiment
• A living lab as a multi-stakeholder partnership
• A living lab as a collective of capabilities
• A living lab as different from other living labs

We decided that each of these angles would need to be covered in an assessment framework, and that we would, therefore, need some insights from the literature regarding how best to assess that particular angle.

2.1. A Living Lab as a Change Initiative

A living lab can be perceived as a change initiative with tangible outcomes comparable to a project or program. It applies an input–output–outcome–impact perspective on performance of a change initiative (e.g., [14]). From a temporal perspective, there are three ways of assessing a living lab as a change initiative: ex ante (before, during design), ex durante (during operations), and ex post (after operations).

Results-based management (RBM) is a common approach for assessing performance both ex durante and ex post. Its focus on the link between performance and results is a strength. A clear perspective on results from living labs would be of added value to our aims, because we focus on the contribution of the living lab to sustainability transition challenges. Often these results are expressed in quantitative assessments using indicators. However, an RBM approach is often considered less suitable for transition challenges in which goals, objectives and relations between measures, expected outcomes and impacts are either not explicit or unknown. Furthermore, in many instances, factors influencing the transition are outside of the sphere of control of the initiative (see Figure 1). Therefore, for transition initiatives, often a theory of change is described [15,16] which has become a rather common way of articulating the change required and the steps along the way. Furthermore, RBM does little in terms of unpacking performance in terms of (social) processes and relationships [17,18], a key dimension in living labs. We decided that our framework would, therefore, need to be capable of considering the flow from inputs to impact, and to pay attention to collaborative and interactive processes. In other words, we would need to deliver more than a standard performance management framework.

2.2. A Living Lab as a Transition Experiment

Transition experiments, at first glance, appear to be closely related to our topic of the contribution of living labs in addressing sustainability transition challenges. For transition experiments, Luederitz et al. (2017) [20] proposed an evaluation scheme that looks promising for our purpose. Their description of transition experiments as “collaborative science–society initiatives” [17] (p. 62) comes close to the type of living labs we discuss in Section 3. Furthermore, their purpose is to present an evaluative scheme that is (i) generic, i.e., applicable to different types of sustainability transition experiments; (ii) comprehensive, i.e., capturing the ultimate outcomes as well as the intermediate and mediating attributes (inputs, processes, outputs) of experiments; (iii) operational, i.e., ready to be applied (including guidance on how to specify the scheme for application to particular cases and contexts); and (iv) formative, i.e., supporting experiments to become more effective and efficient [17] (p. 63). Williams and Robinson (2020) [21], and Williams (2019) [22], also developed an assessment framework for sustainability transition experiments. Both frameworks offer useful categories along the lines of which systematic assessment can be structured.

However, upon closer observation and when trying to apply the approach of transition experiments in the three cases discussed in the next section, we ran into a number of limitations. First of all, the evaluation of sustainability transition experiments tends to be approached according to an input–output–outcome process. This may be suitable for projects, but—as already discussed in 2.1—we think it does not sufficiently pay attention to the social interactions and learning processes that are at the heart of living labs. Other limitations we found relate to (1) the wide variety of categories applied (in the case of Luederitz et al. (2017) [20], more than twenty), (2) the inclusion of difficult concepts, such as “intra- and intergenerational equity”, which are hard to grasp, and (3) the clustering of assessment categories in ways that do not follow the flow of processes. This renders these approaches more suitable for formal/academic use, and less suitable for supporting key actors in living labs.

Due to these shortcomings, we could not merely choose one of these evaluative schemes and needed to develop something new to suit our purposes. Nevertheless, many traces of the work of Luederitz et al. (2017) [20] can be found in the framework we present in Section 4.

In addition, related to transition experiments, we realized that assessing the extent to which a living lab can contribute effectively to sustainability transitions in a multi-level perspective (MLP) [23] means that we need to take into account the ability of the lab to position itself as an initiative in relation to the niche and regime levels (illustrated in Figure 2). The niche level relates to innovations emerging in conducive conditions, whilst the regime level is the dominant system configuration.

2.3. A Living Lab as a Multi-Stakeholder Partnership

Living labs consist of a diverse group of actors working together for a common goal, and sharing resources (in terms of money, knowledge and network) in joint experimentation. For capturing the functioning of such processes of collaboration and learning, reflexive approaches for evaluation are relevant [24]. Such approaches focus on learning processes of participants, on the evolution of the entire process and the view of one’s own achievements, and often use qualitative methods such as interviews, surveys, workshops and case studies. Although very suitable to understanding the process, the weakness of this approach is its lack of insight provided into results and the wider impacts of living labs. In deciding which aspects should be reviewed, we were informed by ten key attributes of effective partnering as defined by Mundy and Tennyson (2019) [25].

In reflective evaluations there is a focus on the individual characteristics of the involved people, but for the functioning of the living lab the capabilities of the partnership as a whole are important when analyzing how the living lab is functioning. In the next section we therefore further expand on this capability perspective.

2.4. A Living Lab as a Collective of Capabilities

This fourth complementary angle on the evaluation of living labs in relation to sustainability transitions is also about the living aspect of it, but it further zooms in on its potential: what the actors are (potentially) collectively capable of (and what not). Perhaps the living lab intended to perform in an area in which none of the participants had relevant capabilities. This is about capacity assessment, not from an organizational point of view, but from a perspective of collective capabilities [26]. In the process of developing insights from the three cases, we found a capability perspective helpful in organizing such insights. We considered the five capabilities as suggested by Baser and Morgan (2008) [27] to interactively shape the capacity of a particular group (see also [28,29]).

In other words, overall capacity is the emergent outcome of the interactive performance of those five capabilities. Applying this perspective to the context of living labs means asking about the capacity of a living lab to contribute to relevant/pertinent sustainability challenges. Such capacity is then the outcome of the state of interacting collective capabilities to which all participants in the living lab contribute.

We designed the assessment framework in such a way that it can be applied for assessing collective capabilities of partners/participants in the living lab. We have not, however, simply adopted the capability descriptions proposed by Baser and Morgan (2008) [23], but have rather tuned descriptions more to the specific context of living labs, inspired by the experiences of the three living labs that we studied.

2.5. A Living Lab as Different from Other Living Labs

Each living lab is situated in a specific context with different characteristics in terms of, e.g., pertinent challenges, geography, and (political and social) room for maneuver. This means that an assessment framework needs to include options for appropriate contextualization and customization. The assessment framework, therefore, needs to be capable of (1) being responsive to the specific nature of living labs, and (2) being responsive to the specific context of each living lab in terms of pertinent (transition) challenges, types of actors involved, available resources, room for maneuver, etc.

Therefore, from the broader literature on the monitoring and evaluation of change initiatives, we selected the practice of defining performance questions as the key to effective monitoring and evaluation [18]. This offers a balanced approach between clarifying information needs in general (how will we know whether a particular aspect of the living lab is working out well), while not fixing it in a list of detailed indicators. In this, we align with the approach of Luederitz et al., 2017 [20], in their evaluative framework for transition experiments.

3. Evaluating Living Labs: Needs and Experiences from Case Studies

In this section we describe the three living labs in the Netherlands in which we explored the evaluation needs, and how this exploration enriched our understanding of who needs to be able to use the framework and for what purpose. We also outline how the living labs’ needs were combined in the framework for assessing them.

3.1. Introduction to the Three Dutch Living Labs

We used three existing living labs in the Netherlands to inform the development of our framework: (i) The Farm of the Future—a research and demonstration farm experimenting with practices of circular farming in arable production, (ii) The Green Circle Cheese and Soil subsidence—a network of businesses, governments and a research institute working on the reduction of peat soil subsidence in a landscape dominated by dairy farming, and (iii) The Green Circle Sustainable French Fries chain—a similar network focusing on sustainability of a French fries chain. In Appendix A the main characteristics of these living labs are described. The authors were involved in these living labs prior to the development of the assessment framework. This gave us a good insight into the general practice of the living labs and their (often implicit) theory of change. In addition, it gave us excellent access to the stakeholders for testing ideas and co-design. Such interactions included discussing the purpose of evaluating, theories of change, the general outline of an assessment framework as well as choice of specific indicators, and the drawing of a visual for each living lab (see Supplementary Materials). These visuals in turn facilitated discussions about the living labs and their evaluation. In addition, the researchers reviewed earlier and partial evaluations of the living labs. As the living labs are in various stages of development, insights into the role of evaluating in these various stages could be used. Through continuous confrontation of the general, conceptual work with the practice of the living labs, we aimed at making the framework robust and usable.

3.2. Insights for the Assessment Framework Arising from the Living Labs

3.2.1. Living Lab ‘Farm of the Future’

In the Farm of the Future an earlier version of the framework was used to discuss and design an appropriate assessment approach with the project leaders. Additionally, a meeting was held with the project team and two sessions with one of the project leaders. Below we share some insights on the functioning of the Farm of the Future and on the resulting requirements on an assessment framework:

• The Farm of the Future is designed to play a role in supporting the transition to circular agriculture, rather than as a project to deliver concrete planned results. The process with the stakeholders is important in ensuring that stakeholders can trust that appropriate results will be generated. Therefore, an implicit and flexible approach in interaction with the stakeholders is used to guide the Farm of the Future. An assessment approach should do justice to the flexible approach of the development of the living lab to adjust to the needs of stakeholders.
• The Farm of the Future connects directly to common goals set for sustainable agriculture in the Netherlands and translates these during the design process of field labs towards location-specific goals. It is also crucial that the Farm of the Future focuses on relevant present-day farming practices. There is a lot of discussion on finding the right balance between innovation on the one hand and feasibility for the current farming practice on the other. An assessment framework should help to assess this balancing act.
• Different processes in the Farm of the Future involve different ways of engaging stakeholders. Researchers are in a lead role, but in the design process especially farmers and other stakeholders are actively involved. The sense of shared ownership is deemed important for effective experimentation, but this is not yet well developed. An assessment framework should shed light on the type of relations between stakeholders inside the lab, and between those and stakeholders outside the lab.
• The Farm of the Future uses vouchers as vehicles to invite stakeholders to implement their own experiments, thus becoming more involved and creating a sense of ownership. In practice, this feature is mostly used for testing in precision farming. The characterization of the FotF as a puzzle atelier, a platform for the joint search for solutions, does not receive much attention in practice. An assessment framework should help to understand how resources are shared and how this affects stakeholder relations.
• The ambitious name Farm of the Future helps to attract funding and a lot of positive attention from local, national and international media. An assessment should help to understand the origin and effects of the image of a living lab. These insights can be helpful for setting a communication strategy and for strategic decision-making.
• A positive image of the living lab and successful results on the ground are two separate things. An assessment framework should help to distinguish between these two different results. It should provide insight into the role of the living lab and its foreseen and unforeseen effects in the world.
• The technical aspects of the farming system receive a lot of attention, while the story of the multistakeholder process to contribute to the transition to circular agriculture is less clearly communicated. It is a challenge to communicate and represent the broadness of the Farm of the Future, encompassing both the big story of transition and the small story of developing agro-ecological systems. The bias towards the technical plays out in the project team, as in the Steering Group and during field excursions. An assessment framework should deliberately encompass both levels of effects.

3.2.2. Living Lab Green Circle ‘Sustainable French Fries Chain’

During the first evaluation, which took place in 2021, the draft framework was used for an ex durante evaluation in the project team. During the assessment of the activities of the first three years of the Green Circle Sustainable French Fries, several lessons and challenges were noted that informed the assessment framework:

• During the first two years, much energy was devoted to co-operation between the four partners involved in the network. This consisted of getting to know each other, understanding each other’s point of view, and identifying the sustainability directions that needed to be addressed and what role every partner could play in the living lab. An assessment framework should enable considerations on how the team worked together and how this evolved over time.
• All partners committed resources to the different projects and activities undertaken in the living labs during the first three years. During the discussions it became apparent that resources were considered from a wide perspective; not only funding in cash, but also time, network contact and available knowledge were deemed important. An assessment framework should be able to review which of the resources were invested in the living lab, in which way the living lab managed to increase resources and how it used its network to further their activities.
• Several communication activities were undertaken over the course of the three years to showcase the way of working in the Green Circle Sustainable French Fries and the first results. However, during the evaluation, the partners realized that we could ‘count’ the number of communication activities but that it was unclear whether we reached the stakeholders sufficiently. It was concluded that we needed to pay much more attention to the assessment of the effect of communication and outreach activities undertaken.
• Working together with different partners also revealed different expectations for the time period in which results of the living labs would emerge. The business partner and province particularly wanted to have concrete results in the first three years, to show success and be able to justify spending time and energy in the living labs. Therefore, during the activities developed in the living labs, a preference was given to activities based on feasibility and the timeframe in which results could be expected. This also fueled the enthusiasm of all partners to work together on concrete activities, and contributed to the success of the partnership. However, the more research-oriented partners felt that sometimes things were moving too fast and that a study was seldom undertaken which considered all options and reviewed all trade-offs in depth. Therefore, the extent to which a living lab can undertake practical activities first needs to be incorporated in an assessment framework.
• The need for action also had a downside, which can lead to internal and external discussion on the way the living lab is moving forward and whether the choices made are indeed contributing to a more sustainable outcome. An assessment should take into account how far different avenues were considered, explored and discarded for good reasons.
• During the assessment, the need was also expressed to assess the real impact of activities: what the living lab achieved regarding the ambitions set to reduce greenhouse gas emissions, increase the quality of the living environment, contribute to biodiversity protection, etc. However, in many instances information was missing when assessing the impact of activities—either because there is still (scientific) debate on how to measure the impact (for instance CO₂ sequestration in agricultural soils), or because activities were unfolding so quickly that time for assessing the ‘before activity’ situation was lacking. An assessment framework should consider this measuring impact prior to the development of activities.

3.2.3. Living Lab Green Circle Cheese and Soil Subsidence

Stakeholders in the Green Circle Cheese and Soil Subsidence were asked early on in our design process what they would need in an assessment framework (18 May 2020). Later, a draft version of the framework was used in an interactive session with the key partners for an ex ante assessment (29 March 2022). At that time, the partnership had existed for three years and the partners wanted to look ahead in order to rethink and re-design the living lab. While the focus was on the future, naturally the experience from the first three years was important in these discussions. The sessions yielded valuable insights into the requirements for an assessment framework.

• The purpose of an assessment according to the stakeholders was, in the first place, to learn about the lab’s own functioning. This is an internal goal. In addition, an assessment could be useful for external communication: to be accountable to the participating organizations and funders, to show achievements, to gain support and connections, and to inform. Because of this, the framework should be suitable for use by the core team of the living lab; it should not be too complicated. Nevertheless, the stakeholders expressed the need for a broad assessment in terms of indicators, not only outcomes and impact, but also support in the network, organizational aspects and learning.
• The participants expect that upcoming agricultural and environmental policy at national and EU level will affect the area greatly. The Green Circle wants to remain relevant in its contribution to transition challenges, and at the same time make things practical and envisionable for the farmers in the area. An assessment approach should pay attention to the raison d’ être of a living lab as well as to its ability to translate big transition issues to actions on the ground.
• The partners’ aspiration is that all farms in the area will receive a basic contract and a plan for making their farm more sustainable. This will be quite a challenge for the partnership, because capacity in terms of person time and funds is not yet available and surpasses the current investments and projects. An assessment approach should help to identify important stakeholders with whom ties need to be strengthened in order to be able to achieve the network’s aspirations.
• The partnership renewed its common ‘dream’, which comprises of a short storyline of the aspirations that guide the joint efforts. In addition, the participants revised the way of meeting in thematic subgroups, since that hinders an integrated way of working. They redesigned the way of working and decided that networks needed to be formed around projects. An assessment framework should support reflection on that which binds the participants together as well as on the structures and processes for interaction.
• The partners reflected on the type of innovations that the living lab supports. While the first phase focused on technological innovations and nature-based solutions, the partnership saw the need to complement those forms of innovation with social, economic and institutional innovation. New actions were formulated to repair this gap, focusing on organization and governance: for example, removing barriers to nature-based solutions. An assessment framework should be able to cope with such an evolution of scope and activities.
• The good reputation of the Green Circle shows in the willingness of people to participate in the network and its activities. Nevertheless, it was concluded that the stories of the living lab can be communicated better to a broader audience for a higher impact. According to the participants, sharing lessons learnt with broader networks is one of the reasons for the existence of a living lab. Therefore, an assessment approach should not only give attention to internal processes, but also to the strategies for external communication.
• In this period of renewing the partnership, evaluation and reflection have been important. This is valued by the partners, but there appeared to be a consensus that new ways needed to be found to make reflection and learning more of a part of the ongoing process. An assessment framework should be suitable for ex durante use.

3.3. Summary of Key Insights from the Three Involved Living Labs to Inform the Development of the Framework

The application of earlier versions of the framework in the three living labs stimulated discussion and reflection on the activities of the living labs, regardless of whether it was used in looking forward at the beginning of the living lab (as ex ante), in looking backwards to the first three years of working together (ex post) or in considering next steps (ex durante). In the two living labs which used the framework from a forward-looking perspective, the discussion on the motivation for the living lab created a joint understanding about the transition they connected to and about which activities were relevant to partners and broader stakeholders.

Despite the differences in the living labs, several commonalities in assessment needs of the living labs could be identified (see Figure 3). With regard to internal interactions, all three labs invested much time in internal co-operation in order to reach a common understanding and trust, and this was considered to be an important aspect to assess. At the same time, this internal focus sometimes hampered the extent of activities undertaken, as there appears to be a tension between investing time internally and developing activities. However, such concrete activities were considered essential to reach results as well as to keep partners engaged and motivated. The framework, therefore, should review both the internal processes and the extent and type of external activities undertaken. All living labs stressed the need to review the interaction of the living lab with its operational context, both in how it communicated its activities, and the extent to which living labs are recognized by and inspire others. An assessment of the products and services provided was considered relevant but at the same time partners considered this rather easy and straightforward. Furthermore, it was acknowledged that, in terms of effects, often small steps were taken in the form of pilot activities, making assessment of the results of pilot activities, for example on climate mitigation and biodiversity, relevant.

Finally, regardless of the content of the framework, an important learning point was that a discussion about the various issues that are part of an assessment with a broad stakeholder group is helped by drawings and visualizations to create a common understanding as well as to engage them meaningfully. In the Supplementary Materials visualization are presented that were used in the three living labs.

4. The Proposed Assessment Framework

4.1. The Assessment Framework and Its Key Features

In this section, we present the Living Lab Assessment Framework that we composed by interactively considering insights from the relevant literature (Section 2) and insights from the experience of three living lab cases in the Netherlands (Section 3). We elaborate on the framework and its key features, the core capabilities that are part of the framework, and how this framework can be applied through a set of living lab qualities and related performance questions. Figure 4 presents the different key features of the assessment framework. The framework is not intended to reflect a linear process. We indicate a particular sequence that presents a general flow of processes from design to effect. However, the processes will play out interactively and iteratively as well. The order of elements is, therefore, not strict. Arrows indicate ideal-type connections, but not actual sequence. All performance areas play a role in the extent to which the living lab is able to (1) facilitate collaborative search and shared learning, and (2) contribute to transition challenges.

In this framework, we assume that a living lab essentially revolves around two core functions:

• Providing a conducive ‘space’ where pertinent transition challenges can be addressed interactively. The living lab brings together different perspectives and fields of expertise to address challenges of shared concern in order to interactively find new ways forward through “collaborative search and shared learning”. This pooling of the capabilities of participants and the associated creation of social capital is a key characteristic of the type of living lab that we focus on here. The ‘space’ can take all kinds of shapes and forms, depending on what is found conducive in a particular context.
• Creating and informing (better) conditions for change, both directly and indirectly (spin-offs), which contributes to relevant stakeholders making concrete changes that address pertinent transition challenges, thus contributing to transition challenges. By doing so, it helps to shed light on that fact that in the short term a living lab may not have big effects, but it can still play an important role in terms of investing in social relationships and in social capital, and this supports wider transitions to sustainability (cf. [30]).

Both core functions are important and interrelated. The collaborative search and shared learning processes make sense to participants because they eventually are meant to lead to something useful and to contribute to positive change in relation to pertinent transition challenges. In other words, the collaborative search and shared learning are not purposes in themselves. The other way around, the conditions for change are expected to improve through effective processes of collaborative search and shared learning, as they will often involve more sense of ownership and commitment to collaboratively agreed ways forward.

The living lab is embedded in what is referred to in the framework as the relevant context. We have looked for common ground regarding what applies in most living labs. However, each context will have its own specific characteristics in which not all capabilities will be equally important. What collaborative search and shared learning processes will involve in terms of concrete activities, and what transition is exactly aspired, will, therefore, vary per living lab as indicated in the framework. Since living labs operate in a wider social, economic, political, and environmental context, it is about a process of interactively exploring ways forward in addressing pertinent transition challenges in a particular context, matching needs with appropriate activities. It is meant to be a search process, in the sense that there is room for adaptively exploring ways forward. The term ‘pertinent’ refers to the transition challenges that this particular living lab seeks to address, and in relation to which it seeks to make a contribution. Because of the specificity of context and transition challenges, performance areas need to be specified accordingly. Some living labs will be operating in a highly complex context in relation to complicated transition challenges, whereas others focus on ‘low hanging fruits’ in a rather conducive context. This places different demands on collective capabilities.

4.2. Collective Capabilities That Help the Two Core Functions of a Living Lab Flourish

The extent to which the two core functions of a living lab can flourish depends on a range of (external and internal) conditions and capabilities that can be conducive or constraining. In line with the five-capabilities approach discussed in Section 2, the framework does not consider individual (organisation) capabilities as much as it focuses on collective capabilities. Therefore, this is about pooling capabilities as group of participants. In the following, we further explain the essence of these collective capabilities, which emerged as important from the three case studies in relation to the seven performance areas.

Related to motivation of the living lab:

• Being responsive. This is about the origins of the living lab, and the way in which motivation for it stems from a sense of urgency to want to contribute to pertinent transition challenges. We consider this capability as unlocking the other ones, and different from the twelve other capabilities described below.

Related to the design and setup of the living lab:

• Being relevant. This is about being relevant in two ways: relevant in view of transition challenges, but also relevant for farmers and other stakeholders in view of (additional) challenges they face because of transition related policies and regulations, etc. These comprise urgent reasons for collaborating in the living lab;
• Being resourceful. This is about being able to secure resources (time, funds, knowledge, networks) needed to organise living lab-related processes.

Related to living lab interactions:

• Being connected. This is about being connected to other actors (outside the living lab) and to other initiatives and developments because of their role in relation to pertinent transition challenges. In other words, it is about preventing the living lab from performing as an island and ensuring appropriate (outward) connectivity. This is, therefore, also about providing follow-up services, guidance, and coaching beyond the living lab, or by ensuring that others provide such follow-up;
• Being in it together. This is about having good relationships between the partners in the living lab and creating opportunities so that participants feel shared ownership over aspirations, processes and outcomes.

Related to living lab actions:

• Being practical and propositional. This is directly related to the motivation of participants to keep participating. Unless something practical is on the table, a crystallization point for action will be lacking and participants may start to gradually walk out of the living lab. Small-scale experiments may belong to the core activities of a living lab;
• Being responsible. This is about balancing the need for being practical with a concern for quality, potential side-effects and long-term effects, trade-offs between different values, etc. It involves being anticipatory (able to foresee implications and consequences of actions), reflexive (monitoring how things work out), responsive (picking up early warning signals) and inclusive (considering implications and interests from a broad perspective).

Related to the positioning and reputation of the living lab:

• Being known. This is about the reach of communication and providing appropriate information for relevant audiences;
• Being acknowledged. This is about strategic communication in combination with reputation management—being able to articulate the relevance, efficacy, and quality of the LL’s contribution to relevant audiences.

Related to living lab products and services:

• Being generative. This is about an ability to bring forth concrete products and services;
• Being motivational. This is about an ability to engage people in living lab activities and to motivate target groups into exploring and considering new practices, e.g., by connecting to people’s core motivations. It includes public relationships management.

Related to the outcomes and impact of the living lab:

• Being effective. This is about the effects of the living lab in terms of social, economic and environmental outcomes and impacts, and reviewing both anticipated and unintended effects and trade-offs;
• Being adaptive. This follows up on being effective in terms of translating implications of findings from monitoring and evaluation into adaptive actions to better contribute to the transition challenges.

4.3. Application and Tailoring

In this section, we present an indicative overview of performance areas (qualities) and related performance questions for the twelve core capabilities, providing some guidance to tailoring the framework to specific living labs, and we suggest methodological approaches for using the assessment framework.

Table 1. Qualities and indicative key performance questions regarding twelve collective capabilities of the living lab. Being responsive relates to the origins of the living lab, and the way in which motivation for it stems from a sense of urgency to want to contribute to pertinent transition challenges. We consider this capability as unlocking and different from the other ones and, therefore, did not include it in the table below. Abbreviation LL = Living Lab.

Performance Areas	Collective Capabilities	Related Abilities	Key Performance Questions
Design and setup of the Living Lab	1. Being relevant	Ability to identify appropriate and feasible contributions to pertinent transition challenges; ability to focus on what matters most to stakeholders.	- Is there a shared understanding regarding which transition challenges are addressed by the LL? - To what extent are these transition challenges reason to collaborate in the LL? - To what extent is the LL relevant to the challenges that key stakeholders (e.g., farmers/businesses) face because of transition-related ambitions? In which landscapes and chains will the LL be relevant? - What are the concrete aspirations of the LL in terms of contribution to addressing the transition challenges?
	2. Being resourceful	Ability to gain access to resources (finance, networks, knowledge, time, etc.); ability to create conditions for the LL that match with its purpose and ambitions.	- Are those stakeholders who are relevant for the transition involved? - To what extent is the LL able to mobilize and secure resources for the LL? - To what extent is the LL able to appropriately organize/facilitate core processes of the LL?
Living Lab interactions	3. Being connected	Ability to connect to initiatives that aim to contribute to the same transition challenges; ability to connect to and influence stakeholders who can make a difference.	- Is the LL appropriately connected to other initiatives that address the transition challenges in complementary ways? How is the LL embedded in larger networks? - To what extent is the LL appropriately connected to stakeholders that matter in relation to the LL ambitions?
	4. Being in it together	Ability to forge and maintain good relationships in the LL; ability to balance diversity of stakeholders; ability to create ownership for the LL among participants and key stakeholders.	- To what extent is the LL able to facilitate good relationships among the participants and stakeholders? - To what extent is there a felt shared ownership of aspirations, processes and outcomes of the LL among the participants and stakeholders? - To what extent are decisions made jointly?
LL actions	5. Being practical and propositional	Ability to organize convergence from diversity of actors and interests, towards implementing concrete activities and experiments.	- To what extent is the LL propositional in terms of putting forward concrete actions, options and opportunities? - To what extent are activities aligned with the aspirations? - To what extent are the proposed actions considered feasible and useful by key stakeholders?
	6. Being responsible	Ability to anticipate (long-term) implications of innovations; ability to be appropriately inclusive (integrated perspective, stakeholder engagement), ability to operationalize shared values and principles.	- How are long-term implications of actions and proposed options assessed in an anticipatory way for planet, people and profit? - To what extent do participants live up to their mutual commitments and shared values?
Positioning and reputation of the Living Lab	7, 8. Being known and acknowledged	Ability to communicate well with target audiences; ability to communicate about the role of the LL in ways that are credible and receive respect.	- How well informed are relevant stakeholders about the (intentions of the) LL? - How useful or innovative is the LL according to key stakeholders? - How credible do key stakeholders consider the LL to be in terms of what it is proposing?
LL products and services	9. Being generative	Ability to implement activities that lead to results; ability to provide services and create products.	- How productive is the LL in terms of concrete outputs that are in line with aspirations of the LL? - How significant and useful are products and services generated and provided by the LL?
	10. Being motivational	Ability to engage, to inspire and to convince.	- To what extent is the LL capable of motivating stakeholders to engage in its activities? - To what extent is the LL influencing/inspiring changes in practices among LL participants? - To what extent are actions and proposals of the LL inspiring/motivating/enabling further/wider change of practices beyond the LL? - To what extent is the LL able to overcome difficulties and complications and move on?
Outcomes and impact of the Living Lab	11, 12. Being effective and adaptive	Ability to achieve/contribute to desired results; ability to adapt and adjust plans based on outcome and impact monitoring; ability to self-reflect and self-renew.	- What are the social, economic and environmental impacts of the LL? - Are effects in line with the intended contribution to addressing transition challenges? - To what extent is there room for adaptation in courses of action (set-up and activities) of the LL?

lists indicative performance questions for each of the twelve core capabilities. These questions are indicative of the type of questions that can be asked to explore the living labs readiness for contributing effectively to pertinent transition challenges. The questions will need to be adapted to the specifics of the living lab: its transition challenges, ambitions, context and the needs of stakeholders. When tailoring the framework to a specific living lab, some performance questions in Table 1 will be more relevant than others. In addition, questions will need to be further specified towards defining information needs and indicators.

The assessment framework can be used in the design phase of a living lab initiative (ex ante evaluation), for reflection on how the living lab is faring (ex durante evaluation), and in retrospective assessment (ex post evaluation). Each application will put its own emphasis on specific dimensions of the framework. Particularly during the phase of preparation and design, the framework dimensions may be used to articulate a theory of change. Furthermore, the assessment may focus more on what is (e.g., what actual performance), or more on what could be (e.g., for performance) or what is required in terms of collective capabilities. Through a focus on collective capabilities, the vulnerability of a living lab may become apparent, and/or it may point to the need to connect to additional stakeholders to strengthen collective capabilities as a living lab. In using the framework during the design of a living lab, the involved stakeholders may take some time to agree to what extent and how the different capabilities are relevant and important for their transition challenge in that relevant context.

The assessment framework may be applied for a quickscan, asking only few questions related to selected dimensions (see Appendix B for a template), or for a more elaborate analysis. Finally, while the assessment framework was developed for self-evaluation by the participant networks of living lab, it may also be used for external evaluation. In self-assessment, the framework can be used to periodically reflect in an interactive and systematic way.

5. Discussion

The framework presented in this article has several strengths and weaknesses.

Given the wide diversity in manifestations of living labs, the presented assessment framework is intended to be applicable to a broad range of living lab types, contexts and purposes. The framework has proven to be suitable for assessing and comparing diverse living labs at different stages of their development. However, the general character also implies that the framework needs to be tuned to the specific context and objectives of each living lab. In addition, it requires operationalization, determining suitable questions and assessment methods. This may require some experience in monitoring and evaluation. In this process of making the framework work for them, the participants in the living lab need to agree on the relevance and importance of each of the capabilities in their situation and how they would like to monitor and assess related performance. Though the assessment of how the living lab contributes to transition challenges is one of the two focus areas, the framework provides less detail for the assessment of the outcome and impacts of the living labs. Depending on the nature of the objectives of the living lab, a quantitative analysis of the effects of the living lab may be needed, e.g., on the amount of carbon sequestration achieved on farms influenced (indirectly) by the living lab, or the increase in biodiversity or social welfare. This framework will need to be complemented by other methods that are specifically developed to assess these effects.

A strength of the framework is that it is concise and can be applied as a quick scan, as well as for an elaborate assessment. One may also decide to use only selected parts of the framework that are considered to be of particular use and relevance to a living lab. It may also support comparative analysis of and exchange between living labs. In this sense, it differs from the more elaborate frameworks we reviewed [20,31].

Furthermore, the framework focuses on the assessment of a living lab and does not offer guidance for deciding whether a living lab is at all an appropriate approach for a specific situation. For this, another assessment is needed: for instance, the assessment tool for the conditions of a living lab developed by Potters et al. (2022) [32].

When living labs are only seen as an instrument of a government to support policy implementation, or as an instrument of researchers to help their research outputs (innovations) become more widely applied and used (scaled up), it may affect the nature and conditions for success of the living lab. This will be similar to dynamics in ‘innovation platforms’ that became popular in Africa [33,34]. By approaching the living lab as a project or scaling mechanism (helping to get particular (pre-developed) innovations applied at scale), the “collaborative search and shared learning” process may lose its edge as the focus will tend to be on achieving more or less predefined desired outcomes. Funding and support may then also become mainly focused on activities and outputs (meeting targets), and less on relationship building and the collaborative search process as such, in which there is room for finding new—and perhaps for participants more attractive—ways forward.

Similarly, living labs are often approached from a limiting input–output–outcome perspective. This tendency runs the risk of approaching living labs as mere projects with interventions and pre-defined outcomes. This may smother the very thing that makes a living lab a living lab. We hope that the framework presented in this paper helps to prevent approaching living labs from such—what may be considered reductionist—perspective. Through the central position of “collaborative search and shared learning” in the framework, we suggest active learning processes and reflexivity to be at the heart of living labs.

Last but not least, the assessment framework can also be used by policy makers and researchers to assess how choices regarding their interaction with a living lab in its different stages may affect core capabilities in the living lab, notably the capabilities of being relevant, being in it together, and being practical. This may point to concerns regarding the extent to which the living lab would still be relevant to (envisaged) participants, or the extent to which participants can have a sense of ownership of the living lab. Since (the concept of) living labs have become rather popular as policy instrument in recent years, policy makers (and researchers) should more consciously consider how they could enhance the capabilities of living labs, or where their interventions threaten to undermine them.

6. Conclusions

In recent years, the concept of living labs has become prominent in the context of innovation and sustainability transition challenges. This includes a range of European policy initiatives. In essence, they are about multi-stakeholder initiatives in which a process of collaborative search for ways forward in relation to transition challenges plays a central role. Finding ourselves in need of an assessment framework that would enable stakeholders and ourselves to meaningfully assess the living lab’s (potential) contribution to pertinent sustainability transition challenges (in particular in agriculture), we were unable to identify a suitable existing framework in the existing literature. We found useful concepts and elements of existing frameworks, and based on insights from this literature and based on the experience of three living lab cases in the Netherlands, we developed a new framework. This framework identifies seven performance areas and twelve associated core capabilities that can be tailored to suit a particular purpose. Given the wide variety of manifestations of living labs, as well as the wide variety of specific contexts in which they operate, the framework is generic and needs to be applied in a customized way. It does, however, offer a systematic basis for such customization. It can be applied ex ante as part of the design processes of new living labs, including the articulation of a theory of change, it can be applied ex durante to, e.g., support a mid-term review of the living lab’s performance, and it may be applied ex post to draw lessons from the experiences of a living lab. It is designed for a collective self-assessment by the stakeholders in a living lab, but it can also be used for an external assessment. In essence, it offers a way of thinking about the living lab in a structured way in light of its contribution to pertinent transition challenges. We developed this framework for living labs in the context of (sustainable) agriculture. However, because the approach articulates collective capabilities that are not specific to the agricultural context, we expect that the framework will be suitable for the assessment of living labs in other contexts as well, and we are looking forward to examples of such applications.