From Experimentation to Sustainability Transformation: Developing a Tool to Better Anticipate Upscaling of Urban Innovation Experiments

Abstract

Urban experiments are increasingly embraced for their potential to transform incumbent socio-technical systems by offering multifaceted, ‘high-quality’ learning. The early literature on sustainability transitions painted an optimistic picture of the impact of experiments, prescribing their role in managing transitions. More recently, scholars have elaborated on the different purposes and functions of experiments; however, they generally stress that, as of yet, there is scarce evidence for their effectiveness concerning transformation in practice. This paper develops a tool for more effective follow-ups after an experiment in practice, by anticipating contextual constraints on upscaling innovations. The tool has been developed through a design science research method by first doing action research on sustainable mobility innovations in four European cities and subsequently testing the prototype of the tool in five other places. Our findings suggest that this new tool improves conditions for wider implementation of the innovation being experimented with, and associated transformation. This is one key starting point for increasing the impact of experiments and accelerating urban sustainability transformation.

1. Introduction

Urban experimentation has emerged rapidly as an approach for urban innovation and urban planning across the globe [1]. Urban Living Labs (ULLs) serve as practical platforms for conducting urban experiments, which are temporary innovation projects. ULLs are, in general, multi-actor, urban collaborations specifically set up to design, test, and learn from social and technical innovations and interventions in ‘real life’, and serve as experimental ‘spaces’ within cities, changing the configuration of materials, people and practices to induce and learn about change [1,2]. They fit into both innovation approaches under the neoliberal logic of urban competitiveness, and also into the promise of more inclusive and open forms of experimentation that are capable of addressing pressing urban policy agendas surrounding climate change and smart governance [3]. They are being rapidly inserted into and overlaid onto existing urban governance structures, practices and networks in cities across Europe.

Living Labs and experimental approaches became popular after computer science scholars set up the MIT Media Lab at the end of the 1990s, but they are rooted in an older field of demonstration projects and pilots [4]. In general, Living Labs are mainly focused on small-scale field tests of new artifacts and techniques, generally addressing the interaction between technology and the user. In the past decades, these approaches attracted increasing attention from policymakers, after studies of sustainable innovation and transition stressed the relevance of innovation experiments for a more fundamental shift to sustainability and the need for socio-technical and societal change. Scholras proposed approaches like strategic niche management (SNM) and transition management (TM), assuming fundamental changes would take place through the ‘upscaling’ of experiments [5,6]. In this context, experiments are often labeled as ‘transition experiment’ and have a more particular definition as ‘an inclusive, practice-based and challenge-led initiative designed to promote system innovation through social learning under conditions of uncertainty and ambiguity’ [7].

Rotmans and Loorbach [8] coined deepening, broadening and scaling-up as ‘the management principles of transition management’, proposed as three subsequent steps. In their approach, deepening refers to ‘deep learning’ about new ways of performing a societal function, broadening refers to ‘repeating an experiment in another context and linking it up with other issues’, while scaling-up refers to fostering institutional embedding of the new practice. Van den Bosch [9] developed these principles into a more detailed prescriptive framework, proposing how practitioners can influence the contribution of experiments to transitions.

Kivimaa et al. [10], based on a review of climate resilience-related experiments, identify four categories concerning the purpose of experiments (i.e., niche creation, market creation, societal problem solving, and spatial planning), revealing that different categories of experiments differ in their outputs and outcomes. Some experiments can have significant outcomes that change the discourse, whereas others facilitate the emergence and diffusion of new technologies or direct changes in the built environment. Still, others allow different types of governance innovation to be employed and tested, contributing to change in policy and institutions. The authors conclude that there is a need for better and longer-term evaluation of the outputs and outcomes of past experiments, but there is little discussion on how this currently happens in practice.

Laakso et al. [11] suggest four categories of the potential functions and uses of experiments: testing, creating profound influence, multiplying influence, and promoting systemic change. Their triangle model can help actors and evaluators working with experiments to discern the particular characteristics and opportunities afforded by a given experiment, but how this can be done in more practical terms remains open. As a more recent survey across urban professionals notes the following [12]: Cities do experiment and learn, but implicitly and without a clear methodology or dedicated resources for capturing learning. Significant time and resources are wasted reinventing the wheel, often repeating tests of technical performance, at the expense of learning how to change (p. 176).

Next to this observation that experiments create little learning in practice yet, academics are criticized for offering little empirical basis for the process from experimenting to sustainability transformation [13]. In this context, sustainability transition or transformation generally refers to the process of fundamental reconfiguring unsustainable consumer- and production systems [14], i.e., shifting resource- and carbon-intensive and polluting practices with unequal accessibility and burdens across social groups, towards significantly more sustainable ones. (Although we are aware of debates that distinguish the terms transition and transformation (e.g., [15,16]), for the sake of this paper, we find the distinction unnecessary, so we use the terms synonymously. In any case, the term needs to be made specific for a particular case in practice, at a particular place and time.) Although the concept of learning is widely used in studies of sustainability transition, transition management, etc., analysis of learning in particular experiments has remained empirically poor [13]. Porter et al. [17] is an exception by checking to what extent TM principles were applied in two water-related experimental projects in Amsterdam, offering both recommendations to practitioners in the particular project as well as suggestions for additional principles. Also, they show the relevance of distinguishing between learning processes at the program level and those of a specific project. Raven et al. [18] criticize transition management for being more forward-looking, and less focused on analyzing the confrontation of niche experiments with the established socio-institutional context (Raven et al. [18] also criticize strategic niche management for being more focused internally on the experiment, in particular on how the experiment (to which they refer as ‘protective space’) needs to be shielded, nurtured and empowered [19]). The transferability and scalability of these experimental solutions or lessons are often limited due to their focus on local, highly contextualized knowledge, missing the link to system-wide transformations [20]. Consequently, while urban experimentation holds significant potential for innovative urban governance and broader societal transformation, it also faces obstacles that impede the upscaling of successful experiments [21], the latter being neglected in the literature. Sengers et al. [7] describe how the early literature on sustainability transitions painted a rosy picture of the experiments as a great source of hope to change such a context:

Experiments are often seen as the seeds of sustainable change that should be cherished and protected since they might flourish to transform incumbent socio-technical systems. They are geared to engender structural change by allowing the actors involved to learn about the kind of structures that are prohibiting wider diffusion of the tested socio-technical configuration by directly initiating change on the small scale of the experiment’s direct environment. (p. 161)

However, authors also describe that in practice, too often, sustainability-oriented experiments are isolated events that fade into oblivion without any effect on incumbent regimes (ibid.). If actors only try to gain legitimacy through ‘experimentation-speak’ without taking hard measures to dismantle incumbent regimes, then they are frustrating rather than fostering sustainability transitions. The authors conclude that there is a need to describe the actual practices in experimentation in more detail and devote more attention to the micro-politics and matters of inclusion and exclusion. Notably, however, these conclusions seem to be focused internally on the experiment, rather than helping to dismantle the regime.

Frantzeskaki et al. [22] noted that, in several cities, impacts on adjacent or interrelated systems have been demonstrated, but that wider impacts of local experiments beyond their specific context of operation are difficult to find. Similar observations have recently been made by Eneqvist & Karvonen [23] and Evans et al. [12]. Beukers and Bertolini [24] provide principles for developing more explicit, testable, and improvable learning strategies in urban experimentation. In a subsequent study, they apply and test a learning strategy, which offers clearer insights of what works (e.g., the use of learning exercises or guiding questions, stimulating ‘self-learning,’ strong moderation, enough time, and a diverse group of participants), and what does not work (e.g., input from individual experts), when organizing learning events related to urban experimentation [25]. However, the study shows that lessons learned from experiments are still difficult to share with those not involved in the experiment, which constrains upscaling potential.

In other words, although a systematic evaluation of the effectiveness of (upscaling) urban experiments is non-existent, in practice, the contribution of urban experimentation to sustainability transformations seems rather disappointing thus far [26,27,28]. This is in line with Castán Broto et al. [29], who, based on an analysis of a database of 400 flagship sustainability initiatives from over 200 local governments, conclude that urban transformative capacity in practice, i.e., the ability of an urban system to reconfigure and move towards a new and more sustainable state, is rather thin. Various explanations have been offered. First, experimentation may be only done as lip-service to transitions [7], and that their focus may be limited to local contextual knowledge [20]. Another explanation is that urban experiments are often project-funded [30], without a budget for longer-term follow-ups, either in terms of subsequent experiments or implementation. Practitioners thus tend to neglect the anticipation of upscaling beyond the scale of the experiment [21], both in time and space [31], and learning processes tend to remain rather implicit and unstructured [12]. In addition, many of the local urban actors participating in experiments do not necessarily aim at transformative change or diffusion beyond the boundaries of the lab [32,33].

In order to enhance the contribution of urban experimentation to sustainability transformations in practice, and tackle some of the limitations mentioned above (in particular the lack of anticipation and poor learning processes), this paper develops a tool to support urban experimenters to learn more about the contextual constraints on upscaling, and accordingly, about the most appropriate ‘next steps’ after an experiment, by anticipating constraints on wider implementation of the innovation that is being experimented with. Our research question is as follows: How can urban professionals involved in experiments be supported to learn more from experiments concerning the most fruitful follow-up of an urban experiment to contribute to sustainability transformation? Our paper is structured as follows. In Section 2, we discuss our conceptual approach concerning the anticipation of constraints and describe the design science research method we adopted to develop the tool. Section 3 describes the results, i.e., (1) the designed tool, and (2) the lessons from the various iterations in the development process of the tool. The prototype of the tool is developed based on constraints and ways to anticipate found in earlier studies. Constraints on upscaling are factors or conditions that hinder upscaling. The prototype is subsequently developed through action research in four European cities and finally tested in sessions in five other places. The final tool can be seen as a set of guidelines to anticipate the upscaling of inclusive urban experiments and hence to improve the conditions for the impact of their results. Section 4 discusses the practical and scientific merits and points for improvement of the tool, and concludes.

This paper contributes to the transition literature a tool to support experimentation that is more than current approaches focused on analysis of and confrontation with the wider socio-institutional context. Rather than offering new insights on each of the constraints that the tool applies, the strength of this paper is that it makes knowledge ‘actionable’, bringing together many constraints mentioned throughout the literature, into a new tool that is developed through a design science method, i.e., a research method involving practical interventions and iterative testing. Based on a review of 217 transition studies [34], this research method is being flagged as currently hardly applied, whilst recommended to increase the practical impact of transition studies to sustainability. To practitioners, we contribute a tool that helps urban professionals take the next steps more effectively after an experiment. As is typical in DSR methodology, we do not claim any quantitative support for the effectiveness of the tool, but base our preliminary insights on the effect of the tool (i.e., validity) on accounts of participants (mostly urban professionals) in iterative testing. They voiced that applying the tool helped them take the next steps more effectively after an experiment, because contextual constraints were explicitly identified, so future plans, projects, or experiments could be designed to better anticipate these.

2. Materials and Methods

2.1. Conceptual Approach

The tool we seek to develop in this paper should help urban experimenters to better anticipate the upscaling of urban experiments and hence improve the conditions for impact of their results on sustainability transformation. The notion of upscaling has not been defined uniformly across innovation and transitions studies [35]. In addition to the early definitions mentioned in Section 1, social innovation scholars [36,37] distinguish ‘scaling up’ from ‘scaling out’ and ‘scaling deep’, taking the perspective of a (non-profit) organization that seeks to expand. ‘Scaling out’ refers to ‘diffusion’: the organization attempting to affect more people and cover a larger geographic area, whereas ‘scaling deep’ means further development in its own community (so taking geographic place as the main dimension of the scale of diffusion). ‘Scaling up’ is reserved for when ‘an organization aims to affect everybody who is in need of the social innovation they offer, or to address the larger institutional roots of a problem (ibid.)’. Although related, these social innovation studies do not address expansion after (multi-actor) experiments, but take the perspective of an organization proactively seeking impact, without an explicit conceptualization of their social environment. Transition scholars have generally advocated a perspective connected to experimentation, such as Kemp and Grin [38] referring to upscaling as ‘the emergence of a set of new practices learned from practical experiments, with corresponding new structure and culture elements’, taking, albeit loosely, a social practices perspective. Nevertheless, there remains a tendency in some more recent studies to echo classical diffusion literature [39] (first published in 1962), and simplify the notion of upscaling as growth of the number of consumers or units of production. Meijer et al. [40], for instance, define upscaling of car sharing as growth of users or members or of driven kilometers or shared vehicles. In similar ways, Bauwens et al. [41] refer to upscaling as the process of offering, to a wider base of beneficiaries, access to Community Enterprises’ products and services, either through reaching different customer profiles or through expanding to new locations. However, other more recent studies include more dimensions in the concept of upscaling. Koehrsen [42] defines upscaling as the dissemination of the novelty, translating niche experiments to a broader social level and seeking to increase its societal impact. This can involve increasing the number of experiments and their geographical expansion, staging showcase projects, accommodating the novelty to the regime, widening social networks by winning ever more actors to support the innovation, and improving the social acceptance of the visions behind the novelty. In this same vein, Meelen et al. [43] take as three main dimensions of upscaling: system build-up, geographical circulation, and reconfiguration of incumbent socio-technical regimes. We follow a slightly narrower definition of upscaling by conceptualizing it as a process with two key dimensions: the expansion of practices under experiment, and the increasing stability of those practices. Expansion is then: an increasing number of practitioners shifting to an emerging practice or reconfiguring their practice (i.e., a particular routine, like car sharing or cycling); increasing stability of a practice refers to stronger embedment in regulations (e.g., standardization), more supporting (material) infrastructures, or stronger collective discursive frames, expectations, etc.

In line with our definition of sustainability transformation mentioned above, urban sustainability transformations, in this study, are not about technological or social innovations or their impacts per se, but about how multiple innovations, including those in policy and planning, shape patterns or forms of reconfiguration in urban practices [44], such as the way people travel, work, live at home, shop, relax, etc. Accordingly, upscaling involves the wider expansion of more sustainable ‘practices’, so ways of doing something, not only by citizens (e.g., traveling less by car, more with car alternatives), but in entanglement with particular business practices (e.g., more sharing schemes) and urban planning practices (e.g., more road space for car alternatives).

When it comes to scaling up and broader urban transformations, many professionals tend towards a narrow focus on market-driven replication of technical innovations that hides the range of processes that are required to articulate solutions into different urban contexts [12]. Funding schemes position commercial markets and technical performance as the motor of change in cities, but pay little attention to how cities develop new organizational processes. According to Evans et al. [12], city coordinators are learning that replication is not so much about technical performance but about finding the right approach to city governance that enables them: The devil isn’t so much in the technology—you can get it working—but the devil is in the stakeholders…. We never bothered too much with these questions and for me, these questions are really essential and this should be the start. (p. 178).

According to Dijk, de Kraker and Hommels [21], scaling up can be more effective when contextual constraints are explicitly identified by conducting a retrospective analysis of the system that facilitates the design of experiments, to better anticipate them. Based on an analysis of an E-bus experiment, the study showed how upscaling can be made more effective by (1) identifying the specific constraints on upscaling for this innovation in retrospective systems analysis, and (2) anticipating constraints by addressing them in urban experiments explicitly, e.g., through formulating joint learning goals about them [21] (pg. 13). These can help making learning processes more explicit and structured than currently the case [12]. On the analysis and anticipation of constraints on upscaling, both the time and space dimensions are significant [31].

Constraints on upscaling of innovations from experiments, i.e., factors or conditions that hinder upscaling, have been discussed explicitly or more implicitly in a range of scientific fields [45]. We will draw from and engage with studies from the relevant fields explicitly when we describe the tool in detail in Section 3.1.

2.2. Method

This study follows an adapted version of the design science research (DSR) methodology [46]. The DSR method consists of an iterative process of design, evaluation and improvement of a model or artifact, through application in practice [47]. Our design process followed four condensed steps—based on the seven steps detailed by Baldassarre et al. [48]—depicted in Figure 1: (1) problem definition and potential solution, (2) tool development and design, (3) demonstration, and (4) evaluation. Iterations between these steps were made, especially based on the outcomes of the evaluation activities, which provided insights for adaptation of the tool improvement, as long as action research and trial sessions in cities were performed.

Figure 1. The steps of the DSR method that were taken to address our research question.

For Step 1, we started with literature analysis involving both conceptual analysis of constraints on upscaling as well as published empirical examples (see [21]. This delivered an initial set of four types of constraints and ways to anticipate these. Subsequently, 11 retrospective cases in four cities (see Appendix A) were analyzed, leading to a preliminary set of 13 constraints and ways to anticipate [49].

At the start of Step 2, this first version (‘prototype’) of the tool consisted of a simple, two-column Table with (13) constraints (left column) and corresponding ways to anticipate (right column), based on desk research. The subsequent core of this Step 2 consisted of action research in four European cities, aimed at in-depth refining of the 13 constraints and ways to anticipate, in order to develop a second version of the tool based on work in different contexts.

Action research, in general, studies an intervention in practice in which the researcher was (at least partly) involved in organizing [50]. Action research is relevant for our study, because the most straightforward way to develop this second version is to do an intervention ourselves and, subsequently, evaluate the effect of it. Potential researcher bias was mitigated by maximizing transparency in elaborately describing our activities and aims, within the limit of the length of an academic article (and its appendices). We developed the tool in Step 2 in four sub-steps (executed in each of the four cities):

• Identify an ongoing urban innovation project that is done with the ambition to be implemented more broadly, i.e., scale up (see Table 1).
• Identify which of the (generic) constraints on upscaling are applicable for this particular project, and determine how to intervene, i.e., anticipate these specific constraints (see Appendix B).
• Intervene through a series of experimental activities (see Appendix C).
• Evaluate the (role of the tool in the) wider impact of the intervention. (Appendix D explicates the interview or focus group formats, with which these interventions were evaluated, while Appendix E presents the results of these interviews and focus groups, by describing the observed effects per constraint, in each city it was applied. Appendix F describes wider effects of the intervention.)

Table 1. Action research activities (2016–2019) developed in four European cities to design the tool.

City	Action Research Activities
Bellinzona (Switzerland)	The municipal government of Bellinzona has been aiming to reduce individual car use, especially by creating new cycling infrastructures and improving bus/train connections, but without success. The local university suggested developing an innovative mobility tracking app providing rewards in a gamified fashion, co-developed together with citizens and other organizations. The app was applied as an intervention to better anticipate constraints on upscaling car alternatives.
Brussels (Belgium)	Building on the experience of a project started by the environmental agency ‘Brussels Environment’ for measurement of urban air quality through eight high-detail wearable devices in 2013, a local civil society organization (BRAL), together with the university, suggested intervening and better anticipating upscaling societal and policy attention for air quality by involving a wide range of citizen groups using low-cost but user-friendly measurement devices.
Graz (Austria)	The municipality of Graz had a project aiming to scale up more participatory forms of mobility & spatial planning, but struggled with how to do this in practice. Graz University applied the upscaling tool in a series of participatory activities on (co)-redesigning a square annex and traffic hub.
Maastricht (Netherlands)	Like Bellinzona’s, Maastricht’s municipal government has aimed to restrain individual car use for years, but without success. A new project in the train station area has a similar aim, mainly planning new cycling crossings. Maastricht University applied the upscaling tool to design a number of participatory visioning & assessment activities to anticipate a number of ‘Maastricht-specific’ factors that might constrain upscaling car alternatives.

After action research in four cities (which led to the identification of a final set of 10 constraints), the initial table was developed into a ‘card deck’, consisting of 10 cards, designed by a graphic designer who also added a cartoon on each card. For three subsequent testing sessions, we shared A4 colored prints and PDFs of the cards. After these, the 10 cards were printed on thick postcard-format paper, so that the tool could be shared physically during final testing sessions in two cities.

The main demonstration (Step 3) is through testing of the tool in five other cities, different than those where the tool was developed. Three of the five testing sessions were organized deliberately in other parts of Europe than the four cities (in western, central, and mid-southern Europe) in which the tool was developed: Helsinki, Istanbul, and Santander. In these cities, the tool was applied to a recent experiment or innovation project together with local civil servants (Helsinki: Last Mile Jätkäsaari; Istanbul: Zemin Istanbul; and Santander: Smart City Santander) and was debated in a focus group (see format in Appendix D). Lastly, the final version of the tool was applied to a recent or ongoing experiment in two cities: Rotterdam (Mobility Challenge Hoogkwartier) and Amsterdam (Mobility BuurtHubs project).

In Step 4 (‘Evaluation’), we assessed the outcomes of testing the tool in the five additional cities, in addition to an evaluation among participants from the four cities in which it was developed. In particular, in the four action research projects, the evaluation analysis was performed in a qualitative way, namely through 18 in-depth interviews and nine focus groups. These are useful methods here because the phenomenon (i.e., upscaling) is new; therefore, the investigator especially seeks to answer why and how questions [51]. In this case, the ‘how’ question was as follows: How was the upscaling tool helpful in learning about broader implementation of the tested innovation? (see Appendix D). Interview and focus group transcriptions were analyzed without formal coding, giving more space to explore and trace any response of interviewees or participants. Software was not used for this. In the five additional cities, instead, the evaluation was performed via one focus group (each), including a presentation of the tool itself and a discussion on how the tool suggestions could be applied to their local cases. In one case, additional interviews with the local civil servants were also performed. Table 2 shortly describes how each action research demonstration was evaluated, as well as how the five testing sessions were evaluated.

Table 2. Evaluation sessions.

Type of Evaluation	Place	Evaluation Methods
Action Research	Bellinzona	One final focus group with civil servants and participants, and an interview with civil servants about (1) the future of mobility for the Bellinzona region and potential measures, and (2) the usefulness of the upscaling tool in designing and testing the app
Action Research	Brussels	Citizens and stakeholders participated in a total of 5 focus groups and 5 in-depth interviews on the potential of participatory measurements of air quality and the potential/limits of the approach.
Action Research	Graz	Two focus groups with members of the Executive Directorate for Urban Planning and three interviews with two external participation experts and one urban stakeholder, facilitated by two researchers, addressing the lessons for redesigning a square, as well as the usefulness of the upscaling tool.
Action Research	Maastricht	Nine interviews with participants (three civil servants, six urban stakeholders), to evaluate their experiences with the experiment, and a 90 min. focus group session with the municipality (to discuss the relevance of lessons learned for the City and Railway plan, as well as the usefulness of the tool).
First batch of (three) trials	Helsinki, Istanbul, Santander	One 120 min. focus group with three to five local civil servants in each city, plus five researchers participating. Together, they discussed both the case content as well as the need to adapt the tool due to local specificities.
Second batch of (two) trials	Amsterdam, Rotterdam	One 75–120 min. focus group with 3–5 civil servants; additionally, two interviews in Amsterdam. Both the case content, as well as the need to adapt the tool due to local specificities, were discussed.

3. Results

3.1. The Tool

The final tool consists of 10 cards with on each card on one side a typical constraint on upscaling, and on the other side a suggested way to anticipate this constraint (see Figure 2). We discuss each combination shortly, also, where relevant, making reference to the literature.

Figure 2. Visual impression of the tool [Source: owned by the Authors].

• Constraint #1: Citizens lack financial, intellectual and temporal resources to participate in the experiment

Some stakeholders may lack financial, intellectual and temporal resources to participate meaningfully in the experiment [52,53,54,55,56]. To participate well, citizens need time, energy and commitment, a certain level of understanding of the issue at stake or of the technology in use, and sometimes also specific economic and intellectual resources or skills. Certain social groups may therefore tend not to participate in the experiment. The flip side of this card suggests three ways in which this risk may be anticipated. Stakeholder requirement analysis and requirement tools may be applied (in relation to desired outcomes of the experiment), to identify who runs the risk of being excluded, their motivations and possible coping strategies. Also, it can help to include all Lab participants’ reflections on who should participate (not only those of the initiators). Finally, the practicalities of the experiment, such as informative and educational material, choice of venue and schedule of meetings, language, and provision of technological support to reduce the digital divide, should be designed strategically [56].

• Constraint #2: Relevant stakeholders remain out of the experiment

Next to the risk that relevant stakeholders are not able to participate, there is the risk that they do not want to participate [52,53,54,55,57]. Certain groups might not be interested in joining the experiment’s activities, since they do not share the urgency to discuss the issues at stake and take action, or even have conflicting attitudes or goals. The experiment may thus run the risk of becoming a low-conflict circle of people sharing priorities, attitudes and goals, while the large majority of citizens would ignore it. The flip side of this card suggests how this risk could be anticipated. As with the previous constraint, stakeholder analysis can be applied to identify the relevant target groups and the reasons why they might/might not be interested in joining the experiment’s activities. This also has implications for how to frame the experiment’s activities in public communication campaigns. This should be aimed at recruiting participants and identifying the specific actions needed to also raise the interest of less intrinsically motivated target groups.

• Constraint #3: Groups and impacts outside the experiment context are overlooked

The experiment may lack or be poor in representatives from the larger urban context, although these might be impacted by the project [52,53,54,55,58]. Likewise, effects beyond the experiment’s boundaries may be neglected (e.g., a decrease in cars in one district shifts traffic to another). This creates the risk that lessons are biased and myopic to the locality of the experiment. The flip side of this card suggests a few ways in which this risk can be anticipated. First, one should explicitly consider the project’s indirect and cross-scale effects in the broader urban context by reflecting on the multiple scales relevant to the experiment (e.g., adjacent streets, adjacent neighborhoods, and other relevant parts of the city). This also involves reflecting on the actors that might be included/excluded on each scale. Second, one should adopt adequate logistic arrangements and outreach strategies to help minimize exclusion, such as convening the experiment’s meetings at different locations and being open to reframe meetings to align with the purposes of the experiment and increase motivation.

• Constraint #4: Existing power structures are reproduced inside the experiment

The setup of the experiment and applied methods may not guarantee that any group or participant has equal opportunities for participating in the discussion, so that every voice is heard and seriously taken into account [58,59,60]. For example, the mayor, technical experts, or simply male participants of the experiment may be given more weight than other participants. The flip side of this card suggests two ways in which this risk may be anticipated. First, one could regularly perform a stakeholder group dynamics analysis in order to understand group structure and leadership relations among group members. Particularly, identify any dominant position among the experiment’s participants due to already existing institutional roles outside the experiment (e.g., political responsibility and lobbying activity). Second, design a communication and management strategy to address all identified target groups, keep flexibility, favor the development of activities along different tracks, allowing each group to adapt to their speed of progress.

• Constraint #5: The experiment’s potential for learning is underexploited

In case the lessons offered by the experiment’s activities are not explicitly monitored, the resulting understanding of their implications may be small and remain limited to a number of participants [1,52,57,58]. In this case, only limited transfer of learning is possible, thus constraining the wider implementation of the innovation. The flip side of this card suggests two ways in which this risk may be anticipated. First, jointly develop a comprehensive learning strategy (i.e., a set of explicit learning questions) aimed at creating focus on what the experiment intends to learn as well as enabling the capturing of the lessons (i.e., as formulating explicit answers to the questions), which in turn enables the transfer of lessons to all relevant actors outside the experiment. Second, make sure that sessions in which lessons are formulated are physical meetings (i.e., people-to-people, real-life interactions, because these make learning more rewarding and comprehensive to all and also ensure tacit knowledge to emerge.

• Constraint #6: The experiment is disconnected from broader societal debate

The experiment may lack coordination with the social, economic, cultural, and political conjuncture. In such a case, the policy climate may not support the adoption of the innovation pursued in the experiment. The broader public may either not share the experiment’s goals and outcomes or find them irrelevant [58,61]. Provided ways to anticipate this constraint are: (1) Design and manage the experiment’s activities with great care for the local context: Consider broader socio-economic, cultural and political aspects, ensure links with the existing public debate, with what a community considers to be its priorities, and what stakeholders consider to be feasible. (2) Maintain a certain flexibility throughout the experiment, be ready to adapt to changing conditions in the outside social and political agendas. Ensure that both the experiment’s objectives and its framing can be adjusted and continuously re-defined with all actors. (3) Place citizens at the core of the process and actively coordinate with other societal developments and initiatives related to the content of the experiment.

• Constraint #7: The ‘co-created’ experiment’s result is not reflected in policy and society

Even if the topic addressed by the experiment is a priority in the social and political agendas, the persistence of conflicts on specific topics may hinder reaching agreements, either inside or outside the experiment. The outcomes of the experiment may therefore lack wide consensus, support and political majority [58,62,63]. The flip side of this card suggests three ways in which this risk could be anticipated. First, open to participation as much and as early as possible and regularly update the stakeholder analysis whenever external conditions change, in order to avoid the exclusion of any relevant stakeholder group. Second, favor the emergence of any conflicting goals within the experiment’s participants and between participants and possible external stakeholder groups not actively engaged, and manage conflicting goals by multi-criteria decision-making techniques. Third, always emphasize and give weight to potential community-level benefits of the options under discussion, against personal or partisan benefits. To this purpose, exploit already existing networks and coalitions and seek new and unexpected alliances between groups of stakeholders, trying to build relationships with successful initiatives already developed by other actors.

• Constraint #8: Stakeholders and institutions are highly fragmented

Fragmented institutional arrangements between and within institutions (‘silo compartments’) may preclude clear distribution of responsibilities among the actors involved in the experiment’s activities, and effective cooperation between them [58,62,64]. Even when some policymakers embrace the experimental approach, further implementation of its outcomes might suffer from this. The backside of this card offers two ways to anticipate. First, foster transparency and collaboration between administrative units, organizations and stakeholders, right from the beginning of the experiment’s process. Second, create occasions for them to interact and become familiar with the process, discussion topics and proposals emerging within the experiment.

• Constraint #9: The urban assemblage is sticky and locked-in

Technical, infrastructural, legal, or financial aspects, such as long-term contracts or legal lock-ins, may cause obduracy of the urban assemblage, thus precluding possibilities for practical implementation of the outcomes of the experiment [62,65,66,67]. In such cases, decisions need to be taken by multiple stakeholders or entities on a political level, sometimes only in particular time periods, and cannot be attached to the outcome of a participatory, experimental process only. Two ways to anticipate this constraint are suggested on the backside of this card. (1) Activate a dialog with relevant actors as soon as possible: by explicating ideas on the future and, where possible, aligning future visions with stakeholders and crucial decision-makers, the potential of more structural changes can be highlighted. (2) Local actors might be empowered by teaming up with supra-urban actors, such as municipalities with provinces or local NGOs with their national counterparts, to increase the chances of sufficient (financial and other) resources for more structural changes.

• Constraint #10: The experiment meets low institutional receptiveness

Local governments and other actors involved in the experiment might be unfamiliar with, or not so open to, co-creation approaches, favoring instead expert-driven ways of thinking and agreement with powerful lobbies. If so, institutions may not have a real commitment to implement the experiment’s outcomes [58,61]. The flip side of this card suggests four ways in which this risk could be anticipated. First, seek the early inclusion of policymakers and local institutions, so that there is time to learn about the nature and rationale of the approach. Second, make sure that organizers of an experiment show genuine commitment and give voice, role, and responsibility to diverse groups of citizens, civil society organizations and experts, so that skeptics might start appreciating the approach and its benefit. Third, carry out multiple successful pilot processes, so that one can build further on positive experiences (if any). Fourth, build on existing practices and procedures of representative democracy to promote dialog between stakeholders, so that skeptics can see the connection with trusted processes.

3.2. How the Tool Evolved During Design Iteration

Section 2 already described the design process of the tool, from a first version based on literature, a second version based on action research in four cities (‘Iteration 1’), to trials on three cases (‘Iteration 2’) and two more tests of the final tool (‘Iteration 3’). Table 3 gives a concise insight into how the lessons from the various evaluations (‘Iterations’) shaped the tool.

Table 3. List of adaptations made based on evaluations throughout the DSR process.

Iteration	Changes	Rationale
#1	From 13 generic constraints to 10 Based on action research in the four cities of Bellinzona, Brussels, Graz and Maastricht (see Section 2 for more details)	The intervention in Bellinzona confirmed the relevance of constraints #1, #2, #5, and #10. The intervention in Brussels confirmed the relevance of constraints #1, #2, #6, and #10. The intervention in Graz confirmed the relevance of constraints #2, #4, and #9. The intervention in Maastricht confirmed the relevance of constraints #3, #8, #9, and #10. The earlier 3 additional constraints were merged into the final 10. We decided to keep constraint #7 for the moment, although it was not applied in the four cases.
#2	Adapt the final wording of the cards Based on three trial sessions in Helsinki, Istanbul, and Santander	We found out that constraints play out more severely in a governance context that has adopted little or no experimental approaches (Istanbul), while constraints are less severe in the opposite case (Helsinki), with Santander being somewhere in the middle. Nevertheless, when formulated in generic ways, all constraints were considered applicable, and when presented as a ‘checklist’ were considered useful as pointers and reflective tools.
#3	No further changes Based on two trial sessions in Amsterdam and Rotterdam	The card-deck design of the tool was found attractive. All 10 constraints (+ suggestions) were found useful. As one civil servant noted, the tool helped to reveal some strengths and areas for improvement around scaling up our hub experiment in a residential area.

4. Discussion

4.1. Contribution to Practice: Typical Effects of the Tool

Based on our qualitative demonstration, evaluation and adaptation of the tool with practitioners, we can state that in general, the resulting anticipatory guidelines help practitioners with reflecting on upscaling and deciding on further implementation better. The nature of ‘further implementation’ can be rather diverse. In Brussels, the methodology helped to anticipate constraints #1, 2, 6 and 10 in particular. Efforts to involve stakeholders that would otherwise be unable or unwilling to be involved were included through various meeting formats, locations and timings (#1), as well as framing their topics in terms of air quality instead of mobility (#2). Despite a lack of participation from civil servants, local politicians were contacted and successfully involved (#10), mainly due to upcoming elections, and accordingly connected to broader debates (#6). The new (participatory measured) air quality data, in addition to the few official measurement stations, stirred new debates. This, and by joining up with a series of other similar (air quality-related) projects, was effective in triggering a broad civic mobilization around the claim for better air. A few months after the intervention (in April 2019), the project participants and other stakeholders organized the Etats Generaux de l’Air de Bruxelles, combining an international research symposium, a hackathon, and different citizens and civil society activities. The initiative brought together different actors striving for a cleaner air in Brussels. It provided a platform for dialog and collaboration, and discussed visions and solutions to realize a healthier city. By bringing together 300 people with all sorts of knowledge about air pollution in Brussels, by developing coalitions and participatory visioning for the city, and by contributing to democratize science and urban governance, the event—and possible follow-ups—can be considered the most tangible upscaling impact amongst the four cases. (At the same time, it remains impossible to establish clear ‘cause–effect’ relationships here, and we do not claim this. However, a range of conditions for upscaling were clearly created by activities in which the tool played a role.)

In the other three cities, the effects of anticipating constraints were also instrumental, although less tangible and more mixed. In Maastricht, application of the tool helped to anticipate constraints #3, 8, 9 and 10. The geographic scope became much broader than just the station area (#3), while the longer temporal scope helped to imagine more fundamental changes (#9). The sessions brought stakeholders together (i.e., in dialog) that would otherwise only be talking with the municipality bilaterally (#8). The intervention with the tool (self-reportedly) provided the municipality with more arguments in favor of a larger car-free area. More specifically, it highlighted the incompatibility of ‘maintaining car accessibility’ versus ‘more space for car alternatives’. The municipality used this in the Mobility Vision for 2040 (issued in the year after the intervention), which included a target to reduce car mobility with 10% by 2030, something without precedent (it was always ‘no further growth’). However, it was unclear whether or what other stakeholders learned from the project.

In Bellinzona, anticipation focused on more attention and efforts to involve stakeholders otherwise unable or unwilling to be involved (#1 and #2), in a governance context that was not so familiar with participation, let alone experimental approaches (#10). For a number of local civil servants, this intervention provided a significant positive experience with this novel way of working together (attesting ‘it inspired me to rethink our working methods’ and ‘seek more direct dialog with citizens’). Also, a range of citizens in Bellinzona said they ‘experienced a new way of being involved in policymaking’: the intervention strengthened reciprocal trust between citizens and policymakers, opening up possibilities for adoption of participatory practices also in future decision making [68]. The whole process, and particularly the final focus group aimed at evaluating the usefulness of the tool, opened up novel possibilities for learning from initiatives activated at the city level, to support broader transformation in urban governance (#5).

In Graz, the application of the tool helped to anticipate constraints #2, #4 and #9. The broader types of meeting formats and communication channels helped to involve more stakeholders than otherwise would have been willing to join (#2), and a neighborhood ‘living room’ leveled the playing field between municipality and stakeholders in the neighborhood (#4). It helped to collect numerous ideas for the redesign of the public square, and fostered social interactions among residents, and contributed to a more positive attitude about their future in the urban area of the experiment in general.

4.2. Ways of Using the Tool

The tool seems to work best in a dialog form, so that a practitioner is questioned by a moderator about their experiment. This way, the moderator can give any necessary clarification or explanation concerning the generic constraint, and can help explore the relevance and application of it to a particular case. As one participant in the development activities noted, you can also work with it alone, but then it will probably yield less.

The tool is primarily intended to help make next steps (i.e., after an experiment) better, but for doing this, it may be useful to apply the tool to both upcoming and recent experiments, since they both can create useful lessons on specific local constraints. Although found useful, it is also clear that these guidelines do not lead to ‘control’ or ‘manage’ an upscaling process. As Van Poeck et al. [69] note, monitoring learning is critical; however, it will remain impossible to establish ‘cause–effect’ relationships or predictions between micro-learning practices today and longer-term, macro societal transitions. The multi-actor and complex nature of the issues dealt with does not allow such an alluring idea. We believe ‘anticipating’ is the most one can do to enable the upscaling process to unfold. In order to trigger and govern longer-term transformations, experimentation generally needs to go hand-in-hand with adaptations of the established policy mix, at multiple governance levels. However, this was outside the scope of our study, and accordingly, we recommend that future research include studying the connection between experiments and the policymaking process.

The guidelines are generic and should be seen as a ‘checklist’ to consider which of the constraints are applicable in the case of a particular project. Hence, there is a need to identify the most relevant context-specific constraints. The guidelines are formulated as suggestions (i.e., suggested ways to anticipate) and are (necessarily) in general terms. Which of the suggestions is most applicable, and how exactly, needs to be determined on a case-by-case basis. Clearly, the devil is often in the details here.

4.3. Contribution to Science

The contribution to current transition studies is twofold. First, we developed a tool that is more than current approaches focused on analyzing and confronting the experiment’s surrounding socio-institutional context. Approaches like strategic niche management and transition management have been proposed, but these have, respectively, been more focused internally on the experiment or more future-vision oriented [18], and less focused on analyzing and confronting the established socio-institutional context. Our tool seeks to prevent bias in niche-internal processes or future endpoints and frontrunners, but balancing these with constraints in the current context. The wider implementation of innovations is usually constrained by a range of factors and structures, typically an interconnected, heterogeneous, and ‘obdurate’ context of infrastructures, regulations, actor practices and interests [21]. Our approach aims to include a broad range of stakeholders, including both powerful established stakeholders (which may have a strong interest in lobbying against particular innovation) and more powerless others. This is much broader than transition management, which proposes to bring together ‘frontrunners’, hence stakeholders with high ambitions for a particular transition. TM’s approach implies more consensus during the experiment, entailing fewer conflicting views, less politics, and hence less learning on (the politics surrounding) upscaling the tested innovation. We believe the improved learning about the politics of wider implementations and upscaling of innovation is a key merit of our approach. At the same time, our approach is maybe less ambitious than Transition Management, in the sense that it does not assume an agreed ‘transition’ or an end-point of a fundamental shift, but supports finding (only) a next step in a particular direction.

Second, we demonstrate a design science research method in a study involving practical, transition-related interventions, which make (existing) knowledge more ‘actionable’, which is novel for the transition field [70]. Zolfagharian et al. [34] find that while current transition research is relatively strong in explaining past transitions and case studies, it seems less strong in methods to explicitly design and test practice-oriented interventional knowledge. They flag this as an important avenue for future research to enhance the methodological rigor and richness of transition studies. Extending the methodological toolbox beyond primarily qualitative process theories might lead to a better understanding of possible intervention strategies—and as such, greater policy impact. Our design science method in this paper, with distinctive underlying assumptions and epistemologies, resulted in insights into the effect of a particular tool on various places. This avenue for research has the potential to increase the practical impact of transition studies on sustainability.

As Wiegman et al. [70] discuss for their case in geothermal energy, the designed tool functions as a boundary object between theoretical and practical/applied knowledge. That is, it mediates between more generalized theoretical knowledge and knowledge that is adapted, made ‘actionable’, to the specific context, which is usually required in practice. We illustrate this concerning the socio-political processes surrounding the experiments in our study.

In Maastricht, the format of the participatory visioning exercise facilitated equal contribution from participants: from businesses, to residents, to the municipality. This was a novel situation, especially for the businessmen, who were used to talking to the municipality bilaterally: they were irritated by the views of residents (qualifying these in post-interviews as ‘fully unrealistic’ and ‘dreamers’), and the feedback from external practitioners (which they qualified as ‘unfounded’ during the workshop). We conclude that the equal position that stakeholders were put in was new for Maastricht, and for this was irritating for some (businesses), while for the municipality, this was reported afterwards as ‘refreshing’. We observed that they integrated various ideas, and they (self-reportedly) learned more arguments in favor of a larger car-free area.

In Brussels, the new (participatory measured) air quality data implied the empowerment of citizens in the air quality debate, bringing in a new type of data (i.e., data at the personal level), in addition to the few official measurement stations. The politics is interesting here: although the administration initially refused to take part in the experiment, the organizers seized the opportunity of local and regional elections to enter into debate with (eligible) politicians and civil servants. This provided, after all, some entrance to the policy development process of the municipality and certainly widened the attention in the public debate.

The app development process in Bellinzona empowered citizens to be co-designers of a mobility app. Forty-six citizens answered the public invitation to join the experiment, and fifteen of them were regularly active in the seven monthly meetings held. The municipality engaged with local businesses to allocate prizes (or ‘points’) for choosing car alternatives. Despite the ample number of referenda in Switzerland, such co-creating processes between civil servants, citizens, and other organizations are relatively scarce. The whole process enabled a cultural change among some local policymakers: via the creation of reciprocal trust between them and other relevant local actors, it supported them to evolve from the role of centralized controllers and city regulators to the role of facilitators of urban decision-making processes, re-balancing previous asymmetrical power relationships.

In summary, the developed tool shows in different ways what Moulaert et al. [71] call ‘empowerment of stakeholders’ and ‘changing relationships’ through its application in the interventions. Applying the tool facilitates reflections on upscaling which stimulates learning on the politics of wider implementation. Notably, we have not argued that every experiment should be scaled up per se; there may be good reasons not to further implement an innovation, so lessons may work out both ways. Also, we did not want to imply that every (even inclusive) upscaling process leads to more sustainability. The (sustainability of the) impact of an innovation, when scaled up, needs a specific assessment. It is significant that the (sustainability) criteria applied in such an assessment are determined in an ‘open’ process, that is, are set regarding the various stakeholder perspectives as well as relevant other expertise. Inclusion of relevant stakeholders is not only important in the phase of preparing and doing the experiment, but also in the evaluation of it. Only after a positive evaluation, further upscaling should be pursued. This implies that the tool we developed is only a starting point, and more work is needed concerning the evaluation of the effectiveness of the tool, concerning monitoring of learning, monitoring of sustainability and equity impacts, and of (changing) stakeholder relations and relative power.

4.4. Limitations and Points for Improvement

A first limitation is that in this paper, we described the ten constraints only concisely. However, each constraint relates to various scientific debates, on which books have been written. We could not do full justice to what earlier literature has debated on them. The strength of this tool is that it brings together many different relevant issues, making them ‘actionable’, but the limitation is that we could not discuss the (10) different issues (and how they played out in our action research) in much depth here. We leave this for future studies (that may not develop tools).

Second, our evaluation analysis in the action research projects in the four cities was based on short-term observations and accounts of participants, based on applying the tool. However, longitudinal analysis of potential upscaling and drivers over several years would provide a sounder understanding of effectiveness. Current observations of effects (Section 4) were more concerning ‘conditions for’ upscaling than for upscaling itself. Future research might also differentiate constraints concerning the two dimensions of upscaling (expanding and stabilizing). In addition, the analysis of effects was sometimes challenged by participants tending to be primarily engaged with the topic of the experiment, less with the tool we were developing, and how it performed. This caused the analysis to be sometimes based more indirectly on the discussion in focus groups and interviews. In addition, coded analysis of interviews and focus group transcriptions would have raised transparency in our analysis. Still, also because of the collective nature of our analysis (i.e., the five authors integrating the findings of their home case, while doing trial sessions together), we are confident that the final form of the tool includes the most relevant constraints and ways to anticipate.

Third, the high diversity of the experiments and the size of the overall nine cities was helpful to learn about how the tool performed in different contexts, and allowed for integration (or ‘triangulation’) of the results from the four experiments. On the other hand, the high diversity of cases prevents the possibility of a sharp cross-comparison of the cases. More tests of the tool beyond the ones in this paper may help to further refine the guidelines for better use in widely different contexts and cases.

Finally, our study argues that one should not naively assume that all experiments are fair, democratic, and inclusive by default because of their co-creative aspirations (see also [72]. Elements inspired to these principles need to be actively designed and monitored. In doing so, it is necessary to assess the various activities and potential impacts of an experiment on their procedural justice (i.e., fair, equitable, and inclusive procedures for those participating), distributional justice (i.e., distribution of costs and benefits across populations), and recognition justice (i.e., acknowledgement of past and present inequalities) (see [73]. Although our study addressed the first two aspects (through constraints 1–2 and 3–4, respectively), it has neglected this third aspect (i.e., recognition justice), and this is something we recommend for future research and development with this tool.

5. Conclusions

In this study, our research question was: how can urban professionals involved in experiments be supported to learn more about the most fruitful follow-up of an urban experiment to contribute to sustainability transformation? This paper has presented a tool to support urban experimenters to learn more about the most appropriate ‘next step’, by anticipating constraints on ‘upscaling’ of innovations under experiment.

We have formulated an anticipative tool for better upscaling urban experiments, consisting of ten guidelines. We developed and tested it in action research of four European cities, and five further testing sessions in other European cities, which delivered the final tool. We suggest using the tool in a dialog form, namely via a moderator questioning a practitioner about a particular experiment. This helps to identify which constraints on upscaling are the major ones. Constraints are clearly case and city specific and thus need to be identified through a specific analysis of each local situation. More tests of the tool beyond the ones in this paper may help further refine the guidelines for use in a range of different areas and cases. Future research may involve more longitudinal analysis of potential upscaling to provide a sounder understanding of effectiveness, but also meta-studies to explore the identification of typical constraints in particular geographical areas, or various types of urban transformation pathways, which possibly could lead to further specification of guidelines for particular types of upscaling or innovations.