Urban Living Labs and Citizen Science: From Innovation and Science towards Policy Impacts

Abstract

City governments are currently exploring different participatory mechanisms in order to meet the growing demand in society for deliberate decision-making. Through developments in sensing technology, data processing and visualization, citizen science is emerging as a powerful tool for the general public to participate in scientific research that informs policy. Citizen science can be used as an approach in Urban Living Labs, whereby public and private stakeholders are involved in innovation and data collection processes together with citizens. However, there is currently little synergy between citizen science and Urban Living Labs, and how science, innovation and policy can be interoperable. Therefore, this article conducts a comparative case study analysis on the participatory processes and related outputs, outcomes and impacts of the FloodCitiSense Urban Living Labs (Brussels, Birmingham, Rotterdam). These initiatives developed an early warning system for urban flooding through the collection and analysis of crowdsourced information. Data on the participatory processes were collected through 11 in-depth interviews and evaluated on its effectiveness in achieving policy outcomes. The discussion reflects on best practices in incorporating citizen science in Urban Living Labs based on the experienced opportunities and challenges in FloodCitiSense.

1. Introduction

Nowadays, cities are facing complex and interrelated societal issues leading to economic, environmental and social crises, also referred to as “wicked problems” [1]. These challenges emphasize the need for innovative solutions that take into account new sources of knowledge [2,3]. In this context, recent technological evolutions such as social media and sensing applications have been used in innovative ways to co-create knowledge and solutions at a pace that was not conceivable before [4]. These technology-enabled ‘smart’ systems offer opportunities to tackle societal problems faced by cities, which have led to the concept of “Smart Cities”. Although the concept of Smart Cities has originally focused on technical components and goals, i.e., the usage of “smart technologies” to address social, economic and environmental problems in a more efficient manner [5], the focus is slowly shifting towards a citizen-centric approach [6]. In this regard, authors such as Caragliu et al. indicate that what makes a city “smart” is the usage of participatory processes [7].

Participatory processes can take different forms depending on the field of application, the methods and the envisioned goals. In a Smart City context, citizen participation in urban governance is referred to as participatory governance or “smart governance” (e.g., [8]). Specifically, it is defined as “the processes and structures of public policy decision making and management that engage people constructively across the boundaries of public agencies, levels of government, and/or the public, private and civic spheres in order to carry out a public purpose that could not otherwise be accomplished” [9], p. 2. Participatory governance offers citizens an opportunity to participate in matters that affect their daily life and to interpret and propose solutions to their problems [10]. Citizen participation in governmental processes is regarded as beneficial [11] and would help to promote democratic values such as social justice, legitimacy and effectiveness [12]. In order to generate public value, participatory governance requires appropriate sets of practices and methods. A plethora of methods are currently available from citizen surveys, focus groups, public meetings, deliberative polling, televoting, study circles, scenario workshops and citizen’s assemblies [13].

This article focusses on two specific modes of participatory governance, namely Urban Living Labs (ULLs) and Citizen Science (CS). Although different in their primary aim–(urban) Living Labs (LLs) are aimed at open and user-driven innovation [14] while CS initiatives are aimed at acquiring new scientific knowledge on a specific phenomenon [15]–both approaches include practices of direct deliberative engagement with citizens. Accordingly, CS is positioned on the side of identification and understanding of a specific phenomenon, while (urban) LLs are rather on the side of designing solutions to a problem. However, the democratization of science [16] and innovation [17], or the shift towards a more equitable engagement between experts and the lay public, are clear virtues of both approaches. Further, in the creation of a Smart City vision, the synergy of both approaches appears beneficial [18]: CS can help in developing and accumulating the skills of citizens (e.g., [19,20]), while ULLs can help in the ability of sharing and exchanging data and services across different urban platforms (e.g., [21,22]).

Taking the stance of governance theory, this article analyses and compares the participatory processes and related outputs, outcomes and impacts of the FloodCitiSense ULLs in Brussels, Birmingham and Rotterdam. The purpose of this comparative case study analysis is to provide comprehensive and clear descriptions of how CS can be incorporated in an Urban Living Lab (ULL) setting, and to review the effectiveness of the two modes of participation in achieving policy outcomes. As such, the comparative diagnoses of the three cases will improve our understanding of which participatory processes or combination of elements contribute to policy outcomes. The FloodCitiSense ULLs aimed to develop an early warning system for urban drainage flooding for and by citizens. The system was co-created and tested through the living lab (LL) methodology in three ULLs: Brussels, Rotterdam and Birmingham, whereby citizens helped to collect rainfall data and flood reports. Data about the participatory processes were collected through 11 in-depth interviews with various stakeholders (public–private–people) from the ULLs. The results are structured along the framework for the Comparison of Participatory Processes (COPP) [23]. This framework provides a theoretical foundation for the description and comparative analysis of participatory processes, which can be used for monitoring and evaluating both processes and outcomes. There are three main assessment dimensions in the COPP framework: (i) context, (ii) process, and (iii) outputs, outcomes and impacts. For each ULL, the context elements are described, while the process and outputs, outcomes and impact dimension focus on the CS and ULL approaches. In the discussion, best practices of incorporating CS in ULLs are described based on the experienced opportunities and challenges of FloodCitiSense. Further, a critical stance is taken towards the necessity of integrating both modes of participation to achieve policy outcomes. As such, the article aims to provide practical insights for practitioners in participatory governance who aim to connect science, innovation and policy objectives.

2. Citizen Science: Participation for Scientific Knowledge Creation

CS is considered as an emerging trend; however, examples date back from the 19th century whereby citizens without formal training in science carried out observations [24]. At that time, amateurs explored phenomena from their living rooms, since being a scientist was not yet considered as a formal profession and laboratories did not yet exist. Today’s modern citizen scientists are equipped with technologies, such as smartphone applications and sensors, and are often guided by professional scientists. Contemporary CS is labelled as a form of participatory action research [25] and defined as “scientific work undertaken wholly or partially by members of the public, often in collaboration with or under the direction of professional scientists” [26]. The participation of citizens in science can happen to varying degrees and in various phases of the scientific research process. CS projects are generally divided into three categories from least to most participatory: (i) contributory, whereby citizens collect data; (ii) collaborative, whereby citizens collect, analyze and help disseminate the results to some extent; (iii) co-created, whereby citizens design the project together with scientists and are involved in all phases of the research process [27].

Within the field of environmental management, participatory processes have been identified as particularly interesting in order to fill in the knowledge gap and counter inadequate or incomplete datasets of professional scientists and government agencies. More than in any other domains, CS appears as specifically promising to generate knowledge and to raise awareness about particular issues and policies [28]. Initiatives focusing on environmental monitoring have emerged, harnessing the power of Community Based Monitoring (CBM) to better understand phenomenon such as soil quality (e.g., [29]), urban air pollution (e.g., [30]) and urban flooding (e.g., [31]). These projects are also labelled as “Citizen Observatories” and are defined as “the citizens’ own observations and understanding of environmentally-related problems, and in particularly as reporting and commenting on them” [32], p. 4. The particular benefit of CS and Citizen Observatories for environmental monitoring is that such projects allow for large-scale collection of data, offering both high frequency and large geographical coverage [33]. The engagement of citizens in science is also seen as an effective way to link citizens with experts and policy makers [33,34]. Benefits from this collaboration include: democratic legitimacy, accountability and transparent governance, trust building amongst citizens and institutions, understanding of societal concerns, creation of innovative strategies, generation and evaluation of new policies, community empowerment and awareness raising [33,34].

Although CS is first and foremost seen as a way to collect data and gain scientific knowledge on a specific phenomenon, it is increasingly being recognized as a powerful tool in governance and decision-making processes [35]. Actors from policy spheres are starting to recognize the benefits of CS as a timely and cost-effective source of knowledge that can support policy implementation and complement official reporting [33]. More specifically, Göbel et al. [35] described CS as feeding into, being affected by, forming part of, and exercising four type of governance: (i) a source of information for policy-making–refers to the fact that CS’s outputs are being used to formulate policies, (ii) an object of research policy–here CS is being regulated through policies to advance research, technology and innovation, (iii) a policy instrument–CS is seen as a disruptive innovation, instrumentalized by policies to reach their goal, and (iv) socio-technical governance–CS is seen as a direct type of governance via non-policy actors, answering societal problems through prototypes developed by citizens. Specific to environmental monitoring, it has been demonstrated that environmental CS projects can contribute to a diversity of sustainable development goals (SDGs) and can contribute to different phases of the policy cycle [36]. CS can contribute to problem definition, early warning, implementation and evaluation of and compliance with policies. Through making valuable and systematic observations, CS can help in identifying problems and issues at an early stage (e.g., declines in species frequencies) and help in providing background data for establishing restrictions or targets (e.g., peaks in plastic pollution levels).

However, the actual impacts of CS on policymaking are still difficult to demonstrate [33]. Policymaking remains largely based on experts’ inputs with little room for lay knowledge coming from citizens [34]. Since citizen engagement is not worthwhile if their contribution is not integrated in the stages of policymaking [34], several challenges must be confronted. First, institutional and organizational culture must shift to fight resistance in acknowledging the value of CS [34]: mechanisms enabling citizens to impact evidence-based processes for policymaking have to be put in place [33]. Ideally, a connection should be sought between the research processes of CS initiatives and the policy cycle [37], not only in the funding but also in the design and implementation of the project. The benefits of connecting with established policymaking processes are that the evidence base would become more complete, strengthened in its scientific approach and could attract more attention and higher acceptance from the public. Further, public institutions have to carefully consider resources and manage expectations, without turning CS into a top-down governance tool [33]. Other key challenges are related to the lack of confidence in CS data by policy makers, the appropriate data scalability, and an easy and straightforward data protocol for engagement processes [36].

From any aspect, the potential role of CS for policy making is still to be further explored and can benefit from an enhanced understanding of the CS and public policy interface. In the next chapter, the second mode of participatory governance is described, namely (Urban) Living Labs, as an approach to foster innovation and co-creation in urban governance.

3. Living Labs: Participation in Open and User-Driven Innovation

The concept of “Living Labs” appears as a buzzword for a multitude of activities in the innovation domain. Its roots can be traced back to open innovation, whereby LLs are seen as a form of open innovation [38] or open innovation network [39], and to user innovation paradigms [40,41]. However, to date, there is no clear integration of the concept of LLs in the open innovation literature stream. Most of the open innovation literature focusses on the acquisition of knowledge from external sources by firms, often in a business-to-business context, whereas LLs operate through public–private–people partnerships [40]. Within the literature stream of user innovation, LLs are placed as a methodology between user-centered design and participatory design [42]. Furthermore, co-creation is regarded as a core activity in the LL and takes place with its diverse stakeholders and users [40].

One of the earlier and most-cited LL practitioners in Europe defined LLs as a “user-centric research methodology for sensing, prototyping, validating and refining complex solutions in multiple and evolving real-world contexts” [43]. Nowadays, the European Network of Living Labs (ENoLL) defines them as “user-centered open innovation ecosystems based on systematic user co-creation approach, integrating research and innovation processes in real life communities and settings” [44]. More emphasis has been put on the network or ecosystem aspect, whereby a distinction can be made between three different levels of LL phenomena: (i) the macro or organizational level, (ii) the meso or project level and (ii) the micro or user activity level [45]. On the meso level, the LL methodology is related to the different stages of an innovation development process, namely through the stages of exploration, experimentation and evaluation [46]. During these stages, the innovation develops from an idea towards a concept and eventually a prototype, which is tested and benchmarked. In the final phase, a post-measurement is executed in order to assess the impact and added value created by the innovation. It is also the stage where exploitation activities take place [46].

A multi-stakeholder participation is a key characteristic of a LL ecosystem [40]. LLs operate as an intermediary environment whereby joint-value creation is supported. The innovation ecosystem can be formed by partnerships which may include prospective user groups, companies, research institutions and universities, local authorities, etc.. The formation of these public–private–people partnerships (PPPP or 4Ps), also known as Quadruple Helix formations, can yield potential advantages, such as greater public involvement, transdisciplinary research, creativity and knowledge exchange [47], and also influence public affairs and policy innovation [48,49]. More specifically, in parallel with CS, LLs can also exercise different types of governance: (i) as an input for direct governance, LLs are used by policy makers to design, explore, experience, and refine new policies and regulations [40], or (ii) as a policy instrument for enriching regional innovation [50].

Urban Living Labs for Sustainable Development

In the pursuit of realizing ‘smarter’ cities, LLs are helpful experimentation environments for solving wicked urban issues [51]. ‘Wicked problems’ are problems difficult or impossible to solve because of their incomplete, contradictory and changing requirements [52]. A number of approaches have been identified in the literature for solving these wicked problems, such as taking a place-based approach, enabling coherent action by diverse actors, involving users as co-creators, supporting a networked governance approach and recognizing the government as an enabler of change. These approaches towards wicked problem solving are inherent characteristics of a particular type of LL [53], namely the Urban Living Labs (ULLs). ULLs have a particular focus on the generation of public value with a place-based focus, often a specific urban site or city, and aim to deliver innovative and transformative improvements across the urban milieu [54]. In this regard, ULLs are showing particular promise in urban environments to tackle sustainability challenges, such as energy efficiency, food poverty, waste management, urban flooding, etc. [55].

Although ULLs have the basic principles of co-creation of innovations in common with traditional LLs, Chronéer et al. [56] found that ULLs tend to place a greater emphasis on the governance model. The activities carried out by the initiatives must be supported by policy makers in order to be sustainable, both at the managerial and financial level. An ULL will thus have a more prevailing political dimension than a traditional LL. Another distinguishing characteristic is that city representatives are often the enabler of the initiative, creating a vision and providing strategic leadership whereas in traditional LLs this role is often filled in by research institutes or organizations [51]. Therefore, the presence of public–private–people partnerships are more prevalent than in traditional LLs, leading to closer collaborations between public administrations and universities [56]. Hence, the role of users is also somewhat different and reveals itself rather as citizens being the co-constructors of space. Citizens co-construct the urban space, either supported by an effective usage of a new innovation or not, or rather they co-construct the experience of the outcome [56]. Finally, another distinguishing characteristic is the utilization of information and communication technologies (ICTs) for both technical testing and collection of data. In this manner, the usage of technologies is inherently part of the methodology, such as the collection of data through sensors or the development of apps [57].

Due to its multi-stakeholder participation, focus on co-creation of the public space and usage of ICT and technologies, ULLs form an ideal experimentation environment for tackling sustainability issues. CS has been often used as a tool for environmental monitoring of particular urban issues, raising awareness and informing policy making in ULLs (e.g., [58]). Citizens are motivated to participate in both innovation activities as well as in decision-making processes, directly interacting with stakeholders from the private and public sphere. As such, ULLs can be regarded as a platform for change, rather than a methodology in themselves. In the case of flood risk management, which is the particular focus of this article, the European Flood Directive 2007/60/EC requires public participation mechanisms to ensure citizens’ involvement in decision-making processes. Through CS, citizens can contribute to mapping and assessing flood hazards at various temporal and spatial scales, which in turn can help to improve prediction models [59]. However, the implementation of a CS approach in ULLs is relatively new, and often results in little synergy due to a lack of interoperability and reusability of data and services developed in each project [60]. In the next sections, we review the practical experience of the FloodCitiSense project and explore how the two different modes of participatory governance can work towards concrete policy outcomes.

4. Materials and Methods

This article analyses and compares the participatory processes and their related outputs, outcomes and impacts of the FloodCitiSense ULLs in Brussels, Birmingham and Rotterdam. Through a comparative case study analysis [61], the implementation of the CS approach in the ULLs is analysed and compared in order to review the effectiveness of the two modes of participatory governance in achieving policy outcomes. As such, the comparative case study analysis provides comprehensive and clear descriptions on how to incorporate CS in ULLs and how the combination of these elements can contribute to policy outcomes. Since the implementation of CS in ULLs is still relatively new, a case study analysis is the ideal method to study and compare similarities and differences of both modes of participation among the three cases.

The FloodCitiSense ULLs applied a CS approach for the collection of citizen-generated data about rainfall and urban flooding. Urban flooding is an emergent urban hazard primarily caused by prolonged or intense precipitation events and by insufficient drainage capacity of the local systems in response [62]. This phenomenon is amplified by several factors: (i) an ageing drainage infrastructure, (ii) less green infrastructure in cities, and (iii) climate change with an increased likelihood of extreme weather events [63]. This type of flood represents a challenge for urban authorities as they are hard to predict across temporal and spatial contexts and can cause significant damage to the urban environment due to their rapid onset and highly localized nature. In this regard, the FloodCitiSense project’s aims were threefold: (i) to co-create an innovative set of CS tools for urban flooding (a mobile and web-based application), (ii) to collect official and citizen-generated data about rainfall and the occurrence of flood events, (iii) to impact policy and decision-makers based on the newly available information provided by an early warning tool.

Data on the participatory processes were collected through 11 in-depth interviews with various stakeholders: six with the partners of the project (three public and three private stakeholders), two with policy makers and three with users. All interviews were conducted online and lasted approximately 45 min. The interviews were conducted in a semi-structured manner but the format of the discussions aimed to be open and narrative-based. The interviews were transcribed and a qualitative content analysis was conducted based on the COPP framework (

Table 1. Comparison of Participatory Processes (COPP) framework and variables [23].

Context	Participatory Process	Outputs, Outcomes & Impacts
Target system elements Levels of governance influencing target system elements Other past/present intervention attempts Pre-existing relationships among participants Participants understanding of the target system elements	Participatory process objectives Instigator of the process Team Origin of the team Selection of participants Size of the group Level of participants process expectations Governance levels engaged Length of the process Number of events Degree of participant retention Setting of exchange Degree of participation Participatory methods and tools	Main outputs Impact on participants Impact on actions Social scale of the impacts Spatial extent Time scales of impact

) [23]. This framework provides a theoretical foundation for the description and comparative analysis of participatory processes, which can be used for monitoring and evaluating both processes and outcomes. There are three main assessment dimensions in the COPP framework: (i) context, (ii) process, and (iii) outputs, outcomes and impacts. For each ULL, the context elements are described, while the process and outputs, outcomes and impacts dimension focusses on the implementation of the CS and ULL. The precise definition of each variable and operationalization of the framework can be found in [23].

It is important to note that the author team who conducted this assessment was part of the project team that outlined the strategies for the participatory processes of the ULLs. However, they were not involved in the direct execution of the strategies and their local activities.

5. Results

The following sections detail the comparative results for the “context dimension”, “process dimension” and “output, outcome and impact dimension” of the FloodCitiSense use cases. The “context dimension” compares the contextual factors among the three ULLs that might influence the implementation of the participatory processes and outcomes. The “process dimension” compares the methodological and procedural choices of the two modes of participation. Finally, the “output, outcome and impact dimension” compares the tangible, intangible, short- and long-term effects of both participatory processes. In the comparison, the effectiveness of the participatory processes in achieving policy impacts is reviewed.

5.1. Context Dimension of the FloodCitiSense ULLs

The three ULLs had the same “target system elements” (

Table 2. Application of the COPP framework to three case studies–context dimension.

Context	Brussels	Birmingham	Rotterdam
Target system elements	Water, livelihoods, infrastructures	Water, livelihoods, infrastructures	Water, livelihoods, infrastructures
Levels of governance influencing target system elements	Macro: RMI Micro: Leefmilieu Brussel, Government of the Brussels-Capital Region, Brusselse Maatschappij voor Waterbeheer (BMWB)	Macro: The Environmental agency in the UK, Flood Forecasting Centre Micro: Birmingham City Council	Macro: NL Alert Micro: City of Rotterdam
Other past/present intervention attempts	Few	Many	Few
Pre-existing relationships among participants	Good pre-existing relationships	Good pre-existing relationships	Good pre-existing relationships
Participants’ understanding of the target system elements	Good understanding of target system elements	Good understanding of target system elements	Good understanding of target system elements

), namely, to develop an early warning service for pluvial flooding. This system would be a new development or improvement of the current systems in place for the three cities. Specifically, Birmingham currently relies on national forecasted flood warnings and maps from the Environment Agency, although these are not yet locally specific for urban floods. In Brussels and Rotterdam, a text message is sent to subscribed citizens through the INCA-BE service or NL-Alert service, which are not distributed in a timely fashion nor location specific. Therefore, the “other past/present intervention attempts” were reviewed as ‘few’ for Brussels and Rotterdam, and as ‘many’ for Birmingham.

The “levels of governance” influencing the target system elements are mostly organizations on the macro (national) and micro (city) level. In all ecosystems, good “pre-existing relationships” were present with public actors or with private actors that deal with water quality or the monitoring of water levels.

Last, the enablers of the ULLs stated that they had a good “understanding of the target system elements”, although the experience of conducting CS research in an ULL setting was new to them.

5.2. Process Dimension of the FloodCitiSense ULLs

The “instigators of the process” were the same actors as the “organization team” for the ULLs (

Table 3. Application of the COPP framework–process dimension.

Participatory Processes	Urban Living Labs (ULL)	Citizen Science (CS)
Instigator of the process	Researchers (VUB 1, TU Delft 2), Private actor (LGIU 3)	Researchers (VUB1, TU Delft 2, Private actor (LGIU 3)
Organisation team	Researchers (VUB1, TU Delft 2), Private actor (LGIU 3)	Researchers (VUB1, TU Delft 2), Private actor (LGIU3), civil society (EGEB 4) and Selly Park South)
Origin of the team	Majority of the team comes from the area and are affected	Majority of the team comes from the area and are affected
Selection of participants	Selected by team	Selected by team, selected by third party
Size of the group	Below 12	Over 50
Participatory process objectives	Resolving or avoiding conflicts, Exploring decision-making options	Gathering knowledge
Level of participants’ process expectations	High	High
Governance levels engaged	Multiple levels	Single level
Length of the process	1 to 5 years	1 to 5 years
Number of events	Multiple events	Multiple events
Degree of participant retention	More than 75% attended the whole process	25% attended the whole process
Setting of exchange	Heterogeneous group	Involved as a group
Degree of participation	Co-design innovationCo-ownership	Co-design innovation, Co-monitoring, reporting and validation of scientific results
Participatory methods and tools	Current and future scenario building, validation workshops	Co-creation workshops, Educational workshops, surveys, participant observations

¹ Vrije Universiteit Brussel; ² Technische Universiteit Delft; ³ Local Government Information Unit; ⁴ Etat Généraux de l’Eau à Bruxelles

). Two ULLs were coordinated by a university (Free University of Brussels, Brussels and TU Delft, Rotterdam), and one by a private non-profit organization (Local Government Information Unit, Birmingham). This contradicts with findings in the literature, namely that public actors are mostly the enablers of ULLs. For the CS approach, the organization team obtained help from two civil society organizations who recruited the participants in Birmingham (Selly Park South) and Brussels (Etat Généraux de l’Eau à Bruxelles). The “origin of the team” comes from the areas that were affected.

The “selection of participants” for the ULL, i.e., stakeholders, and the CS approach, i.e., citizen scientists, was performed by the organization team. The “size of the group” in the ULL was below 12 and for the CS over 50 per city. For the CS approach, it should be noted that the sample of participants was not representative of the whole population of the three cities. Further, as often in voluntary participatory processes, a self-selection bias was at play. Most participants took part in the CS activities following a pre-existing interest in flood and/or scientific processes.

The “objectives of the participatory processes” were different for the two modes of participation. The main objective of the ULL process was to develop an early warning system that would help to alleviate the conflicts that arise from the management of urban drainage flooding. In this sense, the goal of the ULL was to design and co-create an innovation that could solve this issue by allowing the collection of data and facilitating decision-making processes between stakeholders. The “organization team’s expectations” were rather high regarding these processes. They regarded the early warning service as an opportunity to be informed in a timely fashion about pluvial flooding, whereby push notifications could be sent. On the other hand, the main objective of the CS process was to gather data through the usage of this innovation. Therefore, the main goal of this process was to gain knowledge of the occurrence of flooding in the three cities and create prediction models. Here also, citizen scientists had high “levels of expectations”. They regarded the early warning service as a direct “feedback” platform, whereby they would receive a means of verification that their reported incident was recorded or that a consecutive action would follow by a public actor.

The “participatory methods and tools” were also different for both processes. In regard to the ULLs, participatory activities mostly took place in the first and third year of the project. In the first year, current and future scenario specifications were collaboratively written by the organization team and stakeholders of the ULL, whereby “multiple levels of governance” were consulted (micro and macro level). In the third year, validation workshops were organized to discuss the sustainability plan of the developed service. These workshops had the objective to validate the earlier gathered input and to set up a collaboration partnership with the city stakeholders, to search for a provider or a consortium of aligned partners for hosting the early warning system and to define possible integrations with already existing services and datasets in the city. This final phase was crucial in order to connect with established policymaking processes around flood risk management. In this sense, the ULL stakeholders could co-design the implementation of the target system elements in their city and take co-ownership. The “situation of exchange” of the participatory processes in the ULL was always in group format, taking the diversity of each stakeholder into account. Overall, a large majority of the ULL participants stayed involved throughout the whole process.

Last, for the CS approach, six co-creation workshops were organized in the first year of the project to collect citizens’ wants and needs in regard to the innovation development. Each workshop consisted of 12 participants with diverse profiles: citizens living in vulnerable areas of flooding, volunteers in flood action groups, civil servants of the local city council and of the Environment Agency, etc. In the second and third year of the project, multiple participatory activities took place. In total, six educational workshops, two in each city, were organized with a total of 74 attendees. The workshops had the goal to educate participants on urban rainfall and flooding, the functioning of the sensors and the application, and how they could contribute to the project. In the third year of the project, the CS activities were mainly remote and recruited a large number of participants to download the mobile application and enter flooding reports. During the project lifetime, the citizen scientists were regarded as a “homogenous group”, all being affected by the target system elements. The degree of their participation was mostly related to the co-design of the innovation, and the co-monitoring, reporting and validation of scientific results. In total, FloodCitiSense reached a total of 264 citizen scientists across the three cities, but the number of active users per month averaged around 60, which demonstrates a rather low retention rate.

5.3. Outputs, Outcomes and Impact Dimension of the FloodCitiSense ULLs

The “outputs of the participatory processes” are strongly linked with their objectives (

Table 4. Application of the COPP framework–outputs, outcomes and impact dimension.

Outputs, Outcomes and Impacts	ULL	CS
Main outputs of the participatory processes	New technology (early warning service), Terms of reference (ULL Brussels)	Model
Main impact on participants	Reduction of conflict (ULL Brussels); Increased collaboration, trust and relationship building; Improved understanding of target system elements	Improved understanding of target system elements, Capacity building, Influence on decision
Main impact on actions implemented by participants	Collective action (ULL Brussels), and individual behaviour change (ULL Birmingham and Rotterdam)	Change in daily practices and actions
Social scales of the impacts	Within and beyond the group involved in the process	Only within the group involved in the process
Spatial extent of the impacts	In and beyond the area where the process was implemented	Only in the area where the process was implemented
Time scales of impacts	Long term	Medium term

). Since both modes of participation had different objectives, the outputs also differ (cf. Section 5.2). The main output of the FloodCitiSense ULL processes is a newly developed technology: an early warning service for urban flooding. This service consists of three tools: a low-cost sensor for rainfall monitoring based on the Internet of Things solutions (180 installed in the three ULLs), and a mobile and a web-based application for reporting flood incidents and visualizing the sensor data. On the other hand, the main output of the CS approach is a data-driven model for determining the critical threshold for urban flood occurrence based on citizen collected data (435 crowdsourced flood reports). However, the technical performance of the developed technology and data-driven model were not the same for every ULL. For the ULL in Brussels, the distribution of the low-cost sensors has a good geographical spread in the Brussels-Capital Region and made it possible to complement the data of the official measurement stations. This result was not achieved in Rotterdam and Birmingham due to technical performance issues with the sensors, and a lack of coverage of the territorial area. Furthermore, other issues, such as the perceived effort in data control and validation, the low perceived reliability of the data, and lack of integration with other services meant that only in Brussels was a Terms of Reference between the stakeholders of the ULL established. Specifically, there was a strong agreement among the stakeholders to form a consortium of partners for hosting the service, with the ‘Brusselse Maatschappij voor Waterbeheer’ (BMWB) as the main provider. This public organization already has the FLOWBRU service in place, which is a water monitoring network for rainfall, water levels and discharges, and water quality; a potential integration could take place here with the early warning service of FloodCitiSense. Furthermore, an integration between the FloodCitiSense data driven-model and the datasets of the Belgian Royal Meteorological Institute (RMI) is possible, which has specific expertise in operational now-casting of rainfall through INCA-BE. In the other ULLs, no agreement or Terms of Reference were established between the stakeholders. The Birmingham City Council (BCC) expressed their concerns about the reliability of the crowdsourced information, especially about the static flood reports with photographs and descriptions of flood incidents. Furthermore, the BCC did not have the resources available to validate the flood reports and provide feedback to citizens about the policy-related decisions. In the Rotterdam ULL, the municipality expressed their concerns about the overlap between the existing application, called ‘Buitenbeter’, and FloodCitiSense. Some of the features overlap and they do not want two similar services for one city. Further, they mentioned that the management and sending of the early warning messages should rather be a responsibility of the Safety Alliance (including the fire and police department) in Rotterdam, and not the city council.

Next, the main “impacts on participants” also differ between the ULL and CS processes. For the ULL in Brussels, a clear impact was found on the reduction of conflict for flood risk management, as well as on collaboration, trust and relationship building. This was less the case for the ULLs in Birmingham and Rotterdam, although for all ULLs there was an impact on their understanding of the target system elements related to flooding. For the impacts on participants of the CS approach, the interviewees affirmed an improved understanding of the target system elements, capacity building and influences on decision making. The interactive discussions with citizens, especially during the educational workshops, allowed for the creation of a mutual understanding between science, policy and society. This resulted in an increased knowledge and greater support for the local flood procedures by citizens. Further, stakeholders of the ULLs testified about a greater willingness in applying an active role for citizens in future flood management strategies. Deliberative decision-making was an element that, in their daily work, the local authorities had not yet succeeded in. It was, rather, a unidirectional flow of information from them to the citizens, such as provision of advice, tips and diverse measures. Citizens affirmed that they now feel heard more by scientists and policy makers around urban flooding issues in their city.

The “main impact on actions” differs, from individual behavioral change for the ULLs of Birmingham and Rotterdam towards collective actions among stakeholders in the ULL of Brussels. As explained above, the exploitation of the early warning service continues in Brussels, but not for the stakeholders in Birmingham and Rotterdam. The impacts of their actions remain at the individual level of knowledge gains about sustainable development. For the CS approach, a change in daily practices and actions was witnessed during the project, since citizen scientists rely on the FloodCitiSense service for monitoring rainfall and flooding in their local area.

The “social and spatial” scales of the impacts of the ULL go beyond the involved group and areas. The stakeholders of the ULLs shared their lessons learned relating to technology development with other cities and made the source code of the application freely available. Further, the low-cost sensors will be deployed in other regions for further technical testing and validation, and stakeholders in the Birmingham ULL will continue their trials to improve the technical performance of the sensor measurements. In regard to the social and spatial scale of the CS approach, it remains limited to the group and area where the processes were implemented. Last, the “time scale” of the impacts of the LL approach are assessed as long term, while for the CS approach this was medium term. Stakeholders in all three ULLs continue their efforts to gain understanding of the target system elements through their increased collaborations and relationship building. For the CS approach, no future monitoring campaigns will be supported through the ULLs, although citizen scientists affirmed that they would continue to use the mobile and web application in the following months and would keep the sensors.

6. Discussion

The comparative analysis of the process dimension and the outputs, outcome and impact dimensions greatly differs between the two modes of participation. Each participatory process had its specific objective and, respectively, a different output and impact on participants. The stakeholders of the ULLs were more involved in the development and sustainability of the solution, while the citizen scientists were more involved in the monitoring and reporting of urban floods. Expectations of participants of the participatory processes were rather high and were only truly met, and only to a certain extent, in the ULL of Brussels. LLs often do not reach the exploitation stage; in the case of Birmingham and Rotterdam the activities stopped after the experimentation phase. This was mostly due to technical performance issues of the low-cost sensors, the low perceived data quality of the crowdsourced flood reports, lack of integration with existing systems and the high perceived efforts involved in data control and validation of the crowdsourced information. Most policy impacts were realized in the ULL in Brussels with a Terms of Reference among stakeholders and a reduction in conflict that arises from the management of urban drainage flooding. Further, through the CS approach, there is an improved understanding among citizens about the target system elements, with an increased level of influence in decision-making processes.

Through the practical experience of the FloodCitiSense use cases, the following best practices on how CS can be incorporated into ULLs for achieving policy outcomes can be identified:

• CS processes can be incorporated into ULLs in order to accommodate multiple levels of governance: nowadays, public and private actors are not greatly aware of the opportunities of CS. CS projects often operate without an ecosystem of public and private actors and are mostly initiated by academia. In contrast, ULLs often have Quadruple Helix formations. Based on the experience of FloodCitiSense, it is argued that the ecosystem approach is beneficial for the translation of science outcomes towards policy making. Thanks to the ULL participatory methods, the data-driven model and early warning system of FloodCitiSense is further exploited at the micro and macro level in Brussels. However, based on the experience of Birmingham and Rotterdam, it is also suggested that public actors should be involved in the co-design of the proposal, especially if this is related to environmental risk governance.
• CS processes can be incorporated into ULLs in order to gather new scientific knowledge and support capacity building: through the CS approach, data is collected for a genuine scientific outcome, and not only for technical purposes. Based on the experience of FloodCitiSense, it was found that CS can result in a better understanding of target system elements and increased capacities of the participants to influence decisions. Citizen scientists obtained skills and knowledge related to flood risk and sensor management. Through both modes of participation, democratization of science and innovation can take place.
• CS processes in ULLs can create a synergy in data and technology governance: ULLs focus on sustainable solutions, while CS focuses on the gathering of scientific knowledge. Based on the experience of FloodCitiSense, both objectives can be combined into one project. However, seeing the issues experienced in Rotterdam and Birmingham, preference should be given to existing and already validated ICTs in collecting data. The combination of both goals, gathering new scientific knowledge and developing new ICTs from scratch, can be very ambitious and challenging. This is especially true for initiatives where policy outcomes are to be realized, and therefore require large-scale datasets with a good spatial and temporal scale. As such, ULL processes can also focus on interoperability aspects with other existing urban platforms and save efforts and costs.

7. Conclusions

The monitoring and management of flood events in cities are typically in the hands of local government agencies. However, even powerful governmental agencies are having difficulties facing such a wicked problem. Participatory approaches developed in the framework of Smart Cities, such as CS and ULLs, offer the possibility to engage a large and diversified group of stakeholders in order to better face the complexities of the issue. The case study of the FloodCitiSense project offers a specific example of such a set-up in the context of flood risk management. This case study showcased that the integration of CS and ULLs, a multi-stakeholder approach with an emphasis on citizen co-creation and participation, can lead to greater support of local policies, although not without a fair share of challenges.

To finish, we would like to emphasize that the results of this article ate based on a sole case study analysis. We invite other researchers to share their experience on the association of CS and ULLs to better understand their delineation and their individual contributions in regard to scientific, innovation and policy impacts as well as their own set of challenges and opportunities.