Applying an RRI Filter in Key Learning on Urban Living Labs’ Performance

: Urban living labs is a practical methodology in improving sustainability in cities by facilitating collaborative learning and innovation in a real-life environment, thereby mainly responding to the needs of users (citizens). The paper aims to ﬁlter a list of key learnings on urban living labs through the lens of Responsible Research and Innovation (RRI). One of the motivations is that key learnings on urban living labs are mainly derived from means-goal e ﬀ ectiveness (MGE) thinking while the urban setting calls for a broader perspective due to complexity and tension from the multi-actor, multifunctional, and multi-scalar character of cities. The ﬁltering reveals almost 40 learnings as ‘overlap’ and ‘exclusive for MGE’. Importantly, ﬁve learnings are identiﬁed as speciﬁc for RRI and potentially enriching living lab methodology: ethical and normative principles like health, safety, security, and equality between societal groups, and a wider distribution of beneﬁts and risks of living lab outcomes, in particular, contradictory sustainability issues. The RRI ﬁltering causes three practical implications: coping with uneven power distribution between stakeholders, limited feasibility of applying the comprehensive learning framework, and challenges of overarching platform structures enabling to better incorporate RRI concerns in living lab methodology. The ﬁndings as presented in an adapted list are new, as RRI values and concerns have seldom been applied to practical innovation and have never been explicitly applied to urban living labs’ performance beyond the borders of e ﬀ ectiveness thinking.


Introduction
The last decade has seen many local initiatives to make cities more innovative (smart) and develop solutions to sustainability problems. Part of these initiatives, living labs, is meant to facilitate collaborative learning and experimentation in a real-life environment, in which the needs of users, mainly citizens, play an important role [1][2][3][4][5]. Today, urban living labs are blossoming more than ever before, being driven by calls for acceleration of sustainability improvement e.g., in energy, transport and health systems [6][7][8][9][10], and fitting into two broader societal trends. These trends are decentralization of (some) public tasks and increased decision-power (participation) of citizens on urban services and enabling users (consumers) to play an important role in innovation (co-creation) [11][12][13][14].
The term living lab originated in the early 1990s, when teachers at university started to see the potentials of specific city neighborhoods for student learning and attempts to solve real-life problems, in Philadelphia, and called them 'living laboratory' [15,16]. Living labs started to be elaborated in the early 2000s at Massachusetts Institute of Technology (Cambridge, MA, USA), when research was moved from laboratories to in-vivo settings to observe interaction of users with innovations in real-life. In Europe, most early experience with user-innovation in living labs has been gained in ICT-based innovation in the European Network of Living Labs (ENoLL), a platform established in 2006 to foster Given the three core values and later extensions, the scope of RRI is broader than effectiveness thinking on many urban living labs. However, among urban policymakers and managers, it is unknown what can be learned from RRI concerns and how RRI may contribute to improvement (elaboration) of key learnings on performance of urban living labs. Accordingly, this paper is an attempt to filter the means-goal effectiveness framework of urban living labs' performance, using a set of RRI values and process concerns. The results contribute to RRI and urban planning/policy literature as they include the first design of an RRI filter to mainly means-goal effectiveness thinking in urban living lab methodology, or broader, urban experimentation in policy-making. The filter includes aside from core values, various process concerns, like diversity and inclusiveness, and transparency and interaction in decision-making, which are only partly similar to existing learnings on living labs. Application of the filter is new as it implicates extending performance criteria with ethical norms and reflection on a wider distribution of benefits and risks of living lab outcomes.
The research questions can be detailed as follows: (1) How would an RRI inspired set of values and concerns (filter) be composed aimed at improving learnings on urban living labs? (2) What are the changes in the living lab framework as a result of RRI filtering? (3) What could be practical implications of this filtering process?
The paper proceeds as follows. Section 2 discusses methodology aspects of the study, and it zooms in on means-goal effectiveness (MGE) thinking in an input-outcomes approach. Section 3 is an analysis of urban living labs learnings based on this approach, starting from the central concepts of participation and co-creation. Section 4 presents the results of the filtering process: design of the RRI filter, its subsequent use in filtering urban living labs' learnings, the elaborated list of urban living labs' learnings, and practical implications. Section 5 concludes and addresses emerging research questions.

Data Sources and Steps
The study makes use of preliminary designed lists of performance learnings on living labs [23,24]. Four data sources have been used in the design and exploration of these lists: (1) a systematic literature review concerning living labs' performance and key factors; (2) qualitative analysis of six case studies selected on the basis of different organizational complexity, using their websites and (two cases) evaluation reports; these were living labs on ambient assisted living of elderly, refurbishment of a shopping mall and of a hospital, and university campuses as living labs; (3) sharpening of results in two workshops and an international conference (in particular, application of the results to urban living labs); (4) interaction and interviews with experts at Delft University of Technology (NL) and Manchester University (UK) (see acknowledgement). Inspiration was also drawn from participant observation in Alexis Nihon Shopping Mall in Montreal (Canada), aimed at increasing social integration of disabled persons [29,44,45]. Accordingly, the methodology in previous parts of the study has focused on qualitative case-studies, the results of which have been tested and adapted in various rounds [46,47]. The last round is the one reported in this paper, namely, elaboration of the most recent results on performance learnings with values and process concerns of RRI.
The rationale behind the list of performance learnings so far is means-goal effectiveness (MGE) and adaptive management. Means-goal effectiveness as used here, refers to implementation evaluation emphasizing whether intended inputs (interventions) are being applied and connected processes being achieved, and to outcome evaluation emphasizing whether the program effects are being reached in the target population's behavior (in our case, users and wider stakeholders) [48]. Broader evaluations include impacts on other urban domains, on broader (ethical) developments and the longer term, using other values than the more narrow effectiveness (efficiency). RRI typically provides such a broad perspective.
The RRI values and concerns to be included in the filter of living labs learnings are derived from RRI literature and expert opinion. In the design of the RRI filter, various process requirements, as addressed by Macnaghten et al. [43] are added, while outcomes of the PRISMA project [49] derived from innovation in the business context of industrial firms, are also used.

Input-Outcomes Approach
The fulfillment of goals of any policy or management endeavor can be approached using an input-outcomes framework. In a simplified input-outcomes approach (Figure 1), inputs refer to design of goals and implementation (intervention), in particular, the organization and management structure, but also selection of participants and of the real-life environment involved, etc.

Input-Outcomes Approach
The fulfillment of goals of any policy or management endeavor can be approached using an input-outcomes framework. In a simplified input-outcomes approach (Figure 1), inputs refer to design of goals and implementation (intervention), in particular, the organization and management structure, but also selection of participants and of the real-life environment involved, etc. Inputs influence the nature of processes that are aimed to happen, in living labs summarized as participation and co-creation. However, various external influences may affect living lab processes and lead to outcomes which are beyond managers' control (exogenous influences). Thus, outcomes encompass intended outcomes but also not intended ones.

Participation and Co-Creation
Two concepts are seen as centered in living lab methodology, namely participation and cocreation. The concept of participation has developed in policy making, urban planning, and design and delivery of public services [13,50,51]. Simply stated, each individual or group affected by an intervention has a voice in the decision making concerned. Already for a long time, participation has been granted by certain public bodies, but more recently citizens that feel dissatisfied with policies, make themselves heard and take direct action by themselves. The overall goal of participation connected to living labs is to facilitate the engagement of individuals or groups of citizens in the design of policies by using direct participatory means, such that their ideas, needs, interests, and values are incorporated in governmental decision-making [51]. Requirements (or values) at the practical level include among others involvement of all relevant stakeholders, representativeness and active citizenship, transparency and accountability, and these are prominently present in living lab performance. Co-creation, simply-stated, is aimed at having stakeholders either make or learn together, or both, in a single project or broader network [22]. An important origin of the concept of co-creation is research on innovation methodology, in which it was found that the quality (value) of inventions and other solutions benefits from suggestions and active input from users (customers) [52][53][54]. The prominent presence of co-creation in living labs comes with the issue to what extent it is driven in formal ways (e.g., procedural time planning) or informally [3,22,32]. If the last is true, cocreation tends to be the result of spontaneous self-organization, which could enhance speed of the learning processes, but could also exclude specific users. Incentives for participants, as their own sake or broader purposes, are a related point of attention. Overall, practical aspects of both participation and co-creation need to be considered and made mutually consistent, as well as consistent with the broader goal of the living lab. In the next section, we follow the input-outcomes approach to living Inputs influence the nature of processes that are aimed to happen, in living labs summarized as participation and co-creation. However, various external influences may affect living lab processes and lead to outcomes which are beyond managers' control (exogenous influences). Thus, outcomes encompass intended outcomes but also not intended ones.

Participation and Co-Creation
Two concepts are seen as centered in living lab methodology, namely participation and co-creation. The concept of participation has developed in policy making, urban planning, and design and delivery of public services [13,50,51]. Simply stated, each individual or group affected by an intervention has a voice in the decision making concerned. Already for a long time, participation has been granted by certain public bodies, but more recently citizens that feel dissatisfied with policies, make themselves heard and take direct action by themselves. The overall goal of participation connected to living labs is to facilitate the engagement of individuals or groups of citizens in the design of policies by using direct participatory means, such that their ideas, needs, interests, and values are incorporated in governmental decision-making [51]. Requirements (or values) at the practical level include among others involvement of all relevant stakeholders, representativeness and active citizenship, transparency and accountability, and these are prominently present in living lab performance. Co-creation, simply-stated, is aimed at having stakeholders either make or learn together, or both, in a single project or broader network [22]. An important origin of the concept of co-creation is research on innovation methodology, in which it was found that the quality (value) of inventions and other solutions benefits from suggestions and active input from users (customers) [52][53][54]. The prominent presence of co-creation in living labs comes with the issue to what extent it is driven in formal ways (e.g., procedural time planning) or informally [3,22,32]. If the last is true, co-creation tends to be the result of spontaneous self-organization, which could enhance speed of the learning processes, but could also exclude specific users. Incentives for participants, as their own sake or broader purposes, are a related point of attention. Overall, practical aspects of both participation and co-creation need to be considered and made mutually consistent, as well as consistent with the broader goal of the living lab. In the next section, we follow the input-outcomes approach to living lab methodology in providing a more complete picture of learnings on living labs performance [23,24].

Urban Living Labs Learnings
Inputs refer to the combination of all working goals and means, like financial budget, time, expertise, tools, and technology, influencing the learning processes in the core of living labs [1,2,[54][55][56]. Literature indicates that many urban living labs are facilitated by local governments, in terms of financial budgets and time, expertise and legitimacy [23,24,28]. With regard to financial budgets, attention to cost structure and cost-effectiveness is needed, but much depends for example on whether the living lab is a temporary or permanent organization.
Consensus is required in the formulation of goals and means, and in creating a shared vision on the future (the problem/issue) including how the living lab can contribute [28]. Reaching goals is not only seen as reaching direct project results, like specific solutions to sustainability problems, but also understanding of the learning processes and wider impacts in urban transition policy [32]. With regard to management of living labs, literature indicates that traditional management would not work in living labs, rather the management models should match with the voluntary character of participants' involvement and their needs for motivation and inspiration, and for preserving trust between stakeholders [23,24,28].
Management takes place on two main levels, namely, on the project or program level (project management) and on the level of core learning processes, and this goes hand-in-hand with some sort of monitoring of processes and outcomes. At the start, there needs to be agreement on leadership and on decentralization of particular management tasks [26]. On the way, however, the management should be able to deal with emerging social complexity, calling for ability to (re)build trust and team spirit, and to use boundary-spanning if needed. The management models differ between living labs dependent on their organizational complexity such as the number of different actors involved [23,24] and on outcomes to be achieved. In terms of outcomes, living labs used for ideation and gaining creativity in solutions can be managed with more openness and flexibility compared to those in which decisions need to be taken on existing solutions in bringing them to market as quick as possible, like concerning selection of projects, selection of business models and financing [19,23,24,28].
Further, as part of the inputs, the real-life environment [1,28] needs to sufficiently represent the problem/challenge domain, yet at the same time, the quality characteristics should sufficiently stimulate creativity and improvisation and facilitate validation/testing of new solutions. Some urban living labs act mainly as testbeds in early stage experimentation at a real-life scale, often without disturbance of rules and regulation. Testbeds are usually at university campuses and can be exemplified by testing solutions coping with extreme weather conditions (prolonged dry as well as intense rainfalls), like in 'WaterStreet', The Green Village (TU Delft, NL). Further testing and improving in the real-life setting of city quarters is a next step [57] and this often comes with regulatory and legal issues. We mention regulation concerning innovation in utilities (such as in electricity grids connected to direct and alternating current); real-estate related legal issues, concerning ownership and access to the site; and legal liability connected to reconstruction and infrastructure works, etc. All these need to be settled, which tends to be time-consuming. A further point of attention deals with the spatial setting of urban living labs, eventually causing influence from activity in adjacent areas in city quarters, calling for accounting for other functional needs in city quarters than the one(s) in living labs [23,24].
With regard to processes, including participation and co-creation, active involvement of users and collaborative experimentation and experiential learning are in the core of urban living labs. Experiential learning is different from traditional learning and matches with learning practice in a real-life context, with open and active participation in the learning activity and with knowledge flows into various directions, in order to identify new opportunities [58]. As earlier indicated, the involvement of adequately selected user-groups is critical as an input, representing citizens or other users affected by the problem but also owning capabilities/skills that match with living labs learning practices, like using simulation tools and scenario thinking [23,24]. In addition, user involvement should take place early in the process and should be sufficiently intensive and rich in terms of multiple experimentation and switching roles [1][2][3]55,59]. However, there is a dilemma involved in selection of participants and wider stakeholders. In principle, living labs rest on consensus on priority for user-needs in the designing of new solutions. This may, however, cause a trend in practice to avoid involvement by certain participants that do not match with consensus thereby allowing to reach results in reasonable time, however, most probably with less credibility and convincing power. Further, in co-creation's learning activity sufficient results need to be gained, and these results are to be integrated into the next rounds of learning [55,59,60]. To facilitate achieving sufficient learning results and feed-backs, trust and coherence need to be built and preserved, calling for respecting personal values within user-groups and between users and other stakeholders, eventually using intermediation if a divide tends to arise [60][61][62]. For example, in the context of elderly people as users in ICT projects (ambient assisted living), trust needs to be preserved through respecting the value of privacy and self-determination [23,24]. Such situations also call for monitoring of participants' satisfaction.
Further, in a wider circle of stakeholders, interaction (learning) is aimed to enhance conditions for upscaling of viable solutions, financing next steps, responding to regulation etc. and developing business models [60,63], however, whether all this plays a role in urban living labs depends on the stages that are covered. Bringing (co-created) solutions to market and upscaling often require inter-organizational interaction and learning with regulators and certification institutes, but also with financial investors, (city) policymakers, market and marketing experts, large companies, etc. With regard to participation, the composition of stakeholders involved is preferably balanced, this in order to prevent the rise of large power distance and needs for boundary-spanning. Upscaling living lab solutions is faced with many challenges, like a transparent selection of the best project, e.g., with regard to applicability of the results to cities' reality, in particular sometimes fragmented urban policy arenas and powerful stakeholder lobbies [57,60,64,65]. As previously mentioned, a certain openness/transparency is typical for the creative character of urban living labs, for demonstration purposes but also serving to incorporate creative ideas from outside [24,28,66].
Outcomes which are intended mainly refer to creation of new solutions, in particular, a better user-quality of them, eventually bringing more sustainability innovation quicker to market [30,59]. A deeper understanding of the learning processes and collaborations themselves is also often seen as part of intended outcomes of living labs [29,59,67,68]. The perspective on intended outcomes of urban living labs is broadened more recently by adding the identification of wider sustainability (transition) impacts [32]. Unintended outcomes may happen, for example, where tension between stakeholders arises and conflict causes delay, or where lack of attention to key processes, like elaboration of user feed-back, causes suboptimal results. Attention for (anticipation on) problematic situations is somewhat exceptional in the practical (urban) living lab literature, but it is increasing more recently [61,69]. Overall, there is a permanent need for monitoring the environment and the living lab learning processes for influences that may render the reaching of intended outcomes doubtful and eventually call for reconsideration of means and goals. Table 1 shows the components of the RRI filter with a distinction between core values and concerns about processes, including some special remarks/details on application to urban living labs' learning. Ethical acceptability, as one of the core values, can be prominently present, even while dealing with sustainability solutions. An example is social housing reconstruction aimed at making houses (almost) energy neutral, while the increased rent cannot be afforded by a minority of poor families, urging them to move.

Application in Filter to Urban Living Labs (ULL) Process and Results
Three core values Among the process characteristics, emphasis is put on openness (transparency) in interactive communication, dialogue and decision-making, as well as responsiveness and adaptive coping with changes in society. In addition, concerns about reflection and anticipation produce a relatively broad evaluation activity, including underlying assumptions and different types of outcomes (e.g., longer term, wider context, wider contribution to transition, unintended/contradictory) and specific values (e.g., those protected by law, specific human values). Furthermore, concerns about diversity and inclusiveness, come among others with equality (gender; minorities) and the need to develop shared visions among diverse stakeholders. Finally, RRI concerns can be extended to include local contexts' needs and culture, eventually social innovation, as well as local power structures.
Note that RRI, as it has developed and framed, is not explicitly dealing with divergent interests and unequal power distribution [38,39,43]. Likewise, points of departure in urban living labs, are often equality, transparency, and balanced stakeholder involvement. This situation seems at odds with traditional and non-western forms of knowledge (production), particular social and religious context, and constraints of specific power distribution, calling for more social innovation, and eventually new ways of negotiation and decision-making [70,71], not identified so far. Table 2 presents 37 'filtered' key learnings of sustainability-oriented urban living labs that tend to be important in many living lab stages. Learnings are structured following the input-outcomes framework: inputs to organizational preparation and real-life environment; processes in user-centered collaborative learning in a small circle (co-creation) and in learning in a wider circle of stakeholders; influences beyond control, and outcomes of the living labs. Most of the learnings deal with participation and co-creation (and wider interaction). Table 2. Elaborated list of learnings on urban living labs.

Preparation inputs (a)
Develop a shared vision on future sustainability and living labs contribution Define leadership and decentralization of decision-making Attract qualified staff to deal with social complexity (tension) and boundary-spanning (b) Remain consistent in focus and avoid showcasing of success; learn from mistakes Incorporate a monitoring and evaluation scheme (participative) Identify and anticipate contradictory sustainability issues Incorporate safeguarding the project from ethical issues Ensure equality between societal groups Incorporate input from groups not directly involved, e.g., policy, NGOs Real-life environment inputs Select high-level representation of the challenge and an inviting arena Organize in first stages (test-bed) a 'regulation-free' environment Give attention to spatial setting to avoid disadvantages from adjacent functions Give attention to ownership/user rights (access) of buildings and infrastructure Give attention to other legal issues, e.g., liability in experiments, privacy, physical impacts Participation and co-creation (processes) Select users on representativeness, motivation, capabilities Provide early involvement of users and adequate methods (multiple methods) Pay attention to key aspects of co-creation: e.g., making/learning together; formal and/or informal, drivers and incentives for participation, and guide consistency Gain sufficient results from co-creation as input to next steps Give attention to satisfaction of users (other partners) with the learning activity Respect critical values of users, particularly vulnerable ones Use adequate methods in steering on learning activity, like participatory action Preserve openness, e.g., in data access and data exchange Apply active use of normative principles, like health, safety, security Influences beyond control Scan the environment and internal processes that may cause not-intended results Provide participation in 'judging' non-intended (adverse) results Adjust goals and means to reality (e.g., factors beyond control), if necessary and possible Outcomes Determine whether outcomes are worth monetary (cost structure and cost-effectiveness) and social investment Act upon a wider distribution of risks of outcomes, including stewardship on the future Act upon impact on sustainability transition processes in cities The elaborated list (Table 2) is the result of the filtering activity of existing lists [23,24]. As Appendix B indicates, we assigned 'filtering labels' to the existing learnings as follows: overlap between means-goal efficiency (MGE) and RRI, exclusive for MGE and exclusive for RRI. Filtering produces the following classification of these learnings: 14 are overlaps between RRI and MGE, 17 are exclusive for MGE and need to be maintained, while six are exclusive for RRI and well applicable to urban living figurelabs. Accordingly, learnings specific for MGE constitute the largest part, at almost 60 percent, particularly dealing with the real-life environment, co-creation and wider stakeholder involvement. Importantly, RRI include a broader scope in a normative approach, mainly adding to preparation inputs and reflection on outcomes for example, safeguarding ethical views (processes); ensuring equality between societal groups; and ensuring reflection and acting upon a wider distribution of benefits/risks of outcomes, including competing trends and future developments, eventually cross-border [32,43]. Competing trends refer to situations in which benefits in a particular sustainability domain cause limitations to other sustainability domains, like greening of urban layout and fighting of housing shortage [72]. In addition, both RRI and MGE include reflection on whether the intervention and results are worth all effort (including monetary), thereby also referring to the limits of RRI [38], which seems implicit in MGE thinking but not often emphasized in urban living labs practice.
Overall, the learnings mentioned in the elaborated list (Table 2) may serve two aims, first, key points of attention in the design and implementation of urban living labs, and secondly, key points of attention in monitoring and evaluation.
Three implications of use of the elaborated list can be foreseen in practice. First, an unresolved issue is what RRI would prescribe in a situation of stakeholders with divergent interests and unequal power positions [38,43]. To arrange collaboration between such stakeholders and creating alignment and consensus between them, may be difficult to achieve, while diversity, openness and inclusiveness tend to be important concerns. Both urban living labs methodology and RRI require a stronger attention to think and act upon divergent interests and power positions in practice.
Secondly, feasibility of practical implementation of the comprehensive list of learnings can be a question mark. To date, many urban living labs are only active in an early stage and at small financial budgets, which may imply that demanding RRI concerns may be easily postponed and pushed forward to next stages [57]. To prevent such a situation, RRI thinking needs to be incorporated from start and developed as a clear input in the design of urban living labs.
To enhance feasibility of practical use, priorities may be assigned to learnings at start of designing the living lab, among others, dependent on the foreseen stage to be reached and the selected character, like open with less strictly defined goals or more closed. In any case, it seems that priority needs to be given to the basic components of preparing and operating living labs, like shared vision, leadership, and decentralization, monitoring/evaluation, key quality of real-life environment, selection of participants, selection of wider stakeholders, quality of co-creation, acting upon unintended results, and attention for monetary and social investments. Possibly, in preparing urban living labs, some options can remain open at start and selection will follow later on.
Finally, with regard to the need to broaden the learnings in design, implementation and evaluation beyond the narrow boundaries of MGE thinking, many urban living labs would not be able to fulfill the additional tasks individually. This situation may be solved by using (existing) platforms or network structures with important overarching RRI concerns. These platforms or networks could be thematic, like healthcare or urban agriculture, because of sharing of some specific RRI concerns. At this level, a practical tool needs to be developed that facilitates dealing with the RRI concerns among individual living labs, for example, by adding a new part to canvas methodology [69,73].

Discussion
This paper provided steps towards a list of learnings on living labs performance that can be used in design and implementation, as well as in monitoring and evaluation, and this list is different from previous ones due to using responsible research and innovation (RRI) as a filter. Living labs as a practical innovation methodology enables co-creation in a real-life environment aimed at responding to the sustainability needs of users (citizens) at high levels of participation. Application of the RRI filter means extending means-goal effectiveness criteria with ethical and normative principles in sustainability, like health, safety, security, and equality between societal groups, as well as a wider distribution of benefits/risks of living lab outcomes, including contradictory issues. In fact, the focus on project outcomes has been replaced by a broader focus. The results contribute to RRI and urban planning/policy literature as these are the first concerning a filter derived from RRI concerns, as well as an application of this filter to means-goal effectiveness thinking in urban living labs.
As always, the results suffer from some shortcomings. Both the RRI filter and the resulted list of urban living lab learnings (Table 2) are preliminary and call for further testing in future quantitative research. In addition, there is the weakness of using constructs of which the validity has not been determined. Accordingly, we propose to collect data (interviews) on a hundred urban living labs in order to statistically test validity of the constructs used-such as active involvement, co-creation, transparency, openness-and to identify statistical relationships between them. This, in a causal context of living labs means-outcomes and broader impacts, in order to identify the dominant relationships using statistical techniques like multiple regression and structured equation modelling. Future research, of a different kind, like social action research, is also needed, namely, to find ways in coping with (considerable) gaps in interests and power between certain stakeholders, and to remove limits of living lab methodology and RRI.