Collective Intelligence to Co-Create the Cities of the Future: Proposal of an Evaluation Tool for Citizen Initiatives

Abstract

Citizen initiatives (CIs), through their activities, have become a mechanism to promote empowerment, social inclusion, change of habits, and the transformation of neighbourhoods, influencing their sustainability, but how can this impact be measured? Currently, there are no tools that directly assess this impact, so our research seeks to describe and evaluate the contributions of CIs in a holistic and comprehensive way, respecting the versatility of their activities. This research proposes an evaluation system of 33 indicators distributed in 3 blocks: social cohesion, urban metabolism, and transformation potential, which can be applied through a questionnaire. This research applied different methods such as desk study, literature review, and case study analysis. The evaluation of case studies showed that the developed evaluation system well reflects the individual contribution of CIs to sensitive and important aspects of neighbourhoods, with a lesser or greater impact according to the activities they carry out and the holistic conception they have of sustainability. Further implementation and validation of the system in different contexts is needed, but it is a novel and interesting proposal that will favour decision making for the promotion of one or another type of initiative according to its benefits and the reality and needs of the neighbourhood.

1. Introduction

Today’s cities require a change in urban planning to address the complex relationships between citizens’ interests and needs, urban transformations, environmental degradation, urban policies, and economic interests in a sustainable approach [1]. Social innovation is one of the pillars of sustainability that is becoming increasingly important as one of the dimensions that merit a multidisciplinary and transversal approach [2,3,4,5,6,7,8,9], where the active participation of citizens is required not only as recipients [10] but also as co-creators of their reality. In this sense, a series of mechanisms and strategies have emerged that seek to involve citizens in decision-making processes in the city, such as participatory budgets, urban centres, living labs [11,12], and innovation centres, in which synergies are created between civil society, public administration, academia, and companies [13] to seek solutions to social challenges; these strategies and platforms are usually guided by public administration. On the other hand, another type of participation and appropriation of space mechanism called Citizens’ Initiatives (CIs) are born in a spontaneous but conscientious way.

CIs can have different natures, guided through top-down processes or self-managed through bottom-up processes. This article focuses on CIs aligned with bottom-up processes, which can be defined as “informal processes of citizen practice that resiliently and adaptively modify the urban environment. They are self-organized, collective practices that work for the urban empowerment of citizens and develop critical processes on the current city” [14,15] (this citation was translated from Spanish by the authors).

These initiatives arise from a variety of problems and perspectives in the city, such as the recovery of public spaces, the creation of gardens, the use of renewable energies, and the protection of the city’s cultural and historical legacy, among others. As Cámara Menoyo pointed out, these are spontaneous processes with heterogeneous results, which, although they do not have a similar scope to the urban planning processes promoted by local authorities [16] (this citation was translated from Spanish by the authors), have a positive impact on local problems and on raising citizens’ awareness.

The role of CIs in the transition to sustainability is a topic of growing interest [17,18] due to recent evidence that they can in one way or another contribute to improving their environment and people’s quality of life, yet there is little evaluation and assessment of their level of contribution [19].

CIs have the potential to be less constrained by top-down structural processes and stimulate small-scale changes that positively impact the urban space and its citizens, as they are not subject to institutional policies and bureaucracies. That said, little is known about the potential of CIs to generate beneficial impacts for the Sustainable Development Goals (SDGs), as well as their leadership in creating fertile ground and contributing to a more sustainable urban city model [20].

This article reports on the work carried out on the basis of the collaboration agreement between InnoEnergy SE and Renault Spain “Development and Piloting of Assessment Model and Certification Scheme for Sustainable City Neighborhoods and Citizen Initiatives_FeliZiudad”. Within the framework of this project, the Barcelona Urban Ecology Agency (BCNecología) and the Universitat Politècnica de Catalunya (UPC) were contacted for the development of an assessment system for neighbourhoods and CIs; the UPC team was in charge of the development of an assessment system for CIs, with reference to their degree and contribution to the level of sustainability of the neighbourhood in which they are developed [21].

The main objective of the project, in relation to the evaluation of the CIs, was to create a system to measure the impact of the CIs on the sustainability of the neighbourhood, through an automatic evaluation process hosted on the FeliZiudad platform. This procedure would be carried out by a CI manager and its results would have an impact on the overall evaluation of the neighbourhood [21].

In order to understand this tool design process and the results obtained, we structured this article as follows: (i) the introduction discusses the concept of CIs, highlighting their role in the transition towards sustainability; (ii) a bibliographical review is framed mainly in the European context, where mention is made of a review article analysing evaluation models in the European context and how these differ from our proposal; (iii) the methodology used to create the evaluation system is detailed; (iv) the development of the evaluation proposal is given; (v) the evaluation process of CIs in Spain is discussed; (vi) the results of the case studies and discussion around them are given; and (vii) the conclusions of the evaluation process of CIs are presented.

2. Background

The first widespread and organised collective movement arose as a reaction to excessive prices and poor-quality products, especially in the food sector. The modern cooperative movement emerged in Rochdale, in the northwest of England, in 1844, in the midst of industrialisation. Although the first cooperatives date back to an earlier period, at the hands of craftsmen, it was at this time that a wave of cooperative movements emerged, based on the principle of “self-help by the people” without distinguishing between consumers and producers [22].

The history of cooperative movements and community development initiatives are intertwined, as both movements have grown out of people’s need to collectively address issues affecting their daily lives. Cooperatives are an example of community-driven efforts to address economic and social inequality, while community development initiatives aim to promote social change by empowering communities to take charge of their own development. Both movements have been instrumental in addressing issues of poverty, access to goods and services, and social exclusion [23].

For instance, research has shown that cooperatives can have positive impacts on poverty reduction, food security, and women’s empowerment [23,24]. Similarly, community development initiatives have been found to be effective in promoting community participation, social cohesion, and sustainable development [25,26].

Over the past 30 years, tools have been developed to support the development of these integrated initiatives for community change and provide visibility for them. In 1995, a team of experts from the University of Kansas created an online tool to guide community processes on their path to sustainability, called the Community Tool Box. This tool provides a range of content and information within the framework of community development and health, enabling the management and evaluation of community initiatives. Although this tool does not focus on assessing the environmental impact of initiatives, it helps communities overcome the barriers that may arise in their search for community development, creating consolidated groups that work together to generate solutions to common problems. As such, cooperative movements and community development initiatives can benefit from each other’s experiences and practices to further promote sustainable development and social change [27].

On the other hand, in Europe, the Treaty of Lisbon, which amends the Treaty of European Union (EU) and the Treaty establishing the European Community, which entered into force on 1 December 2009, opens a new window for citizen participation by giving citizens the opportunity to participate directly in legislative development [28]. In the same vein, cohesion policy is the main investment instrument of the European Union [29], whose objective is to “support job creation, business competitiveness, economic growth, sustainable development and the improvement of the quality of life of citizens in all regions and cities of the EU” [30].

In 2020, the European Commission developed two pilot schemes to facilitate and enhance the active participation of citizens in social cohesion projects. In the first pilot scheme, the Organization for Economic Cooperation and Development (OECD) provided advice and training to the authorities managing the funds for the implementation of new initiatives to encourage citizen participation, in a coworking model where the needs and opinions of all the groups involved were considered. In the second pilot scheme, the European Commission earmarked EUR 250,000 to fund and support innovative CIs from small local civil society organizations. This favoured the inclusion of small and local civil society institutions in European funding programs, an action that generates citizens’ ownership of the changes brought about by the cohesion policies developed [31].

Simultaneously, in recent decades, CIs have emerged in Europe to address the many early 21st-century challenges facing society at the local level. Through them, citizens collectively strive to achieve common goals, based on the self-management of the participants. As a result, new collectives emerge and the associative fabric gains strength in the management of community needs, as well as in the improvement of their immediate environment [32].

In Spain in recent years, there has been a proliferation of such collectives and associations that empower citizens in the promotion and implementation of sustainable practices [33]. In this type of association, citizens play an active role in the self-management of resources and in the generation of collective spaces for the debate of local problems, leading projects to reconfigure the physical and social context in which they are located [34]. In this sense, several research studies attempt to analyse CIs, whether from the context [35,36], the typology [37,38], or the way they are managed [17,39].

The evaluation of these types of collectives and associations is scarce. Our review of the existing literature identified only three systems in Europe aiming at the measurement of the level of empowerment of citizens to modify and improve their environment. These are:

• Towards European Sustainable Societies (TESS)—United Kingdom, Italy, Spain, Finland, and Romania (2013–2016).
• Multidimensional assessment of the environmental and socioeconomic performance of community-based sustainability initiatives (MDA)—Italia (2019).
• Criteria for assessing the transformation potential of sustainability initiatives (CATPSI)—Germany (2019).

An in-depth review analysing these three evaluation models can be found in [40]. In summary:

In 2019, Celata and Sanna created an evaluation system assessing the impact of initiatives on their surroundings in four areas: environmental, social, economic, and community development. They evaluated 37 initiatives and classified them according to their typology: food cooperatives, solidarity shopping collectives, community gardens, community energy, recycling, and mobility initiatives [18].

In 2013, the European research project TESS (Towards European Social Sustainability) was launched, which analysed the role of community initiatives on the road to sustainability in cities, through the creation of an evaluation system and the subsequent analysis of 63 case studies, which made it possible to monitor and report on the social, political, economic, technological, and environmental impacts of community initiatives, as well as their savings in carbon emissions [20].

Finally, the project Kriterien zur Bewertung des Transformationspotenzials von Nachhaltigkeitsinitiativen (criteria for assessing the transformation potential of sustainability initiatives) developed a system for assessing the sustainability and transformation potential of initiatives in order to reveal their value and facilitate recognition and support by public administrations [41].

The analysed systems share a common structure comprising criteria, dimensions, indicators, and a questionnaire. However, they differ in their weighting of parameters, namely, innovation, resilient communities, recommendations to administrations, organizational capacity, scalability, transformation potential, and political dimension. The evaluation of initiatives in these systems is time-consuming due to the need for extensive interviews involving up to 130 highly specific and lengthy questions. Notably, while sustainability is a primary focus, these systems do not prioritize the improvement and transformation of urban space.

In our system, we incorporate the spatial dimension into the evaluation of citizen initiatives, recognizing its significance in influencing urban dynamics. Our aim is to develop an accessible tool that all initiatives can use to self-assess and reflect on sustainability. To ensure robustness and inclusivity, we are currently collaborating with experts from various fields and stakeholders to create a more holistic and comprehensive tool. We aim to improve the design to be scalable and replicable, with flexibility to adjust the catalogue of criteria, indicators, and questions based on its application.

3. Methodology

The evaluation system for CIs created has followed a number of phases over the course of the project (about 2 years), which are depicted in Figure 1 and described below.

3.1. Step 1. Initial Approach

The first phase was to identify previous relevant works by analysing the publications in the databases and repositories (Scopus, Web of Science, Dialnet, Oxford Journals, UPCommons, and Recyt) as well as consulting the social academic networks (Academia.edu and ResearchGate) and the Google Scholar search engine, using, initially, two keywords: “Citizen initiative” and “community participation”. Due to the limited information obtained, other keywords were included in the search: “Self-assessment + neighbourhood impact”, “Community participation”, “Community initiatives”, “Indicators + neighbourhood impact”, “Community participation + evaluation”, “Neighbourhood impact + sustainability”, “Community self-management + neighbourhood improvement”, “Citizen self-management + appropriation of public space”, “Citizen self-participation”, “Community empowerment + self-assessment”, and “Citizen empowerment”. All of these were searched in Spanish, English, French, and German over a period of two months. This search resulted in the identification of the three evaluation models mentioned before.

3.2. Step 2. Criteria Selection

For the selection of the criteria, FeliZiudad researchers (professionals in architecture, urban sociology, and materials) made a review of the existing literature and identified the most relevant evaluation criteria among all the evaluation systems reviewed (the three specific ones but also other relevant systems, both theoretical and applied). In the case of all three systems, the criteria had been selected in processes of consultation, literature review, surveys, and validation with experts and stakeholders of the initiatives, for example:

In TESS, consultations and interviews were conducted among the project researchers (natural and social scientists) and the actors closest to the initiatives, and a ranking of criteria was collaboratively elaborated according to their impact and importance in the sustainable development of the community.

In MDA, surveys were carried out in which the initiatives themselves identified those fundamental aspects that mark their objectives. From all the aspects identified, two exclusion criteria were established: on the one hand, the criteria had to have been considered important by a minimum of 65% of the initiatives surveyed and to have obtained, in terms of importance, an average of 7–10 points; on the other hand, these criteria had to be able to be converted into measurable evaluation indicators.

Finally, CATPSI analysed various scientific articles and 14 existing sustainability evaluation systems, mainly in the German context, in order to define the evaluation criteria [41].

From there, the FeliZiudad team set up a discussion table to select those criteria that were most related to the CIs and those that have a potential to be worked on by them. Among the criteria selected were the impact of the initiative on the public space, networking, the robustness or maturity of the initiative, change through learning and training, organisational structure, creation of quality employment, sustainable commitment, human resource orientation, and the quality of the initiative. In this way, three general blocks were established, which contemplated these criteria.

3.3. Step 3. Tool Design

For the creation of the evaluation tool, three levels were structured: blocks, topics (evaluated by indicators), and, finally, a series of questions responding to these indicators. In order to establish this catalogue of questions, indicators were defined and objectives were established for each indicator, so that the questions would be as coherent as possible with the indicator and could be evaluated correctly.

3.3.1. Evaluation Blocks

The proposed sustainability assessment system is organised in three blocks, each covering a potential area of impact of a sustainable CI: metabolic efficiency, social cohesion, and transformational potential, allowing us to address social, environmental, economic, spatial, and urban metabolism aspects. In addition, similar to the TESS system [40], a fourth domain was included, which is not scored but serves to filter those initiatives that meet the minimum evaluation criteria, as detailed in Section 4.1.1.

3.3.2. Topics and Indicators

Once the blocks were established, these were split into topics, which, in turn, were characterized using a range of relevant indicators that had to be able to measure the performance of CIs in that specific topic. These indicators were chosen on the basis of existing literature and a process of analysis by FeliZiudad project members.

In the evaluation system, 33 indicators were established and defined. The indicators were distributed so that each block had a similar number of indicators.

3.3.3. Questionnaire and Weighting System

Once the indicators were defined, the data collection method was determined, which usually takes the form of a questionnaire.

A total of 79 basic questions (objective) and 40 supplementary questions (subjective, depending on the typology of the initiative) were established. These questions were carefully selected, trying to be coherent with the indicator to be evaluated, collecting as much information as possible and using concepts that can be understood by the initiative. This resulted in a total of 119 questions distributed in the defined blocks. The rating system allowed scores between 0 and 100 points for the basic questions, and between 0 and 50 points for the supplementary ones.

3.4. Step 4. Selection of Case Studies

To verify the evaluation system, a series of case studies were selected to provide feedback on the process and to detect its strengths, weaknesses, and possible improvements.

First, five CIs were selected from those previously registered on the FeliZiudad website (this page is currently unavailable) (project page). The selection was made, aiming at the greatest diversity of cases in terms of activity, size, area of influence, and age of the CIs. This evaluation was carried out based on the information available on the internet.

3.5. Step 5. Results and Feedback

Then, an analysis of the results was carried out and improvements were made to the system. Therefore, the application of the evaluation system was carried out in a peer process, with two to three researchers (from the fields of architecture, materials and urban sociology) evaluating each initiative independently, and in a collective review process afterward, agreeing on a consensus regarding the final results for each case. This process allowed a critical analysis on the indicators and questions and minor adjustments in the weighting system.

3.6. Step 6. Changes to the System and New Case Studies

In this step, the questionnaire was simplified, reducing it to around 75% of the original questionnaire. The simplification was motivated by the evaluation time needed to answer the 119 questions, which was in the range of 2 h, so one of the main objectives was to achieve a more reasonable time, around 25–35 min. Details of the process can be found in Section 6.2.

3.7. Step 7. Dissemination of Results

The project sought to disseminate the results through presentations, conferences, seminars, and congresses [42]. Of particular note is the article reviewing the current systems for evaluating community-based initiatives, analysing their creation process, barriers, and future prospects [40].

4. Evaluation Tool

The proposed evaluation tool consists of four blocks. The first block defines the minimum criteria that an initiative must meet to be considered for evaluation, with a total of five indicators, while the following three evaluation blocks contain 33 specific indicators to be evaluated and scored through 119 questions and their predefined answers.

The questions require single- or multiple-choice answers, which facilitates the automation of the system in an online format. The few additional questions that allow for open answers do not score in the evaluation system but allow for qualitative information relevant to the understanding of the activities carried out by the evaluated initiatives.

4.1. Evaluation Blocks

The system for assessing the sustainability of initiatives is organized into four blocks. The first one (block A) is not scored but serves to identify in a first step those initiatives that meet the minimum criteria for assessment from those who do not (Figure 2). The three other blocks are the main evaluation blocks, identified from the review of existing assessment systems. Each covers an area of impact: (1) Social Cohesion, closely linked to the Social Solidarity Economy (SSE) and the level of integration in the neighbourhood, (2) Metabolic Efficiency, related to resource management and environmental quality, and (3) Transformation Potential, linked to the capacity of CIs to influence new models, practices, and sustainable habits.

4.1.1. Block A. Minimum Criteria

The first block of the evaluation system incorporates the minimum criteria that initiatives must meet in order to be evaluated, both in relation to the type of organization and the activities they carry out. Thus, in order to be evaluated in the FeliZiudad system, initiatives must fulfil all the conditions mentioned below:

• To have, as their main objective, the promotion of the sustainability of the neighbourhood and/or the community in which they operate.
• To be a nonprofit organization (formal or informal).
• To be made up of at least two people (single-person initiatives are not included).
• To have carried out activities with tangible results prior to the evaluation process.

The results in this block will not add points to the overall evaluation of the initiative but will serve as a filter for its participation or exclusion in the evaluation.

4.1.2. Block B. Social Cohesion

The social dimension is often the most relevant impact of CIs. These are often based on actions that generate collaborative networks between people with diverse realities and common objectives or visions. The human factor is the main resource of the initiatives and, for this reason, the initiatives generally pay special attention to the inclusion of disadvantaged groups and to the creation and care of the community, regardless of the specific objective they pursue.

The Social Cohesion block includes 10 indicators with 44 questions that evaluate the impact of the initiative on the strengthening and care of the community and the physical environment in which it is developed.

4.1.3. Block C. Metabolic Efficiency

The second block refers to the contribution of initiatives to the improvement of the metabolic efficiency of the neighbourhood, both at the household level and at the level of urban metabolism. This block analyses the capacity of the initiatives to influence aspects such as the circular economy, waste reduction, energy consumption, energy self-sufficiency, etc., of the neighbourhood.

The Metabolic Efficiency block includes 11 indicators with 31 questions that evaluate the impact of the initiative on the efficient management of resources and the reduction of waste and emissions.

4.1.4. Block D. Transformation Potential

The transformative potential generated by the initiatives is one of the most valuable contributions of this type of citizen organization in the holistic processes of transforming our society and our cities towards more sustainable models [16]. Transformation potential is defined as the capacity of an initiative to create a sustained impact over time in the community in which it participates, whether through influencing people’s daily habits, generating spaces for debate, or activism aimed at transforming existing policies and regulations in terms of social rights and environmental protection.

The Transformation Potential block is made up of 12 indicators with 44 questions that evaluate the impact of these in terms of changes in habits and policies, collaboration networks, the dynamism of public space, the creation of new facilities, etc.

This distribution in blocks makes it possible to organize both the topics and the indicators so that it is possible to choose the degree of aggregation in which the evaluation results are presented: overall score (a single value), score by blocks (3 values), score by topics (13 values), or score by indicator (33 values).

4.2. Indicators

An indicator is defined as a measuring tool that allows us to evaluate in a simple and reliable way the fulfilment of the defined objectives and the achievements or development of an initiative or intervention proposal [17].

For the evaluation of the level of sustainability of the CIs, 33 indicators were defined. These indicators were selected because, according to the literature review carried out, they are essential in the study of the proposed evaluation blocks (Social Cohesion, Metabolic Efficiency, and Transformation Potential). The assessment system is designed to break down information or areas of analysis into manageable “blocks” or “topics” and use “indicators” to qualitatively measure CIs (

Table 1. Evaluation blocks, topics, and indicators of the methodology.

Topics	Indicators
BLOCK A. MINIMUM CRITERIA
AA1 Organization	AA1.1 Type of organization
	AA1.2 People
	AA1.3 Track record
AA2 Activity	AA2.1 ActivitiesAA2.2 Impact
BLOCK B. SOCIAL COHESION
BB1 Coexistence and social participation	BB1.1 Social inclusion index
	BB1.2 Gender parity
	BB1.3 Accessibility and change of mobility model
BB2 Empowerment	BB2.1 Level of self-management
	BB2.2 Level of transparency
BB3 Relationship with the environment	BB3.1 Level of participation in the social fabric of the neighbourhood
	BB3.2 Revaluation of public space
	BB3.3 Protection of the existing sociocultural fabric
BB4 Employment and social economy	BB4.1 Participation in the SSE (social and solidarity economy)
	BB4.2 Quality job creation
BLOCK C. METABOLIC EFFICIENCY
BC1 Materials	BC1.1 Reduction of material consumption
	BC1.2 Circularity of materials
	BC1.3 Low material impact
BC2 Energy	BC2.1 Energy efficiency effort
	BC2.2 Level of energy self-sufficiency through renewable energies
BC3 Water	BC3.1 Reduction of water consumption
	BC3.2 Circularity of water
BC4 Air quality	BC4.1 Reducing the carbon footprint of mobility
	BC4.2 Creation of urban green areas
BC5 Waste	BC5.1 Waste reduction
	BC5.2 Waste management
BLOCK D. TRANSFORMATION POTENTIAL
BD1 Social innovation	BD1.1 Encouraging behavioural change
	BD1.2 Promotion of new policies
	BD1.3 Holistic view of sustainability
BD2 Creating debate in society	BD2.1 Transformative learning
	BD2.2 Dissemination and replicability
	BD2.3 Collaborative networking
BD3 Robustness of the initiative	BD3.1 Consistent objectives and results
	BD3.2 Support network
	BD3.3 Financial sustainability
	BD3.4 Continuity of the initiative
BD4 Activation of public space	BD4.1 Dynamism of public space
	BD4.2 Creation of new facilities

). The use of a modular framework offers several advantages, such as flexibility, scalability, and ease of use. The system can be adapted to the realities of different CIs. The use of indicators also provides a standardised way of measuring progress or performance in different areas, allowing for more effective monitoring and evaluation.

Overall, this allows users to better understand and manage the data derived from this evaluation system.

4.3. Rating System

A catalogue of 119 questions (79 basic and 40 supplementary questions) was established for the three blocks B, C, and D, allowing scores between 0 and 100 points (basic) and between 0–50 points (supplementary). The questions are divided into basic questions and supplementary questions, with different ratings and weighting systems.

The basic questions are distinguished by the objectivity with which their answers can be interpreted and assessed in terms of their contribution to a greater degree of sustainability of the neighbourhood and its social fabric.

The supplementary questions are considered to be of great interest for the evaluation of initiatives, but the objectivity with which they can be evaluated is conditioned by the type of initiative. Therefore, for the final score, a weighting of this score (0–50 points) with a factor of 0.4 is proposed, which allows for an additional score of 20 points in the final calculation, totalling a maximum of 120 points (100 points for basic questions and 20 points for supplementary questions).

4.4. Evaluation Workflow

The evaluation workflow is structured in a series of steps that require the intervention of a key actor of the initiative or a key actor of the FeliZiudad project. Figure 3 details the steps to carry out this evaluation. In the data entry process (create user profile and fill in the minimum criteria), a person in charge of the initiative will fill in the data, which will take 2 min and 5 min, respectively; if the initiative complies with the minimum criteria detailed in Section 4.1.1 “Block A. Minimum criteria”, the person in charge of the initiative will be able to fill in the questionnaire corresponding to blocks B, C and D, which will take 2 h (this time corresponds to the questionnaire of 119 questions); from this moment on, a whole validation process will be carried out by an external evaluator who will check the veracity of the information entered in the questionnaires through the documentation provided by the initiatives.

5. Evaluation of CIs in Spain

Figure 4 shows the flowchart of the system evaluation process with case studies, which lasted approximately one year, distributed as follows: 1.5 months in the evaluation of the initial five initiatives (this includes the collection of information on the initiative); 2 months for the simplification process detailed in Section 6.2; 4 months in the evaluation of 25 CIs, as in the evaluation of the five CIs, this contemplates the time for information collection and, finally, the data analysis and discussion which merited 4.5 months.

The data collection process was carried out by analysing and searching for information through digital media (the initiative’s website, social networks, publications on different pages, etc.). An attempt was made to gather reliable information that could answer all the questions evaluated.

For the evaluation of citizen initiatives, teams of three people (FeliZiudad researchers) were formed to evaluate the same initiative, which made it possible to compare the results of each of them and calculate an average.

5.1. Evaluation of 5 Initiatives

Five CIs were selected from those previously registered on the FeliZiudad website (project page). The selection was made looking for the greatest diversity of these with respect to their activity, size, area of influence, and age. This evaluation was carried out by members of the FeliZiudad team on the basis of the information available on the Internet, because we wanted to have this first approximation and check the feasibility and suitability of the system.

The selected CIs focus on (1) sustainable mobility, (2) neighbourhood improvement, (3) waste management, (4) energy management, and (5) cohousing (this classification was made by the authors according to the main activity carried out by the initiative; the names are not mentioned by copyright).

This evaluation was carried out using the 119-question questionnaire, with a maximum score of 120 points.

5.1.1. Sustainable Mobility Initiative

This CI was founded in 1987 by a group of urban and touring cyclists who sought to promote the use of bicycles as a means of transport in Barcelona.

It was constituted as a people-centred working cooperative with a social and environmental commitment in the area of sustainable mobility, promoting the bicycle as an economical, ecological, and healthy means of transport. Its activities are centred around the use of bicycles, offering training courses in cycling, bicycle repair, safety, physical spaces for the self-repair of bicycles, excursions, and rides, as well as strengthening the social and associative fabric of the neighbourhood (Figure 5).

The CI has established a working network in collaboration with the social fabric of the neighbourhood, the local productive fabric, educational institutions (schools, institutes, educational centres, universities, etc.), public administrations, companies, etc. [43,44]. Figure 5 represents the different positive impact of the initiative in different categories. The central grey circle represents the evaluated initiative, from which four coloured circles correspond to the general benefits of the initiatives, and, finally, the more specific benefits are derived. This explanation of the configuration of this figure is the same for Figure 6, Figure 7, Figure 8 and Figure 9.

5.1.2. Neighbourhood Improvement Initiative

This initiative was born out of the neighbourhood’s struggle to obtain public facilities, such as a school, a health centre, a park, and a library, for a neighbourhood that lacked these spaces, but where there were unused spaces that could serve this purpose and that were rooted in the neighbourhood and in the city’s architecture. After years of waiting and the delay in the execution of the General Metropolitan Plan of Barcelona (MPGM) of 1975, which stipulated that this area would be used for facilities, social housing, and green zones [45], neighbours organised themselves and set a deadline for the start of the works, 1 June 2011, otherwise they would have to occupy the site in a movement they called “ticking clock”, which began on 24 January 2010. Before the date was reached, the neighbours were called to a dialogue table that ended in the cession of one of the spaces [46,47,48].

Since then, the initiative has been working to create spaces and facilities for social housing, parks, green areas, and urban allotments, promoting cooperatives that strengthen social cohesion, empower the social fabric of the neighbourhood, and create a solid and common neighbourhood network. It is a completely open organisation, which is constituted as an intergenerational and heterogeneous meeting space and common facility [32,33]. This self-managed initiative has given back to the neighbourhood an identity, influencing essential aspects such as social welfare, the environment, neighbourhood and urban development, and creating new management models (Figure 6) [49,50].

5.1.3. Waste Management Initiative

This initiative was created in 2015 by four women at risk of exclusion, as a result of a training and social work process initiated by the organisation “Architects without borders”. Their work is based on the management of used household oil (collection, storage, recovery, and reuse), to be used as an energy resource. This initiative brings social benefits (inclusion of vulnerable groups and social cohesion), economic benefits (creation of new jobs), and environmental benefits (reduction of waste and pollutants) (Figure 7) [51].

5.1.4. Energy Management Initiative

This initiative stems from an interest in promoting renewable energies, specifically solar panels. It seeks to influence the economic and energy management model, providing environmental, social, and economic benefits (Figure 8).

5.1.5. Cohousing Initiative

This initiative arose from a neighbourhood movement to recover an area of the neighbourhood for collective use. Its first steps date back to 2012, although it was formally established in 2014 when a group of people of different ages and disciplines sought solutions to solve the housing problem, although they encountered the vicissitudes of finding decent housing.

They considered the idea of creating a housing cooperative, for which they investigated different models around the world. Once they had decided to do so, they began the process with the aim of meeting the need for decent housing in the neighbourhood, which in turn would be affordable, sustainable, and strengthen community ties through the equitable sharing of these facilities (Figure 9) [45,52].

6. Results and Discussion

6.1. Results of the Evaluation of 5 Initiatives

Table 2. Results of the evaluation of 5 CIs.

			Blocks
			Block B		Block C			Block D
CIs		Basic score	Suppl. score (0.4)	Results	Basic score	Suppl. score (0.4)	Results	Basic score	Suppl. score (0.4)	Results	Final Score
Sustainable mobility		19.0	10.5	23.2	12.5	7.0	15.3	23.5	13.5	28.9	67.4	Medium contribution
Neighbourhood improvement		15.0	7.0	17.8	10.5	5.0	12.5	19.0	11.0	23.4	53.7	Medium contribution
Waste reduction		12.5	4.5	14.3	5.0	1.5	5.6	19.5	10.5	23.7	43.6	Medium-low contribution
Energy management		4.0	4.5	5.8	7.0	2.5	8	9.5	6.5	12.1	25.9	Low contribution
Cohousing		14.5	11.0	18.9	19.0	6.0	21.4	22	11.0	26.4	66.7	Medium contribution

and Figure 10 present the results of the evaluation of each of the selected initiatives obtained after applying the procedures and criteria described in the previous sections.

In order to assess the level of contribution of the initiatives, the following ranking was established:

• <24.9 points: CIs with low contribution.
• 25–49.9 points: CIs with medium–low contribution.
• 50–79.9 points: CIs with medium contribution.
• 80–99.9 points: CIs with medium–high contribution.
• >100 points: CIs with high contribution.

The results of this first assessment round show that according to the established ranking categories, three of the five initiatives have reached a medium contribution (scoring 67.4 points, 53.7 points, and 66.7 points, respectively). One of the initiatives is rated with 43.5 points, considered as a medium–low contribution, and therefore some improvements would be recommended.

One of the initiatives is rated with 25.9 points, considered as an initiative with a low contribution and therefore would require many improvements for considering it as a CI with a high contribution to the sustainability of its neighbourhood.

The average scoring of all five CIs was 51.46, so if we evaluated the initiatives as a whole, they would pass the system being rated as an overall medium contribution. From this, we could draw the conclusion that if there are several initiatives working on different sustainability issues in a neighbourhood, their average score could be an indicator in order to understand their overall impact. Alliances and cooperation networks would probably allow them to strengthen and diversify their activities and possibly achieve better results.

6.2. Simplification of the System

One of the aspects that was identified in the evaluation using the 119-question catalogue is the fact that it required an execution period of 2 h, with more difficulties in block B, due to a greater degree of subjectivity in this block with respect to the others. All this added to the time needed to search for information on the initiative, which depended on how accessible the information was, as well as the degree of sufficiency and validity of this information to carry out the evaluation. In the case of information, these disadvantages would be eliminated when the process is outsourced, as the person completing the evaluation would be someone who knows or is familiar with the initiative (manager, coordinator, etc.). On the other hand, in terms of time, we decided to speed up the process, reducing the catalogue, in order to guarantee a more user-friendly evaluation. In this sense, some guidelines were established for this reduction:

• Maintain the same number of indicators.
• The questions should be clear and coherent.
• The contribution of an initiative to the improvement of sustainability and quality of life in the neighbourhood can be deduced from them.
• There remains a balance between the blocks and indicators:
• The total number of questions remains between 30 and 35 questions.
• Finally, the information can be verified via the web (at that time we could not contact the initiatives directly due to data protection).

At the end of the simplification process, 33 questions were retained, which corresponds to the number of indicators in the system. The score for each question ranges from 0 to 1, so the maximum score to be obtained is 33 points (Figure 11).

Once the simplification of the system was completed, a verification of the results obtained in the previous evaluation with 119 questions and the new simplified questionnaire of 33 questions was carried out. For this verification, one of the previously evaluated initiatives, called “Sustainable Mobility”, was selected. The objective of the comparison was to determine whether there were significant differences between the results obtained with the new set of questions and those obtained previously. The selection of this initiative was based on the availability of detailed information on its website, as well as on news and other third-party sites.

Table 3. Comparison between the evaluation system with 119 questions and the simplified system.

	Sustainable Mobility CI
	Questionnaire of 119 Questions					Simplified Questionnaire
Blocks	Basic scores	Supplementary score		Results	Percentages	Results	Percentages	Differences
			Factor 0.4
Block B	19.0	10.5	4.2	23.2	23.20%	8.5	25.76%	2.56%
Block C	12.5	7.0	2.8	15.3	15.30%	5.23	15.85%	0.55%
Block D	23.5	13.5	5.4	28.9	28.90%	10.75	32.58%	3.68%

shows the results obtained, in which it can be seen that in the section on the blocks a similar trend is maintained in the distribution of points, with block D having the best score, followed by block B, and finally block C. In relation to the results in general, we can see that in the evaluation with the simplified questionnaire the values are increasing. This is due to the fact that in the questionnaire with 119 questions, there are some that generate inconsistent answers and that there is a greater degree of ambiguity, which affects the result obtained, while in the simplified questionnaire, we tried to address each indicator with clearer and more concise questions.

6.3. Evaluation of the 25 Initiatives

Following the same procedure as in Section 5.1, 25 initiatives were selected from the FeliZiudad project website. Initiatives from different Spanish cities were included.

As in the previous process, this evaluation was carried out by three members of the team and using exclusively information available in the digital media, due to the pandemic situation in that moment and data protection issues that did not allow contacting the initiatives directly. For this evaluation, the simplified questionnaire of 33 questions, with a maximum score of 33 points, was applied.

Figure 12 details the results of the initiatives evaluated in Bilbao, Zaragoza, Valladolid, Barcelona, and Madrid.

In the city of Zaragoza, five CIs in the areas of animal care, art and culture, food and well-being, social model, and food sovereignty were evaluated. The initiative that achieved the highest score was the social model, which is a social market present in several locations in Zaragoza, with a strong social component and solid results, encouraging changes in habits and community support. It was followed by the food and well initiative, which is an urban organic garden with a marked impact on the social fabric and which promotes food sovereignty. With respect to the blocks, all CIs showed a good performance in block D, while in block C, the results were the poorest, except in the case of the social model initiative, which obtained 54.45% of the maximum score.

In the city of Valladolid, three initiatives were evaluated (water management, social cohesion, and awareness raising and education). The initiative focused on water management only obtained around 30% of the total score, as it is focused on a very specific action. In the case of the initiative focused on social cohesion, it achieved a good score in block D, as it is seen as an initiative with some robustness (funding sources, support networks, clear objectives, and continuity plan), and, finally, the initiative focused on awareness raising and education also achieved a good score in block D, but in block C, it obtained a score of 0, because they do not work and have no impact on aspects of metabolic flows efficiency.

In the case of Barcelona, three initiatives were evaluated, focused on awareness raising and education, neighbourhood improvement, and sustainable mobility. In general terms, we observed that, although these initiatives address different issues, their incidence in block C always remained below the other blocks., In contrast, in the case of block D, a good trend was maintained, practically approaching the maximum score possible, which shows that the CIs evaluated are robust and have the support of the neighbourhood and other entities.

In Madrid, three CIs were evaluated, two in the field of sustainable mobility and one focused on neighbourhood improvement. Two of them (second and third in Figure 12) obtained 15 points out of 33. The sustainable mobility initiative had more impact in block C, since its activities focus on improving environmental quality and boosting new mobility models. The second obtained a higher score in block B, as its activities have a direct impact on the cohesion of the neighbourhood.

Another aspect to highlight is that although the first and the third initiative work in the same area, the latter obtained a higher score since it has implemented strategies that seek to influence not only mobility but also other aspects, such as reduction of material consumption, waste reduction, dissemination, continuity of the initiative, and financial sustainability. Finally, the three initiatives performed well in block D, since they actively promote habit change, have the support of the community and other institutions, have a good level of transparency, and have strategies for their continuity.

Figure 13 details the results of the initiatives evaluated in Zamora, León, Palencia, Badajoz, Valencia, Córdoba, A Coruña, Canary Islands, Sevilla, Gijón, and Bilbao.

In the cases of Palencia (energy management CI), León (art and culture CI), and Zamora (art and culture) CI), these initiatives obtained low scores, for two general reasons, because they have an almost null incidence in block C, and in the rest of the blocks their incidence is low; this is due to the fact that these initiatives are one-off projects, which, although they have had favourable results, have not remained active to increase these benefits and have a greater impact on the indicators under study.

In Valencia and Badajoz, only one was evaluated for each city, in the area of the food and wellbeing and social model. The Valencia initiative, which is a vegetable garden, achieved a good score, in contrast to the Badajoz initiative, which is a soup kitchen. These differences may be due to the fact that although the soup kitchen is a solution that provides support to vulnerable groups, it does not have an impact on a change of habits and awareness, where groups are empowered and change their way of conceiving food, in addition to the fact that the vegetable gardens are a solution for renaturalising urban space. If we look at the score obtained by the Zaragoza garden and compare it with the Valencia garden, we realise that it is relatively the same, which gives us a clue that a garden is an interesting proposal to promote the sustainability of neighbourhoods.

In A Coruña and Cordoba, a waste management initiative and another focused on awareness-raising and education, respectively, were analysed. The two cases obtained almost the same total score and maintained similar scores in each of the blocks. It is interesting how waste management initiatives can have a greater impact on the social domain than CIs focused on raising awareness and educating. In this regard, the inclusion of groups at risk of exclusion and the participation in the social and solidarity economy were aspects that scored high in the evaluation tool.

One CI in Seville and one in the Canary Islands were evaluated. The CI in the Canary Islands has a broad scope as it works in several cities in the archipelago (La Palma, Tenerife, Gran Canaria, Fuerteventura, and Lanzarote), which focuses on blurring art and culture in the cities where it is present. In the case of Seville, the CI analysed focuses on the renaturalisation of spaces, creating a new leisure and wellbeing area. Block C was in the minimum ranges in both initiatives. This is due to the fact that in the case of Seville and the Canary Islands, these initiatives evaluated are very specific and do not affect aspects such as waste, water, materials, and energy management.

In the cases of Bilbao and Gijón, only one single initiative was assessed for each city, but with different areas of action: on the one hand, health and healthy living and, on the other hand, sustainable mobility, successively. Both initiatives obtained a higher score in block D, reaching a final score of 16 and 17.4, successively. In the case of block B, the Gijón initiative had a lower score than that of Bilbao, which may be due to the fact that the latter is more rooted in public space and community life, with a greater impact on the social cohesion of the neighbourhood.

6.4. Performance of CIs by Blocks and by Typology

According to Figure 14, it is noticeable that the block that reported the highest degree of compliance is block D (potential for transformation), as most of these CIs have been well welcomed in their cities and have received the support of the community, the city council, and other initiatives or related organisations. Most of the CIs have an active role in creating the basis for a change in thinking and reflection on sustainability, either from a specific area or from a set of actions dispersed among the areas that arise from the concept of sustainability (social, environmental, and economic). The second block in which most of the CIs perform well is block B (social cohesion). CIs, being the result of citizen self-management, understand the need to create neighbourhood collaboration networks and create transparent management mechanisms. In contrast, in the case of block C (urban metabolism), the initiatives do not tend to cover all aspects (water, energy, materials, and waste management), but, rather, they tend to focus on specific aspects with small actions.

On the other hand, with regard to the typologies of initiatives, it can be inferred from the analysis that those with the best performance (20–33 p.) are those initiatives that focus on issues such as sustainable mobility, neighbourhood improvement, social model, food and wellbeing; those with intermediate impact (10–199 p.) initiatives focusing on food sovereignty, awareness raising and education, waste management, social cohesion, naturalisation of spaces, and health and healthy living; and, finally, low impact (0–9.9 p.) initiatives focusing on water management, animal care, energy management, and art and culture.

From a scientific perspective, it is important to highlight that the analysis of the effectiveness of CIs should be approached from a holistic viewpoint that considers multiple factors that may influence their overall performance. In this sense, the previous paragraph identified the impact according to the typologies of CIs evaluated, but it is necessary to point out that the evaluation of their success also depends on aspects such as:

• The consistency of their objectives with the real needs of the community.
• The support received from the community, other organisations, and the administration.
• The public policies of the specific city.
• The spatial and beneficiary coverage.
• The versatility of its activities.
• Its trajectory.
• The degree of dissemination.
• The level of awareness and empowerment of citizens.
• The level of the city’s relationship with other contexts (cities) that can serve as examples or promoters of the same.
• Socioeconomic condition of the population.
• Depopulation.
• The ageing of the population.

Other aspects not mentioned in this article may be conditioning factors that hinder the development of the CIs but must be analysed in depth.

6.5. Evaluation of Activities vs. Evaluation of Organization

The developed evaluation system pretends to adapt to the diversity of initiatives and assess the different areas of sustainability they contribute to. Being a holistic assessment, the tool favours initiatives that have diversified actions and a broad range of impact on a neighbourhood. On the other hand, it evaluates aspects not only directly related to the activities but also, and above all, to the organisation, so that in reality what we are evaluating is a cultural change in the way things are performed: inclusive, taking into account the resources used and the waste generated, egalitarian, democratic, etc.

6.6. Full Questionnaire vs. Reduced Questionnaire

With regard to the system, several aspects could be ascertained from the study carried out with the general questionnaire with 119 questions. On the one hand, the tool provides us with valuable information both on the organisational level and on the activities carried out, which have a direct or indirect impact on the achievement of the objectives and aspects of sustainability that we have set out for the system. On the other hand, with regard to the time associated with its application, a long time is required for the search for information and the evaluation by the evaluators. A key aspect in this process is to reduce subjectivity on the part of the evaluators as much as possible, trying to work with the same information, and checking for elevated divergence in the different evaluations.

Once the system was simplified and a larger number of initiatives spread throughout Spain and of different natures were selected, the process was more agile, although more detailed information was lost regarding specific aspects of the activities they carry out, specific quantified outcomes, specific strategies for waste, water, energy management, aspects of language, etc.

6.7. Single Topic CIs vs. Diverse Topics CIs

Regarding the topic coverage and the evaluation results obtained, the following can be stated:

The evaluation tool, by its nature, tends to value initiatives that act in a diversified way, while initiatives that are very good and specific to a single topic (topics (energy management, waste reduction, etc.) have a very low evaluation.

On the other hand, more diverse CIs (neighbourhood improvement, sustainable mobility, social model, and food and wellbeing) tend to score higher, as the positive impact is usually distributed among the three blocks of the assessment tool in an ascending manner. Other aspects that can enhance these impacts of more diverse CIs even more are the support received from local administrations (local sessions, grants, etc.) and partnerships with other social entities, which create a range of new possibilities and a more comprehensive and cross-cutting approach to problems.

6.8. Potential Applications of the Tool

The evaluation model proposed for the CIs can be understood as an evaluation tool and, at the same time, as a continuous assessment tool for learning and improvement over time for the promoters of initiatives.

Detailed results of the evaluation can serve as inspiration for the continuous improvement of social, environmental, and energy-sustainability initiatives, both at the organisational level of the initiatives and at the level of their activities. For this reason, a questionnaire in digital format is proposed in which best practices and areas for improvement can be reported.

Given that this evaluation can be renewed periodically, the improvements can be incorporated and even serve as an incentive or condition to obtain the support of the administration or other interested parties, as well as to assess and disseminate the positive impacts of initiatives to the citizens.

7. Conclusions

The path towards sustainability is closely related to new institutional and political paradigms and emerging models of practices, technologies, lifestyles, attitudes, and values [53]. In this sense, the CIs, collectives where citizens self-organise through collaborative practices, networking, and knowledge transfer, come into their own, creating spaces for coexistence, reflection, and meeting where social innovation is the backbone. These multidisciplinary collaborative practices are developed on the basis of active citizen participation, improving individual and community life in the neighbourhood.

The main objective of the CIs is to tackle neighbourhood problems through self-management and neighbourhood empowerment, adopting more sustainable lifestyles [54]. These citizen models are transformed into solid platforms based on bottom-up processes where the social fabric is dynamised and, on many occasions, neighbourhood dynamics are transformed. Although this impact is known by the beneficiaries, it remains diffuse within the context of the city and even in some cases for the social fabric of the neighbourhood where the initiative is located.

These social dynamics require monitoring, reflection, and analysis of their activities and organisational process and the creation of political strategies, in order to understand and take advantage of the potential of the CIs to generate improvements in their environment, in the quality of life of citizens, and, therefore, in the sustainability of the community. This is where it makes sense to make these experiences visible and quantify their Impact through systemic evaluations that allow confirming their diverse positive effects on the sustainable development of communities.

In order to establish an evaluation framework, it is necessary to have a holistic vision of all the dimensions addressed to a greater or lesser extent by the CIs and to build an interdisciplinary, integral, and collective knowledge that helps to understand these processes and, at the same time, promote them. In this process, academia, public administration, and civil society are called upon to participate as promoters and final beneficiaries. To this end, channels of interaction must be opened between civil society and the administration, with the support of experts who can mediate and channel these learning and decision-making processes.

A simple, robust, and inclusive evaluation model for CIs, on the one hand, allows for a better understanding of their holistic contribution to solving neighbourhood issues and promoting healthy lifestyles, and, on the other hand, enables benchmarking between different initiatives and fosters their continuous improvement. Undoubtedly, this evaluation system should be inclusive, highlighting the role of the evaluated initiatives as good practices.

The presented evaluation system was developed based on the in-depth analysis of existing evaluation systems, with special reference to three international proposals. Moreover, the system was designed within the framework of a collaboration agreement in Spain, focusing on the evaluation of Spanish CIs. However, although this proposal takes into account European references in its development, it is intended to be applicable to all types of CIs. Indicators with their universality, in most cases, can allow quantifying this aggregate contribution to the overall sustainability of a neighbourhood, but it is interesting to evaluate the sociopolitical context of the environment where the initiative is developed as an input to the process of identifying barriers and opportunities that can promote this social innovation [55]. For example, it may happen that in some countries, public policies are not oriented to promote and support this type of participation or that citizens are not aware or do not have the means to implement them. Another aspect to consider is the adaptation of the catalogue of questions depending on the reality of the initiative and its environment, i.e., it does not make sense to ask about waste management in a country where this is very well solved, but rather to focus on those sensitive aspects of the city.

In a next step, the presented system needs to be implemented and validated in different contexts, creating a framework for dialogue with the different agents involved and experts in the different areas addressed in the CIs. This will allow the generation of an even more balanced tool that encourages open and plural citizen processes, highlighting the activities it carries out as good practices, since the contribution of one initiative cannot be assessed as better than another one without taking into account a holistic analysis of the local context, the social structures, the level of citizen participation, the decision-making structures, and its potential to create new habits and learning, among many other factors. A series of optimization processes are currently underway in order to achieve this balance. We consider this implementation in countries with other socioeconomic, cultural, and political particularities in Latin America.

To address subjectivity, we involve external agents in the evaluation process, ensuring an objective and fair evaluation of initiatives. Our aim is to create a simpler and more compact methodology that facilitates a friendly and comfortable interaction between the interviewer and the user, and between the platform and the user in digital format.

Finally, continuous feedback is prioritized to ensure the effectiveness and relevance of the system over time, promoting a culture of sustainability and innovation in urban development. The authors are convinced that the presented evaluation tool for CIs is a novel and interesting proposal with the potential to support decision-making processes for the promotion of CIs based on a holistic analysis of their benefits and the match with identified needs of the neighbourhood.