Development of an Evaluation Framework for Smartness and Sustainability in Cities

Abstract

The transformation of cities into smarter and more sustainable environments is essential towards achieving the objectives of the decarbonization of the economy. In this sense, holistic strategies are required to design and implement urban regeneration strategies. The EU-funded REMOURBAN project has developed an Urban Regeneration Model which provides the mechanisms to implement more efficiently these processes, integrating technologies, business models, management procedures and evaluation mechanisms, where the evaluation is sought as the main supporting mechanism throughout the various phases of the city transformation process. The framework developed considers two levels of evaluation: city level, to assess both sustainability and smartness of the city as a whole from a comprehensive and integrated perspective, and project level, to provide a clear identification of the impact of implementation of technologies and solutions on three key priority areas (sustainable districts and built environment, sustainable urban mobility and integrated infrastructures and processes) aimed at achieving the city high-level goals. This paper introduces the Urban Regeneration Model and describes the evaluation framework and its implementation in the city of Valladolid (Spain).

1. Introduction

Global population living in cities is increasing every year, so urban development and optimization is a challenge that has been explicitly considered as one of the 17 Goals to Transform Our World included in the Sustainable Development Goals (SDG) (Goal 11: Make cities inclusive, safe, resilient and sustainable) defined by the United Nations [1]. In addition to this specific goal, cities can also contribute to achieve other SDGs.

To achieve these Goals, new policies shall be implemented to promote the economic and social development by improving the quality of life of citizens, taking care of the environment and optimizing the use of natural resources. In the current technology-based society, the use of information and communication technologies (ICTs) can, and should, play a key role for improving sustainability on cities.

In this context, the concepts of “smart” and “sustainable” have been used interchangeably, but they are not synonymous: sustainability is a goal per se but smartness (intelligence or the use of technologies in cities) is a means for sustainability. When both concepts are combined, a “smart sustainable city” is obtained, which can be defined in the following way according to UNECE-ITU [2]:

“A smart sustainable city is an innovative city that uses information and communication technologies (ICTs) and other means to improve quality of life, efficiency of urban operation and services, and competitiveness, while ensuring that it meets the needs of present and future generations with respect to economic, social and environmental as well as cultural aspects”.

Therefore, a smart sustainable city is an advantageous city with an economic sustainable urban development, environmentally responsible, with a high quality of life for all its citizens using information and communication technologies in a highly-efficient way [3,4].

REMOURBAN (Regeneration Model for accelerating the Smart Urban Transformation) is a large-scale demonstration European project (Grant Agreement No 646511), the purpose of which is to accelerate the urban transformation towards the smart city concept taking into account all aspects of sustainability. Keeping in mind this purpose, an Urban Regeneration Model has been developed and validated in the three lighthouse cities of the project (Valladolid-Spain, Nottingham-UK and Tepebaşı/Eskisehir-Turkey) accelerating the deployment of innovative technologies, organizational and economic solutions to significantly increase resource and energy efficiency, improve the sustainability of urban transport and drastically reduce greenhouse gas emissions [5].

The Urban Regeneration Model (Figure 1) defines a holistic methodology composed of several phases and decision-making processes that aim to support the understanding of the city objectives and needs in order to implement a set of strategies for a sustainable and smartness-oriented regeneration of the city.

Three key priority fields have been identified for the Sustainable Urban Regeneration process where REMOURBAN provides a catalogue of solutions aimed at packaging the technologies with their related financing and societal aspects. These fields are “Sustainable Districts and Built Environment”, “Sustainable Urban Mobility” and “Integrated Infrastructure and Processes”. These three priority areas have been those selected and integrated into the model as energy in buildings, mobility and their intersection with ICTs represent three of the main assets that need to be addressed when facing sustainable and smart projects in cities.

Management, Evaluation and Finance are the three key frameworks defined within the overall model, which establish the main enablers for the city transformation, and last but not least, four phases are the guiding thread of the model that, in addition to guiding the user in the process of transforming the city from a methodological point of view, include tools designed to support this process and facilitate the transformation of cities into more sustainable and smart environments.

The evaluation framework as mentioned before is one of the three frameworks defined in the Urban Regeneration Model and can be considered as the main supporting mechanism throughout the various phases of the city transformation process. The framework considers two levels of evaluation: city level, to assess both sustainability and smartness of the city as a whole, from a comprehensive and integrated perspective, and project level, to provide a clear identification of the impact of implementation of technologies and solutions on the three key priority areas (sustainable districts and built environment, sustainable urban mobility and integrated infrastructures and processes) aimed at achieving the city high-level goals.

The objective of this paper is to show the Sustainability and Smartness evaluation approach at city level that REMOURBAN project has carried and also to show the results of the evaluation at city level of Valladolid, one of the three lighthouse cities of REMOURBAN project as example of how to implement the evaluation process at city level and how to take advantage of the results to define new strategic objectives for the city. The evaluation of the Sustainability and Smartness of Valladolid shows how the REMOURBAN project together with other external factors (out of REMOURBAN scope) have taken influence in the progress and transformation of the city of Valladolid.

2. Contextualization of the Sustainability and Smartness Evaluation Approach

An evaluation framework is needed to assess smart and sustainable features within the urban development of cities. Therefore, in the process to become a smart sustainable city, establishing a reliable metric is a key point to support cities to identify strengths and weaknesses and consequently set priorities for action. Otherwise, what is not measured cannot be known and thus cannot be managed. Related to that, particularly during the last decade, multiple initiatives have been proposed for assessing smart and sustainable cities, most of them defining a list of Key Performance Indicators to evaluate different characteristics or facets of the cities. Some of these examples are schemes of indicators developed to evaluate the Sustainability of Urbanistic Activities [6], schemes developed by EU-funded projects that aim at collecting and analyzing existing frameworks and developing integrated approaches [7], recommendations on KPIs for smart and sustainable cities related to the Sustainable Development Goals [8,9], indicators for measuring energy poverty proposed by the European Commission [10], existing ISO standards for city services and quality of life [11] or initiatives aimed at aggregating data from projects under the umbrella of Smart Cities and Communities [12]. All these existing schemes deal with a specific set of indicators that measure smartness or sustainability with a focus on specific aspects as quality of life, sustainability towards Sustainable Development Goals, etc. The scheme proposed within this paper has been developed based on the analysis of these existing schemes where the selected indicators offer a consistent framework aimed at integrating all relevant indicators that can measure and evaluate the evolution of sustainability and smartness in cities, while offering a method for aggregating them into the composite indices that can support the design and evaluation of Smart and Sustainable City Policies. The main key point to be considered in the definition of an adequate evaluation framework to assess smart sustainability cities is the idea that sustainability is a goal, but smartness is a means for sustainability.

Sustainable cities offer higher standards of quality of life and well-being of its citizens, providing access to basic services and improving their participation in making-decision processes. In order to achieve these benefits, technological solutions are part of the answer, contributing to the efficient use of resources, improving public services and making life easier for the citizens. Nevertheless, a city with a high level of technology or intelligence is not necessarily a sustainable city, so sustainability is a broader concept that takes into account all processes in the city as a whole.

Regarding metrics, composite indicators [13] can be built in the form of to quantify certain criteria. Thus, a sustainability index can be defined for measuring the sustainability of the cities, using indicators to assess the main characteristics of the city, according to sustainability criteria. Furthermore, the influence of smartness or intelligence on the sustainability of the cities should be assessed and for that, there are mainly two different approaches:

• The first one is to define specific indicators related to technology or ICTs and to include them in the global index [14], defining a “sustainalligence” index, i.e., a “sustainability and intelligence” index.
• The second one is to try to analyze the effect of smartness indicators on the sustainability indicators, taking into account that one specific smartness indicator can affect many different sustainability indicators, and even the effect can be positive or negative from one to another. Besides, this effect is difficult to isolate.

It is considered that the most appropriate approach for assessing this influence is the second one, because the application of the first methodology can result in strange values for the index that combine indicators of many different levels.

The second approach, however, requires a wide database of indicator’s values sampled in the same conditions over a long period of time, for applying analytic and statistical techniques to get relationships between indicators and try to isolate the impact of smartness on sustainability.

REMOURBAN evaluation approach is an intermediate vision between the two previous ones in order to make feasible the assessment objectives assigned to the project.

As the main goal is to evaluate sustainability of cities, a Sustainability Index (Su) is defined, but in order to make a complementary assessment, a secondary index is also proposed for the cities, a Smartness Index (Sm), that tries to measure the technological advance of the cities in the main areas of interest of the cities.

Figure 2 shows an overview of this concept of the evaluation framework, considering smartness as a concept that can have an impact of all facets on the cities but play an independent role in the sustainability concept, as mentioned above.

3. Sustainability and Smartness Evaluation Framework

The framework is based on the definition of global indices as the calculation and aggregation of indicators grouped according to categories and priority objectives (Figure 3). Taking into account that the main goal for cities is to improve its sustainability, the main index is defined at City Level to assess its sustainability, the Sustainability Index (Su). On the other hand, considering smart technologies and services as means that can contribute to achieve sustainability goals, an additional index is defined for cities, the Smartness Index (Sm).

Following the scheme showed in Figure 3, the design of an index has the main objective of developing a single metric to measure the accomplishment of a specific purpose, a model for effective communication.

Measurable objectives are related to the main purpose of the index but are focused on more specific objectives that can be more easily evaluated. Usually, they are associated with global aspects or issues that can aggregate some core categories used for making an adequate organization of the indicators, which are the key variables, measured from real or calculated data. Indicators are used to assess a specific characteristic and they are helpful to diagnose problems and discover patterns.

To show the relationships between them visually, Figure 4 presents the general scheme showing particular Measurable objectives and Core categories and its relationships with indicators for the Sustainable Index (Su) at City Level. This scheme is analogous to that used for the Smartness Index (Sm), with both indices sharing the same structure, as can be seen in Figure 5.

In accordance with the schemes shown above, Sustainability and Smartness Indices share the same Measurable Objectives and Core Categories but may only be differentiated through the concept of smartness.

The first two measurable objectives defined for both indices are Urban Organization and Environmental and Resources, with these two objectives being easier to measure and quantify than the third one, Citizens and Society. This third measurable objective is probably the most important part in the process of transforming a society into a smarter and more sustainable one, but at the same time it is the most difficult to measure. Many aspects related to this are usually subjective and, therefore, difficult to quantify.

Everything in cities is interrelated and decisions in one area affect the other. Therefore, measuring is so important and establishing a clear and simple measurement method helps the right decisions to be made.

4. Evaluation Supporting Tool: STILE

A valuable computer-based tool, named SmarTness and SustaInabiLity Evaluation Tool (STILE) has been developed as one of the core services that form part of the REMOURBAN ICT solutions. Being a web application hosted on the Cloud, STILE can be used by anyone with a web browser, without any installation needs or specific technical requirements.

STILE was conceived as the service to support, automate and help to achieve the objectives set out in the evaluation framework. Therefore, in line with the evaluation framework, this tool allows for a quantified measurement of the cities’ progress on the road to sustainability and smartness on one hand, and the performance of REMOURBAN project in terms of efficiency and effectiveness of its interventions on the other hand. This way, STILE arises as the cornerstone to reinforce the communication between stakeholders and decision-makers in the cities.

STILE enables evaluations to be run for any of the REMOURBAN lighthouse cities at any moment. When an evaluation is launched, STILE takes the set of monitored variables stored in the Global ICT Platform for that city and the corresponding time period.

The tool, at a first step, calculates a set of indicators taking those variables as inputs, by applying the indicator formulas defined in the evaluation framework. Then, the set of formulas and calculations (normalization, weighting and aggregation methods) designed in the evaluation framework to obtain the Core Categories and Measurable Objectives from the indicators are programmed as part of the tool, obtaining finally the smartness and sustainability indices (Sm and Su, respectively). Since the indicators are normalized based on thresholds and weighted to calculate the Core Categories and Measurable objectives, STILE enables the creation of “base cases”. A base case establishes the structure of thresholds and the weighting scheme to be set when running the different evaluations. This way, a base case will act as a reference to be used as threshold and weighting scheme on a set of evaluations that can be compared (in order to track results over the time or at different cities) while, on the other hand, it is possible to create different base cases and process the same input datasets (variables and raw indicators) against them whether or not a refinement of the threshold and weighting scheme is required, allowing the flexibility for continuous improvement of the evaluation framework. While a starting base case is hardcoded in STILE, new base cases can be created by STILE users by running a simulation where the values for thresholds and weights can be manually set, and then converting the simulation into a base case to be used by STILE evaluations.

The key benefit of using STILE is not only the quantification of two indices, but a powerful presentation of the whole data set from variables to index, with all intermediate calculation levels, in a graphical way, making it easier for the user to have all information at a glance. STILE allows the identification of strengths and weaknesses of the city, to identify which are the aspects that most need further improvement of the city from the point of view of sustainability.

The way for the user to run a new evaluation by means of a simple form and the way results are depicted are aspects that were carefully designed under usability requirements to make the tool really easy for the user to use, while providing full information through self-explanatory charts.

From a practical and visual perspective, Figure 6 shows the STILE tool allowing us to select the city for which we want to run an evaluation:

Using the STILE visualization solution to represent the whole data set from variables to the final index makes it easier for understanding information, because all figures are displayed on just one screen, quantified and depicted in a hierarchical way for a deeper insight into grouping levels and dependencies, as can be seen in Figure 7.

Besides, the user can dig into any level or branch to get more information, just by clicking on each of the elements in a fully interactive way, which helps them to better understand the final value of the indices, based on its indicators, core categories and measurable objectives.

This way, the main objective of the STILE tool is to help in the assessment of sustainability and smartness in cities, supporting decision making when some new interventions or improvement of the existing ones is under discussion or evaluation in the city.

We can go a step further, by using STILE capability to compare evaluations. This way it is possible not only to have results for a certain moment in time, but also to assess the progress and evolution of smartness and sustainability in a city over time.

Based on this capability, STILE allows for the following scenarios:

• Progress assessment comparing with baseline situation. In REMOURBAN this is especially useful, since it enables us to compare a city’s situation before REMOURBAN with the situation after the implementation of a project’s interventions, to be able to quantify and evaluate the effects of those interventions. The calculation of the baseline in STILE has two main inputs. On one hand, indicators are calculated over the values gathered for the different variables before REMOURBAN interventions. These values can be requested to the Global ICT Platform if they are already available there, or can be manually provided and entered in STILE through the simulation form. On the other hand, a threshold and weighting scheme to be used in the normalization and aggregation formulas, is required. For this purpose, the hardcoded starting base case was used to run the baseline evaluation. This base case was created with the baseline values for thresholds and weights specified by the evaluation framework.
• Progress assessment in two different timeframes, reinforcing decision-making by providing a clear report on whether interventions are performing as expected or, on the contrary, some improvements or corrective actions should be taken.

This is how STILE presents evaluation comparison:

By means of a mirror disposition (Figure 8), both evaluations’ results are displayed, so it is possible to compare every single piece of information. This will enable us to go into details that finally lead to different results of Su and Sm indices at different moments in time.

Additionally, the red line above depicts some problems with the variable stored in the Global ICT Platform that is related to the corresponding indicator, category and objective of that branch. In the comparison, we can check when the problem has been solved.

There is another way of obtaining smartness and sustainability indices by combining the automated calculation capability of STILE together with human knowledge to provide missing data (variables expected to be in the Global ICT Platform, but missing) for example, or different thoughts on how weights and thresholds could be applied to positively impact on the indices.

In summary, STILE lets users to assess smartness and sustainability in cities in an objective and quantified way, which allows for deep insight and understanding of the progress of the city over time and the efficiency of the interventions undertaken improving those factors. Index information broken down into a granular and hierarchical tree of layers (categories, objectives and indicators), provides even more information to reinforce effective decision making when it comes to acting on the interventions that are actually related to the aspects detected to be weaker or subject to improvement.

5. Sustainability and Smartness Evaluation Methodology

The bases for the evaluation process are the KPIs (Key Performance Indicators) which are normalized, weighted and aggregated to calculate the Su and Sm global indices. The city level evaluation aims at supporting the development and assessing the impact of the high-level city strategy in terms of its integrated urban plan.

The methodology defined in the evaluation framework is applied to calculate the indices of each of the lighthouse cities before and after the implementation of the interventions. The application of the methodology for the calculation of the indices requires the following steps:

5.1. Scope Definition

The scope of definition is the whole city “City level”. The measurement of the city progress has allowed the lighthouse cities to have detailed information about where their cities have been progressing in a positive or negative way. REMOURBAN project has had impact at city level and this can be appreciated in the variations of indicators/indices during the reporting period (after the interventions), but it is important to remark that as the scope is very broad as we are talking about whole cities and the project impacts are limited to the demo areas (specific energy interventions in the demo-buildings of the selected district, electro mobility actions with new e-cars and e-charging points distributed throughout selected areas of the city, etc.), the variations in the progress of the cities due to the project actions sometimes is limited and also other external factors (out of REMOURBAN scope) could have influence.

5.2. Baseline Definition for the Lighthouse Cities

The baseline refers to the status of the cities before the starting of the REMOURBAN project. The main idea behind the baseline definition is to have a reference with which to compare the status of the cities after the implementation of the actions through the project execution. In this case, the specific reference year for the baseline was the year just prior to the start of the project (2015).

5.3. Reporting Period for the Lighthouse Cities

For the lighthouse cites the reporting period covers the last two years of the project, which means from January 2018 to December 2019. Four reporting points have been defined with the idea to assess the status of the lighthouse cities each 6 months during the reporting period.

For these four reporting periods, lighthouse cities have been in charge of collecting all the related information and provide them to the global platform from which the STILE tool has collected all the needed information to generate the different evaluations.

5.4. Data Collection and Analysis

In accordance with the general schemes showed in Figure 4 and Figure 5, indicators are the lowest level of the evaluation framework. They are valuable to establish a diagnosis of starting points, to track progress towards defined goals, to benchmark and to analyze the effect of some actions and assist in the decision-making process.

Nevertheless, significance of indicators is closely related to a good definition of them. The main criteria for selecting indicators included in the REMOURBAN evaluation framework are those shown in Figure 9, and the procedure for constructing the indices supposed an iterative process of analysis which resulted in the redefinition of some of the indicators and the way to group them.

Table 1. Indicators for Sustainability Index. Urban Organization.

KPI	Unit	Definition	Source
Urban Compactness	Meters	Relation between the usable space of the buildings (volume) and the urban space (area)	Sustainable Urban Model [1]
Green Areas	Square meters per inhabitant	Extension of green zones in the city per inhabitant	Sustainable Urban Model [1] ISO 37120:2014 [2] CITYkeys [15] UN-SDG [16]
Mixed-Used Development	Square meters per inhabitant	Extension of recreational and commercial areas in the city per inhabitant	Self-defined. Based on Sustainable Urban Models [1] ISO 37120:2014 [2] CITYkeys [15] UN-SDG [16]
Occupied Dwellings	%	Percentage of occupied dwellings in the city	ISO 37120:2014 [2]
City Accessibility	%	Percentage of public transport and building accessible to disabled people	Self-defined
Informal Settlements Area	%	Size of informal settlements as a percentage of city area	ISO 37120:2014 [2]
Bike Route Network	%	Length of bicycles and path lanes in relation to the length of city streets	CITYkeys [15]
Average Journey Delay	% Minutes delay/vehicle-km	Average delay per vehicle and km by traffic congestion	CITYkeys [15]
Public Transport Infrastructure	km/100000 hab	Length of public transport infrastructure (high and light capacity) per inhabitant	ISO 37120:2014 [2]
EV (electro-vehicle) Penetration Rate	%	Percentage of EV over total number of vehicles	ITU-L1603 [8]
EV Public Charging Points	Number of EV public charging points per square kilometre	Public Charging Points for Electrical Vehicles over the city	Self-defined
EV kWh Recharged	Kilowatt hour in a period per 100000 inhabitants	Total kWh recharged by EV in the public EV charging points of the city	Self-defined
Sustainable Mobility Share	%	Public transport trips and EV trips per capita	Self-defined. Based on ISO 37120:2014 [2]
Average Building Energy Demand	Kilowatt hour per square meter	Average of energy demand of total buildings in the city	EN15643 [7]
Thermal Comfort	%	Percentage of number of hours per year inside comfort range in city buildings	ARQ-BIO [4]
Indoor Air Quality Comfort	-	Indoor Air Quality (IAQ), referred as the air quality within buildings	EN13779 [3]
Environmental Certified Buildings	%	Rate of environmental certified buildings in the city	ITU-L1603 [8]

,

Table 2. Indicators for Sustainability Index. Environment and resources.

KPI	Unit	Definition	Source
CO2 Emissions	Tonnes of CO2 per inhabitant	Annual tonnes of CO2 (and CO2 equivalent) emitted per inhabitant	ISO 37120:2014 [2], CITYkeys [15]
NO2 Air Quality	μg/m3	Daily average of NO2 in the city	ISO 37120:2014 [2]
PM10 Air Quality	g/m3	Daily average of particulate matter (PM10) in the city	ISO 37120:2014 [2]
PM2.5 Air Quality	g/m3	Daily average of particulate matter (PM2.5) in the city	ISO 37120:2014 [2]
Noise Pollution	%	Rate of population in the city affected by noise pollution	ISO 37120:2014 [2]
Final Energy Consumption	Megawatt hour per inhabitant	Final energy consumption in the city	CITYkeys [15]
Primary Energy Consumption	Megawatt hour per inhabitant	Primary energy consumption in the city	Self-defined
Renewable Energy Penetration Rate	%	Percentage of total energy used generated by renewable sources within the city	CITYkeys [15], ISO 37120:2014 [2]
Energy Poverty	%	Percentage of households unable to afford basic energy services in the city	CITYkeys [15], IND-Epoverty [10]
Water Consumption	Litres per inhabitant per day	Daily average water consumption per capita	CITYkeys [15] ISO 37120:2014 [2], ITU-L1603 [8]
Potable Water Supply Service	%	Percentage of city population with potable water supply service	ISO 37120:2014 [2], ITU-L1603 [8]
Wastewater Treatment	%	Percentage of city’s wastewater connected to any wastewater treatment system	ISO 37120:2014 [2]
Sanitation services	%	Percentage of the households with access to improved sanitation services	ISO 37120:2014 [2] UN-SDG [16] ITU-L1603 [8]
Urban Solid Waste	Tonnes per inhabitant	Solid waste collected	ISO 37120:2014 [2], CITYkeys [15]
Solid Waste Treatment	%	Tonnes per inhabitant	ISO 37120:2014 [2], UN-SDG [16] ITU-L1603 [8]

,

Table 3. Indicators for Sustainability Index. Citizens and society.

KPI	Unit	Description	Source
Gross Domestic Product	€ per inhabitant	City Gross Domestic Production per capita	CITYkeys [15]
Employment Rate	%	Employment rate	ISO 37120:2014 [2] UN-SDG [16] ITU-L1603 [8]
Disposable Income	€ per inhabitant	Average money available for spending after taxes	CITYkeys [15] UN-SDG [16]
Population Living in Poverty	%	Percentage of people living below the poverty threshold	ISO 37120:2014 [2], UN-SDG [16])
R&D Expenditure	€ per inhabitant	R&D expenditure per capita in the city	ITU-L1603 [8] UN-SDG [16]
Labour Productivity	€ per person employed	Labour productivity as the valued added per person employed	ITU-L1603 [8] UN-SDG [16]
Gender income equity	Dimensionless	Ratio of average hourly earnings of female and male employees	ITU-L1603 [8] UN-SDG [16]
New Businesses	Number of new businesses per 100000 inhabitants	Number of new businesses created in the city per 100,000 population	ISO 37120:2014 [2]
New Patents	Number of new patents per 100000 inhabitants	New patents per 100,000 population	ISO 37120:2014 [2]
Adult Literacy Rate	%	Percentage of literacy adults over the total population	ITU-L1603 [8] UN-SDG [16]
Secondary Education Completion Rate	%	Percentage of students who complete secondary education over total number of students originally enrolled in secondary education	ISO 37120:2014 [2], ITU-L1603 [8] UN-SDG [16]
Higher Education Degrees	Number of city inhabitants with high education degrees per 100000 inhabitants	Number of public universities in the city per 100,000 population	ISO 37120:2014 [2] ITU-L1603 [17] UN-SDG [16]
Access to Basic Health Care	%	Percentage of population with access to basic health care services	ITU-L1603 [8] UN-SDG [16]
In-patient Hospitals Beds	Number of in-patient hospital beds per 100000 inhabitants	Rate of in-patient hospital beds	ISO 37120:2014 [2] ITU-L1603 [8]
Emergency Service Response Time	Minutes and seconds	Average time to respond to emergency calls	ISO 37120:2014 [2] ITU-L1603 [8]
Population Dependency Ratio	%	Rate of Population Dependency in the city	CITYkeys [15]
Average Life Expectancy	Number of years of average lifetime	Average time an inhabitant is expected to live	ISO 37120:2014 [2] ITU-L1603 [8]
Suicide Rate	Number of deaths by suicide per 100000 inhabitants	Suicide rate per 100,000 population	ISO 37120:2014 [2]
Crime Rate	Number of reported crimes/100 000 population	Crime rate per 100,000 population	ISO 37120:2014 [2] CITYkeys [15] UN-SDG [16]
Internet Access Rate	%	Percentage of households with internet access	ISO 37120:2014 [2] CITYkeys [15] UN-SDG [16] ITU-L1603 [8]
Voter Participation	%	Voter participation level in the city	ISO 37120:2014 [2] CITYkeys [15]
Open Public Participation	Checklist	Promotion of inhabitants’ participation in public affairs	ITU-L1603 [8]
Innovative/Green Public Procurement	%	Percentage procurement using innovative and environmental criteria	CITYkeys [15]
Open Data	%	Percentage of available open data of cities	CITYkeys [15] ITU-L1603 [8]
Incentives to Promote Sustainable Actions	Number of incentives per period	Program of Incentives for Sustainable Development	Self-defined
Awareness Initiatives	Number of awareness initiatives per 100 inhabitants	Number of awareness initiatives	Self-defined
Investment in Smart Cities and Communities Projects	Million €	Investment in Smart Sustainable City Projects	Self-defined
Climate Resilience Strategy	-	Resilience strategy in cities	CITYkeys [15] UN-SDG [16]

,

Table 4. Indicators for Smartness Index. Smart Urban Organization.

KPI	Unit	Definition	Source
Street lighting	%	Street lighting management using ICT	ICT-ITU [18]
Public space WiFi Coverage	%	Percentage of Public Space with free access to wireless local area network (Wi-Fi)	CITYkeys [15]
3G/4G Mobile network	%	Percentage of the city covered by at least a 3G mobile network	Self-defined
Fibre optic coverage	%	Fibre optic coverage in the city	Self-defined
Traffic monitoring	%	Availability of traffic monitoring using ICT	ICT-ITU [18]
Parking guidance systems	%	Availability of parking guidance systems	ICT-ITU [18]
Real-time traffic information	%	Availability of real-time traffic information	ICT-ITU [18]
Integrated transport services	Dimensionless	Number of different modes of public transport integrated in a smart card	Self-defined. Based on SMCITY-IND [6]
Trips using integrated transport system	%	Number of trips made using the integrated system for public transport	Self-defined
Automated homes	%	Percentage of automated dwellings in the city	Self-defined. Based on KPI-ITU [19]
Broadband subscriptions	Number of broadband subscriptions per 100 inhabitants	Availability of broadband subscriptions	ICT-ITU [18]

,

Table 5. Indicators for Smartness Index. Smart Environment and Resources.

KPI	Unit	Definition	Source
Meteorology monitoring	Number of ICT weather monitoring stations per square kilometre	Application of ICT based monitoring system for weather stations	Self-defined
Air quality monitoring	Number of ICT air quality monitoring stations per square kilometre	Application of ICT based monitoring system for air pollutants	ICT-ITU [18]
Noise monitoring	Number of ICT noise monitoring stations per square kilometre	Application of ICT based noise monitoring	ICT-ITU [18]
Smart electricity meters	%	Availability of smart electricity meters	ICT-ITU [18]
Real-time information for energy	%	Proportion of households provided with energy real-time information system	Self-defined
Smart grids penetration	%	Percentage of total energy demand served by advanced meters	SMCITY-IND [6] ICT-ITU [18]
Gas system management	%	Gas system management using ICT	ICT-ITU [18]
Smart water meters	%	Availability of smart water meters	ICT-ITU [18]
Water supply management	%	Water supply system management using ICT	ICT-ITU [18]
Sewage system management	%	Sewage system management using ICT	ICT-ITU [18]
Drainage system management	%	Drainage system management using ICT	ICT-ITU [18]
City water monitoring	%	Application of city water monitoring through ICT	ICT-ITU [18]

and

Table 6. Indicators for Smartness Index. Smart Citizens and Society.

KPI	Unit	Definition	Source
Employees ICT Sector	%	Employees belonging to ICT sector	ICT-ITU [18]
E-commerce	Number of transaction per 100 inhabitants	Application of e-commerce transactions	ITU-L1603 [8] ICT-ITU [18]
Computing Platforms	%	Application of computing platforms	ICT-ITU [18]
Teleworking System	Number of people using teleworking system per 1000 workers	Penetration of teleworking system	ICT-ITU [18]
ICT in Secondary Education	%	Application of ICT in secondary education	ITU-L1603 [8] UN-SDG [16]
Use of e-learning System	%	Use of e-learning system	ICT-ITU [18]
Home Health Monitoring	%	Percentage of patients with Home Health Monitoring Systems	Self-defined
ICT Penetration in Health System	%	Percentage of hospitals using ICT based Health Monitoring Systems	Self-defined
Smartphone Penetration	%	Availability of Smartphones in households	ITU-L1603 [8]
ICT in Crime Prevention	YES/NO	Availability of ICT based systems to prevent crime in the city	SMCITY-IND [6]
ICT for Disaster Management	YES/NO	Availability of ICT based systems to manage disasters	Self-defined
Online City Information	%	Availability of online city information and feedback mechanisms	ICT-ITU [18]
Online Civic Engagement Channels Use	%	Online civic engagement	ICT-ITU [18]
Online Government Services	%	Percentage of government services that can be accessed via web	Self-defined
Investment in ICT	%	Amount of city investments in ICT issues	ICT-ITU [18]
ICT in Public Procurement	%	Availability of ICT systems in Public Procurement purchases	Self-defined
Use of Social Media	Number of municipality links in social media channels per 100 inhabitants	Total number of municipality links in social media channels	ICT-ITU [18]
City Web Apps/Services	Number of government city apps/services per 100 inhabitants	Number of government web applications/services	Self-defined
Web Apps/Services Use	Number of visits of city app/services per 100,000 inhabitants	Number of visits to government applications for city services	Self-defined
Citizens Registered in City Web/Services	%	Percentage of citizens registered in government applications	Self-defined
Apps Downloads	%	Percentage of apps downloads	Self-defined

show the list of indicators included in each of the indices, their definition and the sources used for their selection or definition.

The collection of data is one of the most challenging tasks of the process and at the same time the quality and amount of data used for calculating the indicators is one of the most critical issues to obtain a reliable index.

The quality of available data is a key aspect in the index construction process; also, a considerable amount of information is required to calculate an index based on the aggregation of indicators.

Most of the data required for the calculation of the indicators at city level are compiled from official statistics and municipalities’ databases. Behind this data collection process there is a great effort of the cities in terms of data monitoring, quality-checking and verification of data transmission from the different data sources, identifying and solving the possible problems that were appearing.

The following approach was followed for analyzing and processing the data collected:

• Same value was sought and compared in different data sources and only reliable data was noted.
• Where available, values were compared to reference values from normative, standards or bibliography in order to check the magnitude and identify possible mistakes in the databases.
• Missing values were replaced, where possible and applicable, with regional or country values or standardized values from accepted references. In this respect, it is important to remark that the tool has implemented a missing data imputation algorithm based on reference thresholds coming from proven bibliography or expert opinions which means that all the required datasets are always covered (in case this situation occurs, this is depicted as mentioned before by the red line branch in the STILE tool graphics).

5.5. Calculation of the Indices

The calculation of the indices is performed using the computer-based Evaluation Tool STILE. This tool calculates and normalizes the indicators, weights and aggregates them in order to calculate the Su and Sm indices in an automatic way based on the methodology and algorithms implemented within the tool.

5.5.1. Normalization Process

The method is based on the establishment of interval values for each indicator. These intervals are defined with the minimum and maximum values. Therefore, the minimum values are substituted by 1, while the maximum values are replaced by 10, being all the intermediate measures relative values in the interval from 1 to 10. The min–max normalization functions depend on whether the objective value is the highest (as is the case of renewable energy penetration rate) or whether it is the lowest (as is the case of CO₂ emissions). The first group of indicators are defined as “positive indicators” while the second group are defined as “negative indicators”.

5.5.2. Weighting Process

The indicators are first weighted by its importance within their category and then aggregated obtaining a partial index for each core category. The same process is followed for aggregating categories; they are weighted according to importance within their measurable objective and then aggregated to obtain intermediate indices for each measurable objective. Finally, weights need to be established for each measurable objective to be aggregated into the final global index. In the case of REMOURBAN project the following weighting methods are considered:

• Equal weights per indicator in each category: This methodology assumes that all the indicators have the same relevance within their category and the weights are distributed equally in each of the categories. This is the case of the core categories and measurable objectives weights.
• Relevance criteria for each indicator: The way in which indicators are weighted is based on the principle that recognised indicators should be weighted more than the other ones. Indicators are ranked following a relevance criteria which gives scores from 5 to 1 depending on the relevance of the source in which they are found, then based on the scores indicators are weighted as a percentage of importance within the same core category.

5.5.3. Aggregation Process

In this case, the additive method is considered for the aggregation of the weighted indicators. This method is based on adding up the weighted and normalized indicator scores.

5.6. Evaluation of the Results

At this point the comparison and detailed analysis between the reporting period evaluation results and the baseline period evaluation results is done. The assessment of the Sustainability and Smartness of the lighthouse city of Valladolid after the project interventions is shown in the following sections.

6. Evaluation of Sustainability and Smartness in Valladolid

The results of the Sustainability and Smartness evaluation in Valladolid in the reporting periods 2018 and 2019 are presented below (Figure 10). The results compare the reporting periods 2018 and 2019 with the baseline period (status before the starting of the project in the year 2015) in order to see the progress of the city along the project reporting period.

The values of the two indices for the three periods can be seen in Figure 10. As can be appreciated in these figures, the variation in the value of the indices is not very significant. This is because they are indices at city level, and project actions that have a great impact in a specific area of the city have a lower impact when their impact on the city is analyzed. What must be considered is that both indices have improved thanks to the implementation of REMOURBAN actions, and the comparison between different periods of the indices, their measurable objectives and their core categories, allows us to know the impact of each of the actions carried out in REMOURBAN and provides the possibility to extrapolate the results to plan future interventions in the city.

With the graphs, the STILE tool allows us to compare the results of the three periods we are studying for each of the indicators, core categories and measurable objectives defined in REMOURBAN evaluation framework.

In the following figures it is possible to see in detail the specific graphs corresponding to each of the measurable objectives for both indices. The improvements are due to the actions carried out by REMOURBAN project but also to others implemented by Valladolid City Council in order to improve the sustainability and smartness of the city. We compare the starting situation of Valladolid City for the indices (Baseline) with the reporting period 2018 and 2019.

The first three graphs (Figure 11, Figure 12 and Figure 13) correspond with the measurable objectives within the Sustainability evaluation while the following three graphs (Figure 14, Figure 15 and Figure 16) correspond with the measurable objectives within the Smartness evaluation.

According to the calculated values in Urban Organization (Figure 11), the key element in terms of sustainability improvement is the electric mobility. We can observe how the penetration rate of electric mobility has increased. Additionally, a substantial increase of the EV public charging points and therefore the kWh recharged is appreciated. All these improvements in electric mobility go hand in hand and are aligned with the efforts done for reinforcing the recharge infrastructure of the city’s public network.

Focusing on the environment and resources (Figure 12), the reduction of emissions seems to have a real impact.

The reasons are mainly collected in the previous picture: a rationalized urban mobility mainly thanks to the e-pioneers citizens’ work and enabling specific infrastructures for charging the e-vehicles.

In any case, it is expected that these indicators will continue to improve thanks to actions as itineraries for certain groups with cleaner alternatives for public transport, nature-based solutions for a sustainable mobility, a more eco-friendly design for the transport infrastructure, a new green cycle lane and re-naturing of existing bike lanes, among other actions.

For citizens and society (Figure 13), there are barely perceived changes in values between 2018 and 2019.

REMOURBAN actions could have had a positive effect on indicators such as the incentives to promote sustainable actions, investment in Smart City Projects and awareness initiatives, but at the time of performing the evaluations no changes are observed.

It is important to keep in mind that citizen actions are far-reaching and very long-term. In any case, Valladolid is concerned by the zero impact on essential issues such as climate resilience, an aspect in which all cities and the world in general are focusing.

However, the city is taking steps in that direction: there is already a road map and circular economy action plan. A climate resilience strategy must take into account, of course, better management of resources and services. This approach will create a smaller footprint, also contributing to climate awareness.

Figure 14 shows those areas in which the city has wider technological developments, apparently without any change due to the intervention of REMOURBAN.

The indicators that could have impacted for improvement (Real-Time Traffic Information or Automated Homes) have hardly done so. Speaking in terms of percentage, these actions in mobility and energy are of reduced dimension, as may be perceived in the Smartness graphics.

It is necessary to wait for a holistic city intervention in order to know if the lack of variation of those variables is really due to the scope of the actions or the fact that they are inelastic elements to change, in terms of penetration.

This analysis will be interesting when the S2CITY project (Intelligent System of Citizen and Tourist Services) has been fully implemented. It is an initiative presented by the City Council of Valladolid to the second call of Red.es Smart Cities. It is about an integrated and intelligent management of the city for its digital transformation. The initiative will reuse, optimize and expand pre-existing platforms in the city of Valladolid, in particular those represented in the picture as the Parking Systems, the already integrated transport services to respond to new challenges in terms of management, data opening, transparency and openness of the model to the city with the participation of citizens.

This project means a global offer of services to citizens: the creation of a unique access to public services through the citizens and tourists card and/or the mobile device, while rewarding those citizens who contribute to build a more sustainable and smart city applying “gamification” techniques. It also applies big data technologies and is foreseen by 2020.

The reading in the graphic of Figure 15 is a bit different from the previous one; the reporting period of 2018 shows a real improvement in smart electricity meters, but then stagnation in the next period. It should be interpreted that this action was carried out around 2018 without increasing the number of sensors and measuring devices so far, which has indeed been the case.

Nevertheless, an increase of this variable is expected from 2020, mainly through the project URBAN GreenUP which is a project funded under the European Union’s Horizon 2020 programme. Its objective is the development of Re-Naturing Urban Plans in some cities with the aim of mitigating the effects of climate change, improving the air quality and water management, as well as to increase the sustainability of our cities through innovative nature-based solutions.

Some of these solutions will be managed by Smart meters in particular for the re-naturing urbanization, green infrastructure and water interventions.

As the previous case, in Figure 16 can be seen the change that occurred in 2018, with some nuances that have occurred since then. According to the calculated values, the key element in terms of smartness is well-being through features as the Smartphone penetration and the governance services. It is important to remark that the Valladolid Council has a goal of improving the accessibility in public spaces through the application of ICT by a Municipal Accessibility Plan: strengthening of ICT public platforms for the e-administration, e-learning, e-inclusion, e-culture and e-health, with the expectation of massive coverage of public services by internet and increasing significantly the number of accessibility information points.

7. Conclusions

The Urban Regeneration Model (URM) can be considered as one of the main results of REMOURBAN project. The development of this methodology has been essential for the appropriate alignment of the project tasks and the understanding of all connections.

City transformation evaluation is one of the three frameworks defined in the Urban Regeneration Model since the evaluation is sought as the main supporting mechanism throughout the various phases of the city transformation process. This framework considers two levels of evaluation: city level, to assess both sustainability and smartness of the city as a whole, from a comprehensive and integrated perspective, and project level, to provide a clear identification of the impact of implementation of technologies and solutions on the three key priority areas (sustainable districts and built environment, sustainable urban mobility and integrated infrastructures and processes) aimed at achieving the city high-level goals.

This article is focused on the REMOURBAN evaluation framework at city level and its main objective is to describe the application of this framework in the assessment of the Sustainability and Smartness of the lighthouse city of Valladolid before and after the project interventions and to show the main results achieved thanks to REMOURBAN actions from the sustainability and smartness point of view. The assessment of the city progress covers two reporting years—2018 and 2019—showing the progress through the different periods.

In order to make the URM more useful and easily replicable, some supporting tools have been developed, one of them being STILE, which is focused on the application of the evaluation framework both at city and project level. The STILE tool requires a great amount of data—which REMOURBAN cities had to collect and provide—for calculating the indicators defined in each category of the evaluation framework. Data collection was one of the most challenging tasks, taking into account the need for a great amount of data with enough quality to achieve accurate results. Most of the data required for the calculation of the indicators at city level were compiled from official statistics of governments, international organizations or municipalities’ databases.

The calculated values by STILE for the city of Valladolid and the comparative among the three periods analyzed (baseline and reporting period 2018 and 2019) have been included in this article and their results have been analyzed. As can be seen, when the values of the indices calculated for each period are compared, their variation during the development of REMOURBAN has not been very significant, although as a general comment, slight improvements can be appreciated.

The reason the variation during the project duration is not very relevant in the indices is mainly because they are indices at city level, and for REMOURBAN actions that have a great impact in a specific area of the city have less impact when their impact on the city as a whole is analyzed. It is also necessary to consider that some actions have long-term effect and it is not possible to appreciate large changes in the project period. What should be considered is that both indices (Su and Sm) have improved thanks to actions such as the retrofitting of a single neighbourhood or the change of a small percentage of vehicles for electric vehicles. If the same actions were carried out in more districts of the city or if the number of electric vehicles increases further, these indices would have a greater change. The analysis of the indices, including their Measurable Objectives and Core Categories, allows us to know the impact of each of the actions carried out in REMOURBAN in the city as a whole and could be used to extrapolate the results to plan future interventions in the city. It can be affirmed that the project has established the basis of how to achieve sustainable and smartness cities based on the implemented actions, but the work should not stop here and cities should continue working on the path laid down by the project.

The tool developed here could be further investigated and improved through the integration of more information on aspects that can contribute to better manage other city assets such as water or waste. Additionally, new developments that are currently under investigation consider the integration of geo-referred data sources in order to extract the required data to model some aspects of the city and to represent the results offered by the tool.

Additionally, the scheme proposed could be further investigated and developed to deal with the need of evaluating the city resiliency against unexpected events such as COVID-19 or climate disasters caused by climate change. It should be noted that the framework shown within this paper has been designed for the purpose of supporting the processes considered within the Urban Regeneration Model depicted and, therefore, to support the evaluation of the areas covered. However, it is important to also note the potential for expanding and adapting the scheme to deal with these aspects.