Towards sustainable and smart cities: Replicable and KPI-driven evaluation framework

: Sustainability is pivotal in the urban transformation strategy in order to reach more re-source-efficient, resilient and smarter cities. The goal of being a sustainable city should drive the decisions for city interventions. Nonetheless, impacts need to be quantified, lacking of standard and/or common methodologies that could be replicable across multiple cities. There exist many initiatives aiming at defining indicators and assessment procedures, but without convergence in the definition of terms and application methodologies, making complex its real implementation. Within mySMARTLife project (GA#731297), a KPI-driven evaluation framework is defined with the aim of covering the multiple pillars of a city (i.e. energy, mobility, citizens, economy) in a holistic way. This methodology also defines the concepts and terms to guide urban planners and/or experts at time of implementing the framework in a specific city. The evaluation framework has been deployed in the three cities of Nantes, Hamburg and Helsinki and some lessons learnt have been extracted, such as the necessity of providing a definition of measurement boundary to avoid interpretations. Thanks to a co-creation strategy, the main difficulties and issues from the cities have been taken into con-sideration for increasing the replicability.


Introduction
Cities have transformed into hubs for modern civilizations [1], but this transformation has impacts in the use of the natural resources. The limited nature of these resources motivates the necessity for more sustainable cities, which may be achieved by the application of new technologies [2]. In this direction, the European Commission has adopted an ambitious plan for the decarbonisation of the European cities and the penetration of renewable energy sources and has established a reduction of the GHG emissions by 55% in 2030 and the climate neutrality by 2050 [3]. mySMARTLife project (Grant Agreement #731297) [6], H2020-funded, aims the sustainable and smart transformation of the three lighthouse cities of Nantes, Hamburg and Helsinki by applying the aforementioned concepts. More than 150 actions are contributing in the pillars of energy, mobility, ICT, citizens, economy and governance to the new urban sustainability concepts. mySMARTLife targets are: a) renewable share of 54%; b) reduction of 55% of the greenhouse emissions because of buildings and mobility. Furthermore, mySMARTLife fosters both Smart Economy and Smart People supported by local economy growth and entrepreneurship. The main driver for designing and implementing this innovative concept of smart city is an integrated urban planning.
Nevertheless, the measurement of the achieved impacts requires a rigorous assessment plan so as to quantify the final numbers in terms of sustainable transformation. This paper presents the evaluation framework that has been prepared and deployed within mySMARTLife project. This framework aims at merging the multiple and diverse verticals of the city with the objective of reaching a global view of the smart city. A set of KPIs (Key Performance Indicators) allows the implementation of such a plan in order to objectively obtain the final impacts. The mySMARTLife framework does not only aim the definition of an affordable and objective methodology for evaluating the sustainability of cities in the multiple verticals, but also, at the same time, simplifying the current methods that complicate the real deployment due to lack of experience by the urban planners.
The paper is structured as follows. Section 2 provides a set of references and previous researches in the implementation of urban sustainable evaluation plans. Section 3 presents the methodology that has been applied in mySMARTLife project for the definition of the indicators and the evaluation framework. Next, section 4 describes the framework and how it has been applied in the different pillars across the cities. Finally, section 5 extracts a set of conclusions and lessons learnt.

Background: Other sustainable evaluation frameworks
Many studies currently try to provide an answer to the quantification of the impacts that a set of actions have in a smart city. A great diversity of methods are focused on the measurement of the sustainability and smartness, but the complexity and multidimensionality of these concepts are one of the major barriers [7]. There exist standards for sustainable rating of the built environment (e.g. LEED [8], BREEAM [9] or CASBEE [10]), which have been adapted to the cities context [11]. However, these frameworks are in favour of the environmental sustainability over economic and social sustainability [12]. The way of application is based on the assessment of the different criteria by comparison to benchmarking values, which restricts the use of these methodologies mainly to those countries that have already defined baseline values, limiting its widely use [13].
Standards such as ISO 37120 [14] or ISO 37122 [15], as well as the indicators for Sustainable Development Goals for the implementation of the 2030 Agenda for Sustainable Development [3] should be the reference for the evaluation of smart cities under a sustainable approach. Nevertheless, although they are a good reference, the difficulties lie in the application. While indicators are well-defined, no methodologies are determined to guide cities and experts at time of deploying such evaluation frameworks.
The European Commission has currently funded 18 projects for promoting European Smart cities and Communities towards the energy transition. Complementary, the initiatives CITYkeys [16] and Smart Cities Marketplace [17] support these projects by defining a set of indicators applicable in multiple axes (e.g. planet, people or prosperity, among others), but with the main goal of being able to compare the European cities and rank them. Many of these EC-funded projects make use of the indicators established by CITYkeys, but the way of how applying the assessment differ from project to project, even though similar Smart City solutions are being implemented.
Angelakoglou et at. [18] proposes a categorization of KPIs in six dimensions (technical, environmental, economic, social, ICT and legal) and the performance metrics against to measure each KPI is selected by each demonstrator city among these options: baseline, business as usual or other threshold. Kourtzanidis et al. [19] summarizes several insights on key characteristics and limitations of currently available urban sustainability and smartness evaluation frameworks and includes indices in a Smart city evaluation framework under a triple axis approach: the project performance index, the sustainability impact index and sustainable performance index. Finally, García-Fuentes et al. [20] defines a framework based on energy and mobility, while the social and economic aspects are in background. Moreover, the evaluation plan relies on an index that normalise the sustainable rank of the city by weighting the actions and prioritising the energy and mobility ones.
Consequently, there is a not a common and accepted evaluation framework for the assessment of a city. Different methods are defined in each of these iniciatives, which differ in the type of metrics (indicators / indexes as rating systems), the scope of the evaluation (only focused on the intervention area / extrapolation at city level) and the objectives evaluated in the domains identified to measure. These issues make even more complex its real deployment in cities. mySMARTLife evaluation framework, presented in this paper, goes a step forward by combining the advantages of some of these assessment plans together standard indicators and defining clear and affordable methods for its implementation across cities. First, the framework overcomes both the intervention area level and the city extrapolation. Second, it is not restricted per domains (i.e. covers both energy and mobility and other pillars at the same degree), which provides more flexibility and adaptability to the city actions (e.g. evaluating the performance of the application of a directive for promoting renewable energy). Third, the framework is defined in a co-creation strategy with the cities experts and urban planners to overcome real problems at time of deploying assessment frameworks, allowing a better replicability.

Materials and Methods
Developing an evaluation framework is a key requirement for the calculation of impacts since this establishes the purpose to evaluate and defines the elements to use in the evaluation. The framework defined in mySMARTLife is driven by KPIs with twofold scope: city and project (intervention area) levels. Multi-dimension has also been considered to cover the multiple pillars in a city: energy/environment, mobility, urban infrastructure, citizens, economy and governance.
The design of this evaluation framework has been the result of collaboration among research centres, technology providers and cities, thus, creating a co-creation strategy between stakeholders to define an evaluation framework that allows the evaluation of impacts of demonstrated solutions in a holistic way and considering the sustainable goals of a city.
The applied methodology consists of the next four main steps: 1) Step 1: Establishing the objectives pursued by cities involved in mySMARTLife defined in their urban plans to setting the basis of the evaluation. The analysis of city plans identified the following challenges: (1) to achieve an economic growth decoupled from resource use to face the current pollution and CO2 emissions, (2) to improve the life quality of citizens and (3) cities operation in a more efficient way. Additionally, these cities through the involvement in mySMARTLife adhered the willingness to progress towards the concepts "smart people" and "smart economy" that allows the participation of citizens in the decision making processes of the cities and the deployment of new innovative business models that make affordable the implementation of energy efficient buildings, city infrastructures and transport for freight and passengers.

2)
Step 2: Identification of dimensions to evaluate aligned the city visions previously defined, the typologies of solutions implemented and their expected performance and impacts. The output of this work is shown in 3) Table 1 below. Impact of the project in the city urban planning and policy improvement 5) Step 3: Selection of the suitable indicators for each dimension after the search of indicators for Smart and Sustainable Cities assessment from bibliography, standards and previous works (see Table 2) and considering a set of criteria.

4)
Since there is not an only indicator system that can be used for mySMARTLife to assess the diverse effects produced by the interventions, different documents deployed under diverse initiatives have been consulted. As a result of this whole process, 151 city level indicators and 128 project level indicators have been defined in an iterative process among technicians from research centers and city partners following the following criteria: • Measurability: The identified indicators should be capable of being measured through the data collection methods established in the project.
• Completeness: The indicators should cover all the type of interventions (district, city infrastructure, mobility, ICT) and non-technical aspects (governance, citizens, finance) deployed in the project as well as the expected type of impacts (environment, economy, social and technical).
• Relevance of the indicator for the objective of evaluation defined.
• Availability of data in the cities for the final selection of indicators since not all the relevant indicators are quantifiable. On other side, to guarantee a proper evaluation of impacts, below criteria have been also considered: • Independence and non-redundancy of indicators.
• Familiarity of persons in charge of evaluation with indicators through a well description of the formulas and definitions and dedicated sessions for clarify possible doubts.

6)
Step 4: definition of the KPI-driven evaluation framework combining the objectives and indicators for the multiple city domains and the assessment plans to evaluate the KPIs. This is the main result of this paper and it is explained in the next section.

mySMARTLife KPI-driven evaluation framework
As stated before, the result of the methodology is the assessment framework, which is being applied in the cities of Nantes, Hamburg and Helsinki. Figure 1 [21, 22] illustrates the proposed framework. First of all, it should be highlighted the two levels of assessment that are included: • Project level includes the more than 150 actions being deployed in the specific areas of the cities involved in the project. Then, the main aim is to obtain the quantitative analysis of the impacts achieved after those actions (e.g. building retrofitting, integration of renewables, electrification of the transport, etc.) as well as the performance of the technological solutions. • City level, which extrapolates the quantified impacts from the executed project to estimate the impact that these actions would have in the city. The outcome of this level is to support cities at time of planning urban transformation strategies by following quantitative and objective methods driven by KPIs.

Figure 1. mySMARTLife KPI-evaluation framework
Within each of the stated levels, a set of categories (named fields in the City level and pillars for the Project level) are established. These are, as mentioned, energy and environment, mobility, urban infrastructures (including the digitalisation of the city through the ICTs and urban data platforms deployment), citizens, economy and governance. All of them are driven by a set of indicators [21,22], which are summarised in Table 3. City level indicators are calculated with data compiled from public databases, mainly from city statistics. For the case of project level indicators, two KPIs categories have been established: • Quantitative indicators as elements to demonstrate the impacts of innovative solutions through the collection of data from meters installed and other data compilation processes. • Qualitative indicators to assess the perception of benefits gained by citizens, companies and the municipality through questionnaires and surveys. Last but not least, the framework complements the indicators and the definition by methodologies and protocols with the aim of supporting cities when implementing the project evaluation framework. Thus, the evaluation framework does not only provide a theoretical indicator-based procedure, but also pathways to apply them to analyse the success of the implemented actions. Table 4 [22] depicts a summary of the proposed methodologies for each one of the project pillars.

Analysis of the project pillars and categories
The general framework is applicable in multiple verticals of the city. As depicted in Table 4, although the framework is defined in a holistic way to consider the cross-domain effects, each of the categories requires its specific evaluation methodologies. These are described in the next subsections.

Energy & Environment
Energy and environment pillar mainly refers to energy efficiency in the built environment and other elements of the cities such as smart lighting (that imply energy consumption reduction), including renewable generation at local level and at district / city level (e.g. Building integrated RES, district heating, PV plants, wind farms, etc.).
To determine high-performance districts, the energy demand, use and self-consumption of the buildings are calculated. To accomplish with it, IPMVP (International Performance Measurement and Verification Protocol) [23] has been selected as it is a standard for the evaluation of the energy performance. Explaining IPMVP is not the objective of this paper, but how it is adapted to the mySMARTLife framework requirements. In this sense, two measurement periods are established: • Baseline: This represents the starting point, that is to say the reference for comparison.
Three methods are available: I. Using the country normative for new and/or existing buildings as reference. II. Simulate the energy behaviour of the building through any simulation software (also applicable for new and/or existing buildings). III. Only for existing buildings and in case of monitoring is available, energy performance based on real data (smart meters or energy bills) is calculated. • Reporting period: It is the period after the construction or renovation of the building, where the final performance is measured. This period requires of real data, being either monitored with smart meters or obtained from energy bills. These two periods are then compared to obtain the final impact, but it needs adjustments, such as climate conditions. This is a routine procedure well-established by IPMVP. However, what is even more important is the definition of the boundary, which is one of the main lessons learnt described in section 4.2. Figure 2 [21] shows how mySMARTLife defines the different boundaries to create a common understanding when applying the evaluation procedure. Many of the existing frameworks fail in the definition of the boundary, generating confusion and complexity and this is how mySMARTLife solves it. Two levels are established in this pillar, building actions and city infrastructure. mySMARTLife sets the boundary for building actions as the combination of the energy demand or use together the delivered energy. As depicted, the boundary surrounds all the elements of the building, including the local renewable energy or local generation systems (e.g. individual boiler) that is used for self-consumption.
When applying at district / city level for shared generation systems (e.g. district heating), the boundary us re-scaled (as drawn in Figure 2). It does not only contain buildings and distribution elements, but also integrates such generation systems to calculate the indicators at consumed energy level (i.e. considering the performance of the different elements in the generation and distribution chain).
Finally, the case of lighting systems comprises the energy consumption of the bulbs and the comparative of energy when light bulbs have been replaced. In this specific case, the adjustment is not made based on climate conditions, but hours of use.

Mobility
The mobility evaluation pursued the quantification of impacts of the mobility actions implemented in the cities of the project as well as the actions performance in terms of: • Reduction of air quality emissions due to replacement of ICE (internal combustion engine) by e-vehicles. • Amount of travels, energy consumption and journey quality of e-vehicles.
• Amount of use and pattern of the charging infrastructure installed.
• Degree of energy managed and supplied to EV by renewable sources.
• Willingness to use multimodality actions and invest in urban freight.
Data collected from transport facilities allow the calculation of KPIs identified by each mobility action with the exception of the impacts in the air quality emissions that need of a specific methodology to quantify the air emissions avoided. Thus, the evaluation approach in mySMARTLife establishes two measurement periods: baseline with ICE vehicles as reference for comparison and reporting period with e-vehicles. Additionally, this considers that distances travelled in both periods are the same. Then, the emissions avoided are measured as a function of consumed fuel or distance travelled per each type of vehicle and applying different emissions factors to each energy form used by them (e.g. diesel, electricity, etc). Standard emission factors for fuels are provided for European countries by Covenant of Mayor and internationally by IPCC whereas average consumptions per distance travelled for each vehicle is shared by its manufacturer.
This means that the vehicle characteristics (energy consumption and type of fuel consumed) are the only factors that change among baseline and re-porting period whereas other external factors to the vehicle are not analysed since the intervention do not have any influence on them (e.g. driving speed, driving style, road characteristics, traffic and weather conditions).

Urban infrastructure / Digitalization via ICTs
Digitalization of the city is also considered in this evaluation framework, which is reached through the implementation of ICT solutions in form of urban data platform. The method for the ICT analysis diverges from the previous infrastructure as the domain is completely different. In this specific case, software metrics are used to measure the level of digitalization of the city. Basically, the ICTs are quantified as: • Number of sensors and data-sets integrated in the urban platform.
• Number of available services.

• Number of available open data and open APIs (Application Programming Interface).
• Number of different users, so that usability can be determined.
• Response time, as performance metric to determine the time that any user should wait to receive the expected results from the urban platform services. • Scalability, as the capability for extending the resources of the urban platform.
• Availability, as the time during which the urban platform does not suffer crashes.

Economy
Economy pillar has as objective the measurement of the actions' effectiveness and the related business models, as well as the monetary impacts of the demonstrative actions in the cities, citizens and companies involved in their implementation. An analysis of costbenefit of the solutions is performed after the calculation of KPIs identified with the data provided by the action leaders once the actions are concluded. The economic evaluation is then implemented as follow: • Financial performance of the actions through the description of the funding/financial model and the identification of the costs and revenue structure associated with the implementation, operation and maintenance of the actions. • Societal, economic and environmental benefits of actions in terms of monetary terms through the evaluation of a variety of aspects such as: jobs created, expenditure in local economy, impact in business units and improvement in air environmental quality among others.

Citizens (social)
This pillar tries to reveal the degree of satisfaction of citizens with the project solutions deployed in the city, analyze the existence of a behavioral change in the society as well as the factors that influence in the level of acceptation. The analysis is rendered through tailored questionnaires according to the object to be assessed and the target audience defined which must be citizens affected by the interventions. The tool, which is distributed once project actions have finished, allows the evaluation of the final acceptance of the local population about new technologies installed, the willingness to invest in similar solutions and/or recommending these to others. This analysis also includes citizens' perception in the technical and economic design of the solution, the amount of information received and the degree of involvement in decision-making. Finally, an analysis of the respondent profile is performed (e.g. age, gender, socio-economic status) for considering this result in future social campaigns focused to upscale/replicate the solutions evaluated.
Additionally, this pillar is addressed to assess the target people reached in citizen engagement activities carried out in the project to inform about benefits of energy efficiency and RES solutions and to empower citizens in the urban transformation planning process. To this regard, it is evaluated the number of people reached and the diverse social background involved.

Implementation in the three cities of Nantes, Hamburg and Helsinki: Lessons learnt
This framework is, as mentioned before, deployed in the three cities of Nantes, Hamburg and Helsinki. During its implementation, the main results are translated into relevant lessons learnt that have been collected by partners involved in the execution of the evaluation.
At design of the evaluation framework, it may be summarized: • The definition of indicators in a joint way among technical partners from research centres and universities not involved in the implementation of actions and city actions leaders supposes a long process since the interests in the evaluation are different and a lack of understanding can occur. However, this process is needed in order to define an evaluation framework complete and comparable among different solutions implemented in different cities and avoiding including those KPIs that cannot be measured in the evaluation plans. • The design of an evaluation framework is a live process since it is required to be updated since unforeseen events arise (e.g. change in the technical/economic solution to be implemented or cancelation of action due to difficulties found for the implementation of the initial plan). • To know the whole context of the action to be executed and their expected impacts in the city is a key requirement for the definition of proper indicators. • It is essential to search different initiatives to find the most proper indicators for each city context. When any alternative does not exist for an evaluation purpose, new indicators can be developed in the project but this process is more complex than expected and require of certain expertise in advance. • The definition of evaluation boundaries of the action under evaluation is key for a common understanding of what to evaluate. During the deployment of evaluation framework, a set of conclusions and lessons learnt are also extracted.
• High quality of data is not reached in all the timeline in which an action is under evaluation. Non-completeness of data and outliers are common issues that appears during the data collection. • A follow-up process of the data collected through meters is needed to identify the best period for calculate KPIs and assure the KPI calculation has been made with quality enough data, therefore, being able of certifying the impact calculation is based on good quality data. • The joint evaluation among responsible of evaluation and the entity in charge of the action implementation is convenient in order to merge the results of the evaluation and the background of the action for reporting KPIs coherently. • Despite the coordination for reporting quality data and KPIs, different templates are used for different actors in the evaluation, which involves a more complex process than initially expected. • The coordination among different evaluation processes is key to guarantee the proper evaluation from a holistic approach. Two main figures should be considered for the evaluation of the actions: a main responsible of all the evaluations and a main evaluation contact in each city involved.

Discussion and conclusions
This paper has presented a KPI-driven evaluation framework in order to allow cities to measure the achieved impacts by the deployment of sustainable actions. In this sense, this framework has been applied in the three lighthouse cities of Nantes, Hamburg and Helsinki to assess the effects of the 150 actions implemented in the environment, economy, citizens and urban planning contexts. But also KPIs and methodologies approached allow evaluating the progress achieved towards the compliance of city targets established in energy transition urban plans. Latest but not least, it can be also helpful in the design of strategies to facilitate the replicability and scale up of sustainable solutions in cities, since the evaluation framework contributes to the identification of barriers to be faced (e.g. low social acceptance or low profitability of technologies installed).
Then, one of the main features of the evaluation framework is the benefit for the cities in terms of having a broad, flexible and replicable methodology for the evaluation of the impacts in terms of sustainability. The framework is thus replicable for any city that is preparing for becoming a smart city and advance towards the sustainability and climate neutrality and aims the evaluation of environmental, economic and social effects of the solutions implemented.
The applicability of the framework relies of the selection of the most suitable KPIs, among a pre-set list of indicators for each of the cities. Although it provides a wide set of indicators, not all are applicable or quantifiable, therefore, cities should firstly identify the ones that are aligned with the actions targets or Smart City urban plans. Cities then can adapt the components of the evaluation framework to their requirements. Although energy and mobility pillars are usually the most important ones in the sustainable transformation, the flexibility of the evaluation framework allows its partial implementation (e.g. just assessing digitalization), but always relying on real data for the reporting period.
The evaluation framework of mySMARTLife compiles KPIs already applied in previous initiatives and new indicators for the assessment of certain aspects not included in the literature. Moreover, the co-creation process, including different stakeholders of a smart city and counting with the experience of experts in different disciplines (energy, mobility, ICT, social, economy and governance), makes possible a holistic approach of the framework. Furthermore, the selection of standard protocols and methodologies in the various verticals (e.g. IMPVP for energy) generate a high degree of confidence and accuracy for the analysis of the real achievements and / or impacts. That is to say, establishing procedures to adjust baseline and reporting periods to be comparable, which is not trivial and many times neglected, generating non-realistic or distorted view of the impacts due to wrong assumptions.
For the case of mySMARTLife cities, the framework and their indicators have been used in a first stage of the project for obtaining the city audits, as well as determining the baseline of the actions. Currently, mySMARTLife project is deep into the monitoring stage, which will drive the final assessment where will apply adjustments to overcome the Covid influence in the values of thermal energy demands of buildings and use of mobility actions. Although already experience has been collected during the baseline, new feedback and lessons learnt will be valuable insights for further improvements. Additionally, related to the future work, other verticals could be integrated, such as waste management, in order to increase the holistic perspective of smart city.

Funding Information
The work presented in this paper is the result of the EU-funded project mySMART-Life, under the H2020 programme with Grant Agreement No 731297.