A Methodological Framework for the Selection of Key Performance Indicators to Assess Smart City Solutions

Abstract

Smart and sustainable cities are expected to form a cornerstone for achieving resource efficiency and sustainability worldwide. In this specific study we introduce a holistic framework for determining a repository of key performance indicators (KPIs) that are able to evaluate both business-as-usual and novel technologies and services related to smart city solutions. The framework includes six steps: (a) Clustering of the technology/service solutions into groups called Transition Tracks; (b) definition of the main groups of stakeholders; (c) definition of KPIs dimensions (or domains); (d) definition of KPIs repository per dimension; (e) definition of the scope of evaluation per KPI; and (f) threshold definition per KPI. The implementation of the proposed framework led to the development of a repository of 75 KPIs categorized in six dimensions (technical, environmental, economic, social, ICT and legal KPIs) with the corresponding levels of assessment and stakeholders’ group of interest. The proposed repository can serve as a great basis for similar projects to monitor and evaluate the performance of their solutions. Tips and guidance based on the actual implementation and lessons learned from a smart city project are provided.

1. Introduction

1.1. The Role of Cities towards Urban Sustainability

Cities are continuously and rapidly growing [1]. In 2015, almost 75% of the EU population lived in urban areas and this share is expected to rise to over 80% by 2050 [2]. Urban areas account for 60–80% of global energy consumption and about the same share of CO₂ emissions [3]. In 2016, nine out of 10 people who lived in cities were exposed to air that did not comply with the World Health Organization air quality guidelines for fine particulate matter (PM_2.5) [4]. Such detrimental effects can be primarily attributed to the actual energy mixture used in both the building and transport sector and the inefficiency of the building stock: an estimated 97% of the EU’s building stock is considered to be energy inefficient, while up to 75–85% of these buildings will continue to be utilized until at least 2050 [5].

Smart and sustainable cities are expected to form a cornerstone for achieving resource efficiency in Europe [6]. They can potentially deliver significant energy savings and increased resource efficiency, always in harmony with the cultural aesthetics of the urban and natural landscape. Technological and scientific advancements, especially when being integrated, are offering a rich pool of solutions that can help make a city a sustainable place to live. Such for instance, include: self-harvesting energy production and management (e.g., smart RES, district heating, smart metering, intelligent street lighting), smart storage (e.g., heat pumps, vehicle-to-grid (V2G) storage, innovative batteries), smart mobility (e.g., electric vehicles (EV), EV-charging infrastructure, solar V2G car sharing), city information platforms (e.g., smart open data city platform, urban monitoring) and citizen engagement and co-creation approaches and solutions (e.g., apps for visualizing energy consumption). In light of such a fast transitioning environment, the need for strategies that help cities to smartly integrate technological solutions becomes more and more apparent. This creates new sources of revenue for projects, new business models for value capture and new opportunities for investors [7]. Accelerating the transition to a low-carbon competitive economy is both an urgent necessity and a great opportunity for cities in Europe [8].

The COP21 Paris Agreement (2015) recognizes the role of cities and calls on them to rapidly reduce greenhouse gas emission and adapt to climate change. The European Energy Union and the Energy and Climate Policy Framework for 2030, established ambitious commitments (a) to reduce greenhouse gas emissions by at least 40% by 2030 (compared to the 1990 levels); (b) increase the share of renewable energy consumed above 27%; and (c) set an energy savings target of 27% by 2030 [9]. The EU is also committed to implement the 2030 Agenda for Sustainable Development [10]. According to this agenda, by 2030 the EU should provide access to safe, affordable, accessible and sustainable transport systems, enhance inclusive and sustainable urbanization and reduce the adverse per capita environmental impact of cities. As a result, European policymakers are orchestrating efforts to make urban areas more sustainable by encouraging smart city initiatives.

Smart city projects play a vital role in this attempt. The key objective of smart city projects is to improve sustainability of the city and quality of life of its inhabitants by demonstrating solutions that are able to solve urban problems in an efficient way [11]. Several smart city projects have been funded by the European Commission (EC) in recent years under the 7th Framework Programme (FP7) and Horizon 2020 Smart Cities and Communities (SCC) with a view to demonstrate integrated commercial-scale solutions with a high market potential that can address this objective.

1.2. The Role of KPIs and Smart City Assessment Frameworks

Although the interest for smart city projects and initiatives has been continuously growing, there has been less progress regarding the evaluation and measurement of their outcomes. Effective assessment is significant to prove the value of smart city projects and initiatives and the benefits delivered to city authorities and all city stakeholders [12]. The implementation of smart city solutions still faces some serious challenges due to prodigious data processing demands and heterogeneity of connected smart components [13]. To support the monitoring of relevant projects and initiatives, Key Performance Indicators (KPIs) can be a universal instrument to evaluate the progress of smart city strategies [14].

KPIs, in general, measure the effectiveness of a project towards the achievement of specific key objectives. The process of selecting KPIs also assists to clarifying the project’s measures of success. In general, indicators (and even more so, KPIs) should express as precisely as possible to what extent an aim, a goal or a standard has been reached or even surpassed. Data that is not linked to standards or specific goals, can be used as quantitative background information (e.g., total investments), but is not suited for evaluative purposes.

The need for a uniform monitoring of the energy smartification throughout Europe has led to initiatives promoting the cooperation and exchanging of know-how among European cities. Such initiatives as the Smart Cities Information System [15] (SCIS) and CITYkeys [16] have created platforms of interaction along with a list of KPIs, each for the evaluation of systems and technologies demonstrated in smart city projects.

The SCIS focuses on the development of indicators to measure technical and economic aspects of energy related measures. These are applicable to European funded demonstration projects for SCC, energy efficient buildings and designated projects funded under the calls for energy efficiency. Launched with support from the EC, SCIS encompasses data, experience and stories collected from completed, ongoing and future projects.

Funded by the European Union HORIZON 2020 program, CITYkeys developed and validated, with the aid of cities, KPIs and data collection procedures for the common and transparent monitoring as well as the comparability of smart city solutions across European cities. CITYkeys was built on existing smart city and sustainable city assessment frameworks. The bases of the framework are the traditional sustainability categories of People, Profit and Planet, but the performance measurement framework includes specific smart city KPIs that go beyond the traditional division into these categories and measure the integration level and openness of the technological solutions.

Table 1. Indicative assessment frameworks by EC-funded projects.

Project Name	Description	Assessment Framework	Ref.
MATCHUP—Maximizing the upscaling and replication potential of high level urban transformation strategies	Based on a citizen-centric approach, MATCHUP demonstrates solutions in the energy, mobility and ICT fields with a view to boost local economies and their quality of life. MATCHUP will deploy large scale demonstration projects in three cities: Valencia (ES), Dresden (DE) and Antalya (TR).	The framework structure designed for the evaluation of the performance of a city is based in the definition of city indicators in four fields (energy efficiency/mobility and transport/ICTs and urban platform/citizens and society), further grouped into 27 domains consisting of 188 indicators. The indicators can be further aggregated into indices.	[17]
SmartEnCity—Towards Smart Zero CO2 Cities across Europe	SmartEnCity’s main objective is to develop a highly adaptable and replicable systemic approach for transforming European cities into sustainable, smart and resource-efficient urban environments. The SmartEnCity concept will be implemented in three cities, Vitoria-Gasteiz (ES), Tartu (EE) and Sonderborg (DK).	A procedure is defined which integrates evaluation protocols to estimate the overall impact and performance of solution at city level with the utilization of high level indicators in the energy, transport and ICT sectors. A generalized evaluation plan is available including indicators for city diagnosis (six domains; 149 indicators), to evaluate interventions performance (139 indicators) and quantify the impact at the city level.	[18,19]
Triangulum—The Three Point Project/Demonstrate. Disseminate. Replicate.	Triangulum sets to demonstrate, disseminate and replicate solutions for Europe’s future smart cites. The cities of Manchester (UK), Eindhoven (NL) and Stavanger (NO) serve as a testbed for innovative projects focusing on sustainable mobility, energy, ICT and business opportunities.	An approach is followed distinguishing the assessment of impacts of demonstration projects and the process through which they are monitored. A seven-stage methodology for developing impact indicators is proposed. Detailed impact and data mapping tables are available. Five impact domains have been defined: energy, transport, citizen engagement, socioeconomic, financial and ICT.	[20]
GrowSmarter—Transforming Cities for a Smart, Sustainable Europe	GrowSmarter aims to stimulate city uptake of smart solutions by using the three cities of Stockholm (SE), Cologne (DE) and Barcelona (ES) as a way to showcase 12 Smart City solutions: from advanced information and communication technology and better connected urban mobility, to incorporating RES directly into the city’s supply network.	The evaluation of the project’s measures is done with the purpose of determining if the expected goals are met and what are the social costs due to the implementation of the measures. The measures are divided into three main categories (low energy districts, integrated infrastructure and sustainable urban mobility) and various sub-categories. For each measure at least one KPI is determined.	[21]

presents a number of indicative SCC EC-funded projects including some key characteristics of the assessment framework, they developed to evaluate and monitor performance.

Apart from relevant projects, scientific studies are also available to introduce various assessment frameworks that attempt to evaluate smart city performance. Lombardi et al. [22] propose a model that involves the civil society along with universities, industry and government for classifying smart city performance indicators. Hara et al. [23] introduce a set of KPIs for smart cities building upon the Gross Social Feel-Good Index including six layers of indicators, i.e., those of environment, economy, comfort, health, safety and satisfaction. Studies are also available proposing indicator systems to measure the progress of low-carbon developments at the city level [24]. Girardi and Temporelli [25] present a methodology called Smartainability that can estimate through quantitative and qualitative KPIs, to what extent smart cities are sustainable due to the deployment of smart technologies. The methodology has been tested only on a district level (Expo Milano 2015 site) so far, and the estimation can be performed before the technologies are deployed. Another interesting study is that of Dall’O’ et al. [26] providing a method for assessing the smartness of a city through a set of indicators focusing on small and medium-size cities and communities. The indicators selected are consistent with the ISO 37120 standard and Sustainable Energy Action Plans under the Covenant of Mayors Initiative. Several studies introduce ranking systems to assess and compare the smartness of cities through indexes but involve the high risk of losing information on the complexity of smart cities [27]. Li et al. [28] propose a systematic approach, utilizing a bi-index method, to identify stakeholders and KPIs for multi-level (from building to district) energy performance analysis.

It should be noted that various definitions of the term “smart city” are available [27] that are moving beyond the inclusion of ICT aspects also referring to quality of life. This lack of uniformity of smart city definitions can lead to diverse results and poses challenges to the target setting of cities. Indeed, Ahvenniemi et al. [29] analysed 16 sets of city assessment frameworks comprising 958 indicators indicating noticeable differences between smart cities and urban sustainability frameworks especially regarding their tendency to highlight environmental, social and economic aspects. Another interesting conclusion presented in this study is that, although decreasing energy should be an important goal for smart cities, the use of energy related KPIs is limited in the smart city frameworks.

2. Methodology

The objective of this study is to present a methodological framework for determining a repository of KPIs that are able to evaluate both conventional and novel technologies and services related to smart city solutions. In parallel, this framework needs to be citizen-centric, since the citizens’ benefits should be the aim of any type of solutions rendering the city smart and more sustainable. Realizing citizen engagement, co-creation of inclusive information services for citizens are crucial drivers and enablers in the urban energy transition. Therefore, citizens play an important role in the smart city demonstration activities.

The proposed framework is adopted under the EU-funded smart city project IRIS [30]. One of the main goals of IRIS is to evaluate and optimize the operation of smart systems, operating on a district level and potentially cover the needs of a whole city, building upon RES-based technologies. RES solutions can feed both thermal/cooling and electrical needs, with interconnected grids, promoting the idea of synergy among the three main energy vectors. However, the current knowledge on how such grids should be designed to operate in a synergetic manner is limited and technological advancements and demonstration activities need to be carried out before the level of technological maturity reaches a readiness level of nine (TRL9) and the technology can be commercialized. Through demonstration activities in three Lighthouse Cities (Utrecht, The Netherlands; Nice, Cote d’Azur, France; and Gothenburg, Sweden), both thermal and electricity grid performance evaluation results will be used to mature the technologies.

The framework for selecting the KPIs has been finalised in collaboration with key partners from the three Lighthouse Cities; thus it reflects the actual experiences and needs of the cities. Our approach differentiates from other similar frameworks for three key reasons: (a) specific emphasis is given on integrating all relevant stakeholder points of view concerning the deployment of smart city solutions; (b) an out-of-the box thinking is adopted through the inclusion of targeted technological and legal KPI dimensions, in addition to the standard (economic, environmental, social, ICT) ones usually applied; and (c) a clear definition of the level of evaluation per KPI is pre-determined. The definition of KPIs is conducted in accordance with other European projects. The framework proposed includes the following six steps: (a) Clustering of the technology/service solutions into groups called Transition Tracks; (b) definition of the main groups of stakeholders; (c) definition of KPIs dimensions (or domains); (d) definition of KPIs repository per dimension; (e) definition of the scope of evaluation per KPI; and (f) threshold definition per KPI. The next steps for aligning the results with monitoring and evaluation planning on a project level are also presented. The specific framework is generic on purpose, in order to satisfy various assessment requirements of a proposed technology solution. A detailed description of the framework is presented in the following sections.

2.1. Clustering Solutions in Transition Tracks

Smart city solutions can be clustered under several Transition Tracks (TT) that cover the majority of the issues that need to be addressed, while aiming at the energy transition of urban systems. The term TT is used to group interdisciplinary and complementary integrated solutions (IS) in terms of the need they address under a common umbrella. The assessment of new technology solutions is a very important step towards the further development of smart cities; thus, the approach on this should be as holistic as possible. To do that, allowing a smooth coordination, management and monitoring of various solutions on a city level and to decrease the complexity of that as much as possible, the IRIS project adopts a categorization under five (5) TTs (provided below), thus covering the whole spectrum of solutions according to their orientation. However, such a categorization cannot be limited to this approach, since other types of schemes can be followed. Specifically, the first three TTs enable the transition towards reduced energy demand and increased shares of renewables and e-mobility in the urban energy and mobility systems, aiming at:

• TT#1: Smart renewables and closed-loop energy positive districts: Integrating (a) a high share of locally produced and consumed renewable energy at district scale; (b) energy savings at building level reducing the citizens’ energy bill; and (c) energy savings at the district level. These solutions integrate high renewables penetration like district-scale PV and biomass for district heating, near-zero energy housing retrofit, energy efficient low temperature district heating and smart public lighting that is energy efficient, powered by renewables and connected to the district energy system.
• TT#2: Smart Energy Management and Storage for Grid Flexibility: Integrating smart energy management and renewable energy storage for (a) maximum profits of renewable power/heat/gas; (b) maximum self-consumption reducing grid stress and curtailment; and (c) unlocking the financial value of grid flexibility. These solutions include smart ICT to interconnect energy management systems at home, building and district level, and to integrate maximal renewables production, V2G storage in e-cars operated in car sharing systems with additional stationary energy storage.
• TT#3: Smart e-Mobility Sector: Integrating electric vehicles and e-car sharing systems in the urban mobility system offering (a) local zero-emission mobility; (b) lower household mobility costs; and (c) smart energy storage in V2G car batteries. These solutions include extensive deployment of (V2G) e-cars, exploitation of (V2G) e-cars in local car sharing systems and district-wide smart (V2G) charging stations powered mainly by renewables.

ICT play a pivotal role as enabler of smart integration, unlocking the synergy potential of divergent energy and mobility solutions and offering new meaningful insights and services thanks to the data generated by the integrated solutions. Given this condition and the fact that cities can act as large-scale demonstrators of integrated solutions, we propose a fourth TT as well namely, the City Innovation Platform. This includes:

• TT#4: City Innovation Platform (CIP): Cutting edge information technology and data framework enabling the above-mentioned solutions, maximizing cost-effectiveness of the integrated infrastructure. Next, the City Innovation Platform with open, standards-based application program interfaces (APIs) provides meaningful data and information services for households, municipalities and other stakeholders, allowing for a data market with new business models. A common architecture, harmonized data models and a sustainable data governance plan ensure the interoperability and replicability of the solutions, transferring them from city to city. The city data market and the service marketplace manage access to all data and services, with appropriate licenses and flexible pricing models in and across cities and allowing real-time KPI monitoring and benchmarking of smart energy and mobility performances.

However, except for the technical sector, smart city projects should start from people—by focusing on citizen needs, embracing citizen-centric design and their search for an integral quality of life. To this end, we propose one additional TT, which focuses on citizen engagement, named as:

• TT#5: Citizen Engagement and Co-Creation: This orients to design and demonstration of feedback mechanisms and inclusive services for citizens to achieve that they are intrinsically motivated to (a) save energy; (b) shift their energy consumption to periods with redundant renewables; (c) use electric vehicles; and (d) change the vehicle ownership culture towards a use or common mobility assets culture. These solutions include game-theory based engagement methods and instruments ranging from co-creating infotainment apps, local school campaigns, offering training on the job to students living in the district by partaking in the demo activities, competitive energy games using the home energy management system, energy ambassadors creating local energy communities, to crowd-funding creating a sense of being part of the solution.

It should be mentioned that the proposed categorization is not absolute and was developed with a view to provide a holistic evaluation covering all necessary smart city aspects. Other projects may need to adapt the proposed TT based on their special characteristics and the technologies they aim to demonstrate.

2.2. Definition of the Main Groups of Stakeholders

The inclusion of relevant stakeholders’ opinion in decision-making, is considered to be of high significance—no one knows better the needs and the other parameters of a problem than the people affected by and affecting it. For smart city projects, a sensible stakeholder categorization can include the following groups, so that most of the stakeholders can be actively participating/represented in the evaluation of the solutions (from a first level) and of the city (to a final level): (a) Distribution System Operators (DSOs), (b) consumers (end-users), (c) technology and services providers and (d) policy-making bodies and governance. Citizens and/or representative citizen groups can also be distinct groups of stakeholders, but in this study we will consider them as part of the consumers group (assuming that all citizens will eventually become consumers of the services provided). The four defined stakeholder groups attempt to represent all the stakeholder points of view concerning the deployment of smart city solutions.

2.2.1. Distribution System Operators (DSOs)

A DSO is responsible for the management and operation of the distribution network of electricity. In addition, depending on the legislation of each country, a DSO or a DNO (Distribution Network Operator) might be responsible for energy consumption requests for reduction. Sometimes, in the competitive electricity market, the distribution of electricity is usually a monopoly controlled by the regulating authorities. It is of high interest for smart city projects to evaluate system performance from the DSO’s point of view. The main aim of a DSO is the sustainability, reliability and flexibility of the system, the ability of the distribution grid to reciprocate to the various consumer needs every single moment (industry and domestic-scale), or the ability to modify the load curve via peak shaving techniques. Similar to the DSOs (a term mostly used for electricity distribution), are the distributors of heating/cooling or other types of energy vectors (e.g., natural gas). Since this specific category of distributors are not named with a standard format, we include them as part of the DSOs. When we refer to DSOs we actually imply either the electricity or the heating/cooling providers, dependent on the type of smart city solution.

2.2.2. Consumers (End-Users)

The citizen engagement has an enhanced role in the modern struggle for increased energy efficiency. Many smart city projects try to ensure and promote the active participation of end users—citizens in the market and grid operations; thus, special focus is granted to the evaluation of the end users’ performance within the context of the projects. Concerning TT#1 and TT#2, the consumers can be classified as residential and non-residential, if someone wants to examine end-user’s role in the grid level in more detail: (a) Residential consumers: Their main interest is the reduction in the energy consumption, as well as in the energy price, with a probable environmental care about the electricity mixture. Residential consumers are willing to renovate their residences with energy solutions that lower the energy bills; and (b) non-residential consumers: Their main interest is grid security and sustainability, as well as the provision of energy for a low price, with a care for a socio-economic improvement concerning the local energy consumption. They include factories, facilities, offices and generally non-residential buildings, municipal or private, with high energy demand, usually in a fixed daily timetable.

Due to the variety of solutions in TT#3, various consumers are identified in the mobility system. The first group of consumers are the citizens, who in the electro-mobility and car-sharing solutions aim at making use of a common pool of electric vehicles for satisfying their mobility needs instead of using their own car. In the other services (e.g., electric buses) the citizens receive the final benefit in the form of less travel time or pollution. The second group of consumers are the public transport operators, which can upgrade their fleet of vehicles to electric ones or have priority at intersections. Concerning TT#4 and TT#5, all citizens of the city should be considered, since they can potentially be end-users of the services provided.

2.2.3. Technology and Service Providers (TSPs)

In this category, the private sector composed of industry, technological companies and service providers, including SMEs, have a crucial role by connecting smart cities eco-system and supporting the provision of the solutions in different ways. ESCOs, aggregators and retailers are interested in monitoring and analysing the behaviour of the end-users, in validating the operational credibility of the technological installations supporting alternative DR schemes, in identifying potential profile deviations, and in evaluating the impact of the benefits generated by the applied policies. Towards this direction, it is essential for smart city projects to evaluate the impact of the different strategies (demand response, storage and EV management) to the different market stakeholders.

Furthermore, the term ‘prosumers’ refers to agents that both consume and produce energy at local level. The growth of small and medium-sized agents using solar photovoltaic panels, smart meters, vehicle-to-grid electric vehicles, home batteries and other ‘smart’ devices, induces the increase in flexibility of the electricity networks. As the number of prosumers increases, the electricity sector is likely to undergo significant changes over the coming years, offering possibilities for the greening of the system. Prosumers could be alternatively included in the end-users category. On the other hand, they invest on energy, sometimes even having profit instead of paying for the energy they consume; thus, they tend to behave more like a market operator. The main interest of a TSP is the profit in an energy venture, a fast payback period of the initial capital cost and a large investment lifetime. Various market operators should be asked for their opinion, beginning from the ones that own the largest share in the electricity mixture in each city, to small prosumers.

In TT#1 and TT#4, the TSPs are responsible for executing and supervising the implementation of the solutions. In some cases, their role is also to promote citizen engagement (TT#5) in order to reach the appropriate business models. At the district level there are various types of market operators, such as housing corporations, who have experience in testing combined energy efficient solutions in buildings and companies manufacturing and supplying electrical equipment who deal with the implementation and exploitation of advanced devices and applications. In TT#2, the traditional utility operators and their expected new business roles should be considered. ESCOs and DR aggregators are the responsible parties to manage the technology to perform DR and negotiate on behalf of their customers with the operator for the provided services. In TT#3, the role of the TSPs is to implement, maintain and run the solutions. They are responsible for both the development and the commercial exploitation of the solutions in the market. They range from traffic management providers and vehicle manufacturers (usually large companies) dealing with the priority service and the electric vehicles, respectively, to service providers (usually SMEs) able to provide car-sharing services or dedicated apps.

2.2.4. Policy-Making Bodies and Governance

The current regulators represent an important stakeholder group that needs to be considered, too. They are responsible for a normal and steady operation of the energy market, its gradual privatization, and they provide the basis of the regulatory framework, which is responsible for the determination of the quality standards and the basic rules. A clear and consistent vision for the smart city has not been adopted by legislators or regulators. Even though there is a great discussion about individual technologies, such as renewables or about specific energy issues, little progress about the overall vision for a modernized smart city and grids is detected.

In the TT#1, the policy making and municipal authorities are responsible for providing installations and services towards the implementation of energy efficient solutions with main objective being the socio-economic development of the district and the reduction of emissions. In TT#2, the municipality is partly responsible concerning the citizen engagement regarding the application and success of the policies developed for the increase of the flexibility of the grid. In TT#3, the policy making and governance authorities are responsible for providing mobility services to the citizens and keeping the pollution levels under desired thresholds. Smart city solutions are supporting them in these objectives by reducing emissions in the urban regions (electro-mobility, car-sharing schemes) while providing a communication channel with the citizens (urban pulse) for increasing their awareness and sensibility towards them. Policy-making bodies should also make sure that the vast amount of data generated during the implementation and monitoring of smart city solutions are organized and utilized in such a way that enhances their decision-making capacity (TT#4). In TT#5 the governance should be tested in its ability to get in touch and motivate a considerable number of end-users, mainly domestic and SMEs, in order to get engaged in the proposed solutions.

2.3. Definition of KPIs Dimensions

A basic step of the proposed framework lies on the definition of KPIs dimensions, namely technical, environmental, economic, social, ICT and legal. These dimensions are complementing each other to set the holistic performance framework and are defined as:

• KPIs measuring Technical Performance, such as the energy consumption, the RES generation ratio, the peak load reduction, etc.
• KPIs measuring Environmental Performance, such as CO₂ emissions reduction.
• KPIs measuring Economic Performance, such as the average cost of energy consumption, the average estimation of cost savings, etc.
• KPIs measuring Social Performance, such as the degree of users’ satisfaction.
• KPIs measuring the Performance of ICT, such as people utilizing apps which enable the residents to monitor and analyse their energy consumptions, home energy management systems, etc.
• KPIs measuring Legal Performance, such as the level of adaptation of electricity/heat integration in the legal framework, legal barriers for usage of biofuels for energy exploitation purposes, etc.

The current proposed dimension categorization is not the only one that can be adopted. There are other relevant frameworks, either close to the one presented (e.g., SCIS) [15], or quite different (e.g., CITYKeys) [16]. We propose the one presented as a more holistic option in studies for systems operation characterized by a medium to high TRL. The legal dimension is a new aspect that is presented in this study and many stakeholders demand it nowadays, given the condition that the current EU legislative framework is not uniform, but fragmented across the various EU countries.

Table 2. Selecting the KPIs’ dimensions.

Questions for the Evaluation of the 2nd Life Batteries Implementation	Dimension
Do they need maintenance often?	Technical
Are there any CO2 savings because of their implementation?	Environmental
Is their cost per year higher than that of brand-new batteries?	Economic
Is the idea publicly accepted or are they not trusted?	Social
Do they correlate well with other components in a smart grid?	ICT
Is their use accepted by the legal framework?	Legal

shows an example of the relevance between the specific dimensions’ categorization with the main questions that have to be posed for the evaluation of a technology solution (e.g., second-life batteries).

2.3.1. Technical Dimension

KPIs in the technical dimension measure the effectiveness of a given solution with respect to the operating parameters and technical constraints acting on electricity/thermal grid and active/passive users, as well as the effectiveness of technology solutions concerning heating/cooling, electrification and mobility, on both a building and a district level. Representative technical KPIs can be obtained by gathering the electrical metrics of the network (e.g., voltages/currents collected along feeders, number of e-charging stations and V2G vehicles deployed in the area, proportion of RES integration) and of customers and producers consumption profiles (e.g., active/reactive energy/power exchanged with the network, usage of the car-sharing vehicles, energy consumption for cooling, heating and hot water). In some cases, the KPIs need to be supported by numerical simulations on the basis of a thermal or an electricity grid model, representing the operation of a building or a district and/or possible actual measurements collected during the grid operation.

The interest in these KPIs changes depending on the perspective of the various stakeholders, such as DSOs, who are mainly concerned about KPIs related to the MV/LV network operation, while customers are focused on KPIs assessing the performance of a new approach/strategy at their premises. However, other factors exist that could affect the relevance of the KPIs considered in the different situations, for example the regulatory framework in force, which could promote an improvement of the quality of service with reference to specific technical indices (SAIDI/SAIFI), or business cases applying in each particular scenario, also in relationship with the target performances defined in the economic dimension.

2.3.2. Environmental Dimension

KPIs in the environmental dimension are important for understanding and evaluating the environmental impact of energy/storage, smart grid distribution, heating/cooling and mobility related solutions and are important for a smart system planning and operation. The environmental KPIs can be used to evaluate the efficiency of the solutions demonstrated in the cities from the viewpoint of the expected environmental impact. For example, there are KPIs that refer to the operational phase (noise and pollen pollution exposure), as well as to the end-of-life phase (EROI). The main focus is on operational phase evaluation through the definition of KPIs that set the framework for day-to-day evaluation, while a life cycle analysis (LCA) methodology can be applied for the determination of environmental aspects and potential impacts of a product or system from raw material extraction through production, use and disposal, while evaluating possible recycling routes following a cradle-to-cradle approach (e.g., second-life batteries).

2.3.3. Economic Dimension

The economic performance evaluation takes into account the business efficiency of each application and usage scenario from the market stakeholder perspective. Different demonstrators offer different value propositions to stakeholders and thus, special focus should be delivered to the definition of KPIs that reflect this specific viewpoint. Among the objectives of smart city projects is to provide market-viable solutions, defining business-oriented KPIs to evaluate the day-to-day performance of the applied tools and applications. For example, the residents of apartments would like to have a view of the economic benefit produced by their flexible consumption behaviour. They may be willing to sacrifice part of their comfort to achieve lower energy bills and they would like to know what the cost/benefit ratio is. Likewise, the business stakeholder (DR aggregator) will like to know the actual benefit from the implementation of DR strategies in a portfolio of customers. Concerning the closed-loop energy positive districts, the local communities try to promote and support energy efficient measures and solutions targeting to economic and business development by reducing the electricity bills and engaging consumers to an energy sensitive attitude. With regards to mobility, the city is willing to reduce congestion and pollution as well as parking places, while the consumers are willing to increase the usage of the vehicles (the system operator) and to increase the availability of shared vehicles (the citizens). Once again, the overall business and economic analysis is closely related to the definition of business stakeholders in the project, along with the selection of business models and associated scenarios to be examined at the demonstration sites of each project.

2.3.4. Social Dimension

The social aspects of energy projects were found to be the less popular among the employed KPIs in previous similar studies. The chosen indicators reveal that attitudes towards energy are interrelated with demand response mechanisms [31] and such KPIs can be used to evaluate the extent to which the end-users (citizens in most cases) are willing to participate and be self-motivated for further demonstration and application of the demonstrated solutions. Generally, the social performance domain visualizes the impact of a technology, scheme or policy to social factors like local wealth, unemployment, satisfaction, or even more specific, like the effect on the use of public transport, the health care system, etc. A popular approach used in literature for expressing the social KPIs is the Likert scale, since it is a sensible way for quantifying a qualitative value. Partners responsible for such KPIs should determine target groups among the various stakeholders and pose them a question that needs a Likert answer.

2.3.5. ICT Dimension

The ICT dimension could be concerned as one of the technology pillars of smart city projects. A smart city tends to connect the various energy operations, including generation and consumption, with a central energy management platform that interacts with citizens and generally all stakeholders. ICT is used as a KPI dimension because it indicates the interoperability of the technology solution presented, its ability to correlate with the rest components of the energy grid, its capability for two-way interaction with the citizens. The KPIs listed in the ICT dimension refer to the City Innovation Platform mainly, regarding the monitoring and sensible control of the proposed technology solutions. This dimension is appropriate for smart city projects regarding the implementation of new components in an existing smart grid.

2.3.6. Legal Dimension

KPIs in the Legal dimension mainly monitor the legislative background concerning the application of the proposed solutions. The specific dimension is not commonly used, but it is of great importance in the R and I, since law-making bodies are often not flexible enough to follow the progress of technology, especially when these are related to strongly regulated/protected markets (energy and mobility). This is a serious problem, especially in EU, since most of the already mature technologies cannot be actually implemented and operate in real-life conditions, because there is not the necessary legal background, allowing their actual life operation. Even more important are the economic results. An immediate legislative support of a new technology can give a serious handicap for its developer and end-user in a world-wide market, where the exploitation of innovations is one of the most serious sources of profit. Generally, market operators (including DSOs and prosumers) need a steady legislation concerning their invested capital, and fast response concerning the legislative background of innovations. The legal KPIs mainly evaluate the governance in terms of legislative flexibility. This flexibility is difficult to be objectively quantified, so the subjective point of view of several stakeholders is needed, usually in the form of a percentage scale.

2.4. Definition of KPIs Repository

The use of quantitative indicators is valuable not only to describe/assess as accurately as possible individual characteristics of a technology, but also to evaluate them, in a simple and on a fair basis way, against other solutions of the same characteristics serving the same role. Such an approach facilitates the direct comparison of available technologies, designed for the same scope. A filtering procedure according to predefined criteria is necessary to narrow down the vast number of potential indicators that can be included in the repository [32]. To achieve having a shortlist of indicators and following bilateral discussions with key partners from the three LH cities, a set of criteria was used in IRIS project and are proposed in this study, using as a basis the CIVITAS framework [33], according to which each set of KPIs should be characterized by:

• Relevance: Each indicator should have a significant importance for the evaluation process. The indicators should be selected and defined in such a way that the implementation of the smart city project provides a clear signal in the change of the indicator value.
• Completeness: The set of indicators should consider all aspects of the implementation of smart city projects.
• Availability: Data for the indicators should be easily available. As the inventory for gathering the data for the indicators should be kept as limited as possible, in time and effort, the indicators should be based on data that either: (a) are available from the project leader or others involved in the innovation case that is being evaluated; (b) or can easily be compiled from public sources, and c) or can easily be gathered from interviews, maps, or terrain observations.
• Measurability: The identified indicators should be capable of being measured, preferably as objectively as possible. For the majority of indicators in the ICT, social and legal dimensions, quantitative measurability is limited. Social sciences provide approaches to deal with qualitative information in a semi-quantitative way [34].
• Reliability: The definitions of the indicators should be clear and not open for different interpretations. This holds for the definition itself and for the calculation methods behind the indicator.
• Familiarity: The indicators should be easy to understand by the users.
• Non-redundancy: Indicators within a system/framework should not measure the same aspect of a subtheme.
• Independence: Small changes in the measurements of an indicator should not influence preferences assigned to other indicators in the evaluation. However, as the current energy systems in many cities are still largely based on fossil fuels, there is a direct relation between a reduction in the use of energy and the reduction of the emission of carbon dioxide. This will lead, to a certain extent, to double-counting the impact.

In the case of the IRIS project the definition of KPIs was conducted in accordance with other projects enhancing the way towards the energy smartification of European cities (Figure 1). The majority of the proposed KPIs were mainly taken by the CITYKeys and SCIS KPI pools. Some KPIs, mostly the Legal ones, did not exist in previous literature. Specifically, the legal KPIs were firstly used in the SMILE project [35]. The list was discussed among the demonstrators in order to make the appropriate additions and adjustments according to the project needs. The first and most important criterion for the definition of KPIs by the demonstrators should be if this KPI can actually be measured or calculated.

The collection of primary (measurement-based) data by the different demonstrators is crucial for the calculation of the KPIs, as well as its overall evaluation in terms of the different pilots and its replication ability. In most cases, the data is described by its units and the time point/period it refers to. The data source directs to the methodology used for the data collection. The most common cases in smart city projects are: (a) existing web services; (b) smart meters; (c) plug-level meters; (d) utility bills; (e) battery management systems and EV charging platforms; (f) grid power quality analyser; and (g) supervisory control and data acquisition (SCADA).

Except for the raw measurements associated with the real-time operation of the city information platforms, many additional parameters, not easily measured, will need to be determined for the calculation of KPIs. These secondary (model-based) data consist of the configuration parameters and normalization factors that will enable the model-based KPIs calculation. In some cases (e.g., retailer or market prices), dynamically updated values can be considered and, thus, interfaces with external service providers (e.g., energy markets) need to be defined.

The second criterion is the importance of the specific KPI for their opinion as an ecosystem (taking into consideration the opinion of their relevant ecosystem stakeholders). These two parameters are enough for their prioritization, based on our experience (see the Results section for the complete KPI repository).

2.5. Definition of the Levels of Evaluation Per KPI

The evaluation of smart city solutions with the utilization of KPIs entails different levels of spatial aggregation which goes from a single building to a whole district or city.

Table 3. Different levels of evaluation (adapted from SCIS [15]).

Level of Evaluation	Description
Building	This concerns the energy performance balance of: (a) the delivered energy required to meet the energy needs; and (b) the exported energy. The delivered energy is to be expressed per energy carrier. If part of this delivered energy is allocated to energy export, it also needs to be specified in the data collection where the electricity produced is not used in the building. In this case the corresponding amount of gas allocated to electricity production shall be specified in order to be able to calculate the energy performance of the building. At the building level the data required is (calculation procedure goes from the energy needs to the primary energy): (a) Energy needs per area of application; (b) energy technologies supplying these energy needs; (c) energy storage units; and (d) delivered energy to each energy supply units expressed per energy carrier.
Set of Buildings	The assessment for a set of buildings is done by the aggregation of building units. The indicators can then be calculated for the sum of the buildings as a group.
Energy Supply Unit (ESU)	At the ESU level, the approach to be followed is similar to the building level. Delivered energy per energy carrier and output energy allocated to energy carrier need to be specified. Additionally, and depending on the energy supply unit, different indicators can be calculated. This evaluation level refers to building integrated energy supply units as well as large-scale energy supply units.
Set of Energy Supply Units	The assessment for a set of ESU is done by the aggregation of energy supply units. The indicators can then be calculated for the sum of the energy supply units.
Neighbourhood	This level of evaluation (area or neighbourhood) is composed by the aggregation of different entities. The energy flows at this point need to be defined. The following information is required to define the energy system: (a) Energy carriers used at the implementation area level and the primary energy factors corresponding to this area; (b) demonstration units involved (buildings, energy supply units, storage units and distribution systems); (c) delivered energy to each ESU and building allocated to the corresponding energy carrier; (d) output energy of each ESU and, if applicable, output energy exported out of the boundary allocated to the amount of delivered energy carrier; and (e) energy flows between technologies and buildings (which ESU is supplying which building or ESU). Due to the complexity of these systems, indicators can only be calculated if a full set of data is available.
City	Most smart city projects demonstrate solutions in city environment (the approach to be followed is similar to the neighbourhood but on a wider scale). Evaluation at a national level may be needed for the legal performance indicators.

presents the different levels of aggregation that can be evaluated using the proposed framework. The level of evaluation per KPI needs to be clearly defined to increase the transparency of the results.

2.6. Threshold Definition Per KPI

After the final definition of the KPI repository, the threshold definition is an important and sometimes difficult task, since it sets the quantified objectives of the project. Each KPI will finally acquire a value calculated throughout the monitoring of the project. The actual evaluation of the presented technology solution has to be done with the comparison of the KPI final value with a threshold that separates success to failure. This separation line can have the form of:

• Baseline: Baseline is a measurement taken in the beginning of the project. If the threshold is the baseline, then the scope is to check the difference in the actual result because of the implementation of the proposed technology solution.
• Business as Usual (BaU): BaU is a more complex threshold, since it takes into consideration the change in the value of the KPI throughout the time period of the project, without the implementation of the tested technology solution. It takes into account the general tendency of the change in the KPI value. The BaU threshold comprises a more realistic view on the tested technology impact on its environment but is more difficult to be estimated.
• Other threshold: A threshold value could be defined by the evaluator, without it being either a baseline or a BaU. This could apply to KPIs that have not been estimated in the past, such as the legal KPIs or some social KPIs that are measured with the Likert scale.

Either way, the threshold is defined regarding the necessary literature survey. The demonstrators have the last word in the threshold determination since they are able to take into consideration the most aspects influencing the performance of the tested technology solutions in their city environment.

2.7. Align with Monitoring and Evaluation Planning

The proposed framework deals with the selection of the KPIs that can be used for the evaluation and monitoring of the IS being grouped under the TT (see Section 2.1) that will be actually demonstrated during a smart city project. As a next step there is a need to define the list of KPIs, not only for the solution demonstration activities, but to go a step forward towards defining the necessary KPIs that will be used for the evaluation of each of the smart cities in a time period, (including an adequate period after the end of the project). This is important as the outcome (positive, negative and its quantification) of a smart city project will be made through this specific list of KPIs (project/city-oriented). It is also significant to describe how the monitoring activities of the solutions will be done, with the use of the selected KPIs and describe the full evaluation approach of the project. In addition to that, some representative KPIs (e.g., those belonging to the economic dimension) can be consolidated to be used for the evaluation of the smart city project itself. To do that, one needs to think of the project structure. Following a top-bottom approach, this starts from the project level, continues with the city level first and the Transition Tracks subsequently, ending to solutions level (Figure 2). Thus, we suggest to start the evaluation following the exactly opposite pathway, in the sense that if we evaluate each solution, then we can evaluate each of the Transition Tracks and then, through them, each smart city and end with an evaluation of the project under examination compared to other similar on-going EC projects.

2.8. Scalability and Replicability Analysis

An additional issue that needs to be taken into account while developing the KPIs repository is that the wide-scale rollout of smart city solutions’ needs to comply with two key requirements; scalability and replicability. Scalability refers to the possibility of increasing the geographical area of a project–piloted technologies without compromising its efficiency and effectiveness, whereas replicability refers to the possibility of applying the same solution/technology to achieve the same objective in a different city/geographical area [36]. Actually, many smart city projects fade out after the pilot stage and/or when the project subsidy dries up, failing to generate scalable solutions [37]. Although the literature on smart cities is growing, much less progress is observed regarding how we can achieve the up-scaling of smart city solutions. Currently, most studies available in literature put an emphasis on the analysis of the scalability and replicability of smart grid implementations. Some characteristic studies are those of Calvo et al. [38] and Sigrist et al. [39].

KPIs have been identified as a required tool for scalability and replicability analysis (SRA) [40]. While there is not a universal way to address smart cities solutions’ scalability and replicability, the main elements impacting them have been identified by experts and researchers [41] and are summarized in

Table 4. Indicators relevant to replicability and scalability in SCC projects [41].

Domain	Scalability	Replicability
Technology	Modularity, maturity of technology, network support, interface	Standardization of the technology, maturity of the technology, interoperability, network support
Socio-cultural	Social compatibility/consent, Interaction	Social compatibility/acceptance, market demand/response to citizenry needs, IT literacy level
Political–Institutional	Regulatory environment, institutional support, ecosystem	Need to change in rules and regulations, Regulatory environment, institutional support, ecosystem
Economic/Business	Possibility to achieve economies of scale, profitability	Macro-economic factors, business model, market design

.

The elements mentioned in Table 4 have to be reflected on as much as possible during the development of the KPIs repository. Smart city projects usually put together a set of different solutions with a view to enable the wide scale deployment. This increases the complexity but also favours modularity (i.e., potentially only part of the project can be replicated). Including integrated solutions consisting of mostly well-established and standardized technologies can help minimize technical and financial risks.

Socio-cultural issues (e.g., differences across regions and countries) may reduce replicability potential and, thus, social KPIs need to be carefully selected to ensure that citizens act as co-creators of the solutions. As already mentioned, the proposed framework follows a citizen-centric approach that inherently requires strong participation from the local residents which can limit the scalability to larger environments. The selection of KPIs that can measure the effectiveness of the dissemination strategy for citizen engagement can lay down a solid basis for its potential scalability.

Since smart city projects respond to a pressing need (low carbon economy and climate change adaptation), political commitment although needed, is pretty much ensured as can be reflected by the EU, national and local strategies that put energy transition and climate change mitigation/adaptation at the forefront of sustainable development (i.e., more than 7700 municipalities are participating at the Covenant of Mayors Initiative). The regulatory environment is more and more favouring solutions introduced by smart city projects (e.g., the new EPBD promoting e-mobility and deep energy renovation of buildings) but major constraints (e.g., possibility to sell energy in some countries) are still there. Knowledge of the regulatory landscape in which the project is positioned is an important factor for success. Adding the legal dimension (and relevant KPIs) in the proposed framework is of significant added value since it enables the evaluation of the regulatory environment, a critical element for both scalability and replicability.

3. Implementation

The implementation of the proposed framework leads to the development of a repository of KPIs categorized per dimension with the corresponding levels of assessment and stakeholders group of interest. The technical, environmental, economic, social, ICT and legal KPIs are presented in the

Table 5. Repository of technical KPIs.

Name of KPI	Definition/Description	KPI Formula	Unit	Level of Assessment 1	Relevant Stakeholders 2	Indicative KPI Threshold/Targets 3
1.1: Degree of energetic self-supply by RES	The ratio of locally produced energy from RES and the energy consumption over a period of time (e.g., month, year). DE is separately determined for thermal (heating or cooling) energy and electricity.	$D{E}_T=\frac{LP{E}_T}{T{E}_C}D{E}_E=\frac{LP{E}_E}{E{E}_C}$ DET/E = Degree of thermal/electrical energy self-supply based on RES LPET/E = Locally produced thermal/electrical energy TEC/EEC = Thermal/electrical energy consumption (monitored)	%	C, D, E, F	I, II, III	EU 2030 target: >27% +CITYXCHANGE [42]: 47.7% STARDUST [43]: 62% (49% on electricity)
1.2: Reduced energy curtailment of RES and DER	Reduction of energy curtailment due to technical/operational problems.	$ReductionofEnI=\frac{En{I}_{baseline}-En{I}_{R\&I}}{En{I}_{baseline}}·100$ EnI = Energy not Injected	%	C, D, E, F	I, III	+CITYXCHANGE: <1%
1.3: Average number of electrical interruptions per customer per year	The total number of customer interruptions (numerator) divided by the total number of customers served (denominator). The result shall be expressed as the average number of electrical interruptions per customer per year.	$N_{EIav}=\frac{N_{CItot}}{N_{Ctot}}$ NEIav = Avg number of electrical interruptions per customer per year NCItot = Total number of customer interruptions NCtot = Total number of customers served	#/year	E, F	I, II	<1.5 interruptions/year [44]
1.4: Average length of electrical interruptions (in hours)	The sum of the duration of all customer interruptions in hours (numerator) divided by the total number of customer interruptions (denominator). The result shall be expressed as the average length of electrical interruptions in hours.	$D_{EIav}=\frac{D_{CItot}}{N_{CItot}}$ DEIav = Average length of electrical interruptions in hours DCItot = Sum of the duration of all customer interruptions in hours NCItot = Total number of customer interruptions	hours	E, F	I, II	<2.5 h per customer per year [44]
1.5: Energy Demand and Consumption	The energy entering the system in order to keep operation parameters (e.g., comfort levels). The energy demand is based on the calculated figures and the energy consumption is based on the monitored data. This indicator can be used to assess the energy efficiency of a system.	$E_{d/C}=\frac{T{E}_{d/C}+E{E}_{d/C}}{A_b}$ Ed/C = Energy demand/consumption (simulated/monitored) TEd/C = Thermal energy demand/consumption (simulated/monitored) EEd/C = Electrical energy demand/consumption (simulated/monitored) Ab = Floor area of the building [m2]	kWh/(m2∙month) kWh/(m2∙year)	ALL	I, III, IV	<158.76 kWh/m2∙year (observed consumption of EU28 residential stock in 2014) [45]
1.6: Energy Savings	The reduction of the energy consumption to reach the same services (e.g., comfort levels) after the interventions, taking into consideration the energy consumption from a reference period.	$E{S}_{T/E}=1-\frac{T{E}_{C/E}}{E{R}_{T/E}}$ EST/E = Thermal/Electric energy savings TEC/E = Thermal/Electric energy consumption of the demonstration-site ERT/E = Thermal/Electric energy reference demand or consumption (simulated or monitored) of demonstration-site	%	ALL	I, II, III	EU 2030 target: >27% GROWSMARTER [46]: 60% REPLICATE [47]: 56% STARDUST: 58%
1.7: Smart Storage Capacity	Includes all the energy storage technologies integrated in the city smart grid containing electricity, heating and mobility. This KPI presents the impact in the use of smart energy storage systems.	$Storagecapacityinstalled=\frac{SC{I}_{R\&I}-SC{I}_{baseline}}{SC{I}_{baseline}}·100$	%	D, E, F	I, III	Highly depends on applied solutions
1.8: Battery Degradation Rate	The capacity losses of the batteries used in project, through use (some cycles) and through time (some years). The conclusions of this KPI concern the effectiveness of this technology, the need for maintenance and, thus, gives useful data concerning the financial feasibility of its integration.	$BD{R}_{c/Y}=\frac{B{C}_{n/Y}-B{C}_0}{n/Y·B{C}_0}·100$ BDRC = BDR per cycle BDRY = BDR per year BC0 = initial battery capacity BCn = battery capacity after n cycles n = number of cycles Y = number of years	%	C, D	I, II, III	Highly depends on applied solutions-technologies
1.9: Storage Energy Losses	The energy losses because of battery storage, including the added voltage transformations. The conclusions of this KPI gives useful data concerning the financial feasibility of its integration.	$SEL=\frac{E_{input}-E_{output}}{E_{input}}·100$ Einput = the energy input in a piece of energy storage equipment Eoutput = the energy output of a piece of energy storage equipment	%	A, B, C	I, II, III	Highly depends on applied solutions-technologies
1.10: Maximum Hourly Deficit (MHDx)	The maximum ratio of the difference between load and on-site renewable energy generation to load for each energy type. It is calculated taking the largest value of those ratios calculated for each hour of the year, for those hours when local renewable supply is smaller than the demand.	If:      ${{\displaystyle \int }}_{t_1}^{t_2}{G}_x\left(t\right)dt<{{\displaystyle \int }}_{t_1}^{t_2}{L}_x\left(t\right)dt$ $MHDx=Max\left[\frac{{{\displaystyle \int }}_{t_1}^{t_2}\left[L_x\left(t\right)-G_x\left(t\right)+S_x\left(t\right)\right]dt}{{{\displaystyle \int }}_{t_1}^{t_2}{L}_x\left(t\right)dt}\right]$	No unit	A, B, E, F	I, II, III	Highly depends on applied solutions-technologies
1.11: Technical Compatibility	An indication of the technical compatibility of the smart city solution, meaning the extent to which the solution fits with current practices, administrative and existing technological standards/infrastructures.		Five-point Likert scale (No unit)	ALL	I, III, IV	Very high: no adjustments are needed; immediate implementation.
1.12: Improved Interoperability	Interoperability is the ability of a system (or product) to work with other systems by providing services to and accepting services from other systems and to use the services so exchanged to enable them to operate effectively together (ISO/TS 37151). The indicator assesses the improvement in interoperability in a qualitative manner.		Five-point Likert scale (No unit)	ALL	I, III	Excellent: the project increases interoperability extensively.
1.13: Energy consumption data aggregated by sector fuel	Energy consumption of the mobility sector. It should be assessed for public transport (before and after) as well as for private vehicles (before and after).	$ReductionofEnC=\frac{En{C}_{baseline}-En{C}_{R\&I}}{En{I}_{baseline}}·100$ EnC: Energy Consumption	GJ	E, F	I, III, IV	Highly depends on city profile. Relevant values for municipalities are available in their SEAP [48]
1.14: Free Floating subscribers	The successful implementation of a free-floating car-sharing system mostly depends on the use of the vehicles, which is highly related to the service subscribers. This indicator assesses the increase in the number of subscribers to the free-floating car-sharing service.		#	A, B, E, F	III, IV	60,000 per city (rough estimation considering that there are currently 3 M subscribers corresponding to 50 cities) [49]
1.15: Yearly km made through the e-car sharing system instead of private conventional cars	The key element of a car-sharing system is the usage of the system, not only in terms of users but in terms of kilometres. This indicator will assess the number of kilometres done using the car-sharing service.		km/year	A, B, E, F	III, IV	9 M km/year (considering KPI 1.16 and an average annual travel per car-sharing vehicle of 15,000 km)
1.16: Number of efficient vehicles deployed in the area	A car-sharing system needs a critical number (mass) of vehicles in order to be useful for the users. This indicator assesses the level of service offered by measuring the number of efficient vehicles in the area.		vehicles/km2	E, F	II, III, IV	>600 vehicles (rough estimation considering that there are currently 30,000 shared cars corresponding to 50 cities) [49]
1.17: Number of EVs charging stations and solar powered V2G charging stations deployed in the area	Charging infrastructure development is critical for the promotion of electromobility and the deployment of EVs. This indicator assesses the level of service with regards to charging capabilities offered by measuring the number of electric vehicles charging stations deployed in the area. Additionally, it measures the number of solar powered V2G stations comparing it with the total number of stations.		stations/km2, %	E, F	ALL	This value highly depends on the country of the smart city [50] REPLICATE: >200 EV units; 228 charging points
1.18: Improved flexibility of service delivery following citizen feedback phases	This KPI measures the improved flexibility of service delivery following citizen feedback phase(s).		Five-point Likert scale (No unit)	E, F	III, IV	n/a

¹ Building (A), Set of Buildings (B), Energy Supply Unit (C), Set of Energy Supply Units (D), Neighbourhood (E), City (F); ² Distribution System Operators (I), Consumers (End-Users) (II), Technology and Services Providers (III), Policy-Making Bodies and Governance (IV); ³ Indicative values extracted from similar Smart City Projects and/or relevant targets set by EU and literature.

,

Table 6. Repository of environmental KPIs.

Name of KPI	Definition/Description	KPI Formula	Unit	Level of Assessment 1	Relevant Stakeholders 2	Indicative KPI Threshold/Targets 3
2.1: Carbon dioxide Emission Reduction	CO2 accounts for a major share of Greenhouse Gas emissions in urban areas. The main sources for CO2 emissions are combustion processes related to energy generation and transport. CO2 emissions can therefore be considered a useful indicator to assess the contribution of urban development on climate change.	$m_{C{O}_2}={{\displaystyle \sum }}^{}\left(E_{del,i}{K}_{del,i}\right)-{{\displaystyle \sum }}^{}\left(E_{exp,i}{K}_{exp,i}\right)$ Edel,i = the delivered energy for energy carrier i Eexp,i = the exported energy for energy carrier i Kdel,i = the CO2 coefficient for delivered energy carrier i Kexp,i = the CO2 coefficient for exported energy carrier i	tones/year, %	ALL	ALL	EU 2030 target: >40% +CITYXCHANGE: 12,801 tonnes/year GROWSMARTER: 60% SMARTER TOGETHER [51]: 50–60%
2.2: Increase in Local Renewable Energy Generation	The share of RE production in itself gives an idea of the rate of self-consumption of locally produced energy, which is an indicator of the flexibility potential of the local energy system. The indicator account for the increase of the renewable energy generation due to the intervention.	$LRE\left(H\right)G=\frac{E\left(H\right)RE{S}_{R\&I}-E\left(H\right)RE{S}_{BaU}}{E\left(H\right)C}$ LRE(H)G = Annual Local Renewable Electricity (Heating/Cooling) Generation E(H)RES = Annual electricity (Heating/Cooling) generated by RES E(H)C = Annual Electricity (Heating/Cooling) consumption	%	C, D, E, F	I, IV	EU 2030 Target: 27% STARDUST: 62%
2.3: Increased efficiency of resources consumption	This KPI measures the percentage reduction in material consumption of the project/initiative.	$E_{RC}=\frac{C_{MB}-C_{MF}}{C_{MBF}}·100$ ERC = Percentage reduction in material consumption of the project CMB = Baseline material consumption of the project [t] CMF = Final material consumption of the project [t] CMBF = Baseline final material consumption [t]	% in tonnes	A, B, E, F	II, III, IV	>20% (considering a Factor 5 of improvement to respond adequately to global trends) [52]
2.4: Reduction in annual final energy consumption	The final energy consumption of the project taking into account all forms of energy (e.g., electricity, gas, heat/cold, fuels) and for all functions (transport, buildings, ICT, industry, etc.). The final energy consumption is the energy actually consumed by the end-user.	$RE{C}_{AF}=\frac{E_{Btot}-E_{Atot}}{E_{Btot}}·100$ RECAF = Percentage of the decrease in energy consumption caused by the project EBtot = Total use of energy per year (kWh) on-site or within the project boundaries before the project EAtot = Total use of energy per year (kWh) on-site or within the project boundaries after the project	%	A, B, E, F	ALL	EU 2030 Target: 27% STARDUST: 58%
2.5: Decreased emissions of Particulate matter	This KPI measures the percentage reduction in PM10 and PM2.5 emissions achieved by the project/initiative.	$percentagechangeinPMemissions=\left(\frac{PMem.\left(\frac{kg}{yr}\right)afterproject}{PMem.\left(\frac{kg}{yr}\right)beforeproject}\times 100\right)$	%	E, F	ALL	End-goal to reach 20 and 10 μg/m3 annual mean concentration for PM10 and PM2.5 respectively (World Health Organization [53])
2.6: Decreased emission of nitrogen oxides (NOx)	This KPI measures the percentage reduction in NOx emissions (NO and NO2) achieved by the project/initiative.	$percentagechangeinNOxemissions=\left(\frac{NOxem.\left(\frac{kg}{yr}\right)afterproject}{NOxem.\left(\frac{kg}{yr}\right)beforeproject}\times 100\right)$	%	E, F	ALL	End-goal to reach 40 μg/m3 annual mean concentration for NO2 (World Health Organization (WHO) guidelines)
2.7: Noise pollution	Prolonged exposure to noise can lead to significant health effects. Urban environmental noise pollution relates a lot to noise caused by traffic. This KPI measures the noise levels before and after the activities of the project/initiative.	(dB level before/dB level after) × 100	dB level, %	E, F	ALL	<50 dB (WHO guidelines for residential areas) [54]

¹ Building (A), Set of Buildings (B), Energy Supply Unit (C), Set of Energy Supply Units (D), Neighbourhood (E), City (F); ² Distribution System Operators (I), Consumers (End-Users) (II), Technology and Services Providers (III), Policy-Making Bodies and Governance (IV); ³ Indicative values extracted from similar Smart City Projects and/or relevant targets set by EU and literature.

,

Table 7. Repository of economic KPIs.

Name of KPI	Definition/Description	KPI Formula	Unit	Level of Assessment 1	Relevant Stakeholders 2	Indicative KPI Threshold/Targets 3
3.1: Payback	The payback period is the time it takes to cover investment costs and is usually considered to assess risks. Investments with a short PBP are considered safer than those with a longer one.	Type A static: $EPP=\frac{EP{I}_{BR}}{m}$ m can be calculated as average annual costs in use savings (€/a)       $m=TA{C}_{after}-TA{C}_{before}$ 𝐸𝑃𝐼𝐵𝑅 (€) = Energy-related investment Dynamic Types can be also applied.	years	ALL	I, III	Highly depends on applied solutions (e.g., 5–20 years for PVs [55]) STARDUST: 2–7 years
3.2: Return on Investment (ROI)	ROI enables the evaluation of the feasibility of an investment or the comparison between different possible investments. It is defined as the ratio between the total incomes/net profit and total investment of the project.	$RO{I}_T=\frac{{{\displaystyle \sum }}_{t=1}^T\left(l{n}_t-T{A}_{afte{r}_t}\right)-\left(I_{BR}-I_{ER}\right)}{I_{BR}-I_{ER}}$ ROIT = Return on Investment (%) 𝑇 = Duration of the economic analysis period: T = 10, 15 and 20 [yr]	%	ALL	I, III	Highly depends on applied solutions (e.g., >20% for PVs [56]). Typical values are: 2–5% STARDUST: 9–44%
3.3: Reduction of energy cost	This KPI can assess the economic benefits of a scheduling strategy for prosumers coordinated by an aggregator. The KPI measures the cost of the energy traded by an aggregator, both as a baseline and when ICT are implemented.	$COS{T}_{REDUCTION}=\frac{COS{T}_{R\&I}-COS{T}_{BaU}}{COS{T}_{BaU}}$ COST = the electricity price at a given period of time	%	ALL	I, II, III	In relevance with the total energy savings goals
3.4: Total Investments	An investment is defined as an asset or item that is purchased or implemented with the aim to generate payments or savings over time.	$EP{I}_{ER}=\frac{I_{ER}}{A_d}$ EPIER = Total investment for all interventions related to energy aspects per conditioned area [€/m2] IER = Total investment (€) Ad = Total floor area of the system renovated [m2]	€/m2 or €/kW	ALL	I, II, III	Highly depends on applied solutions
3.5: Total Annual costs	The total annual costs are defined as the sum of capital-related annual costs, requirement-related, operation related costs and other costs. The total annual costs are related to the considered interval of time (year). To make different objects comparable the same types of costs have to be included in the calculation.	$TA{C}_i=C_E+C_{O\&M}+C_F$ 𝑇𝐴𝐶i = Total annual energy cost of the system after the intervention (i.e., energy, operation & maintenance, financial) for year i (€/year) 𝐶𝐸 Total annual cost of the system supply (€/year) 𝐶𝑂&𝑀 Total annual cost of the operation and maintenance of the facility (€/year) 𝐶𝐹 Total annual financing cost, if applies (€/year)	€/year	ALL	I, II, III	Highly depends on applied solutions
3.6: Financial benefit for the end-user	One dimension of value creation by the smart city projects/initiatives is the extent to which they generate cost savings for end-users. Cost savings can be generated, for example, through a reduction in energy use, the generation of renewable energy on site, or reduction in housing costs.	Financial benefit = TotalCostref − TotalCostR and I	€/household/year	A, B, E, F	II, IV	>1700€/household/year (considering energy savings of at least 50%—see KPI 1.5—a median cost per unit of energy for households across the EU28 to be 0.24 €/kWh [57] and annual energy consumption for all end-uses is 14,318 kWh per active dwelling (2014 data) [45]
3.7: Grants	Grants are non-repayable funds that a grant maker, such as the government, provides to a recipient, e.g., a business, for ideas and projects to provide public services and stimulate the economy.	$G_{rER}=\frac{G_{ER}}{I_{ER}}$ GrER Share of the investment in energy retrofitting that is covered by grants (%) GER Total grants received for the energy retrofitting of the district (€) IER Total investment for all the interventions related to energy retrofitting (€)	%	ALL	II, III, IV	Highly depends on applied solutions
3.8: Fuel poverty	The indicator is derived from the UK definition, according to which households are considered as energy poor if their energy bill consumes 10% or more of the household income. This KPI measures the change in percentage points of (gross) household income spent on energy bills.	percentage point change in income spent$=(\frac{Energycostsbeforeproject}{Grosshouseholdincome}·100\%-\frac{Energycostsafterproject}{Grosshouseholdincome}·100\%)$	%	A, B, E, F	ALL	100% (cut off energy bills by half)
3.9: CO2 reduction cost efficiency	Costs in euros per ton of CO2 saved per year. Many smart city projects are intrinsically aimed at reducing the amount of CO2 emitted during their lifetime. Those projects which prove to be able to significantly reduce their carbon footprint, whilst keeping the related costs at a minimum, are considered to be interesting projects for up scaling.	This indicator is calculated on an annual basis, taking the annual reduction in CO2 emissions, and the annual costs of the project (which is the annualised investment plus current expenditures for a year). Note: Only the additional costs for energy/CO2 related measures (to the extent discernible) are taken into account in the total costs’ calculation.	€/(ton of CO2 × year)	A, B, E, F	I, III, IV	Highly depends on applied solutions
3.10: Stimulating an innovative environment	The extent to which the project is part of or stimulates an innovative environment. A project can stimulate an environment that enhances innovations, either by being part of it or by contributing to it.		Five-point Likert scale (No unit)	E, F	III, IV	Excellent: the project is an essential part of and stimulates an innovative environment.
3.11: Awareness of economic benefits of reduced energy consumption	This KPI measures the awareness of economic benefits of reduced energy consumption		%	E, F	II	n/a

¹ Building (A), Set of Buildings (B), Energy Supply Unit (C), Set of Energy Supply Units (D), Neighbourhood (E), City (F); ² Distribution System Operators (I), Consumers (End-Users) (II), Technology and Services Providers (III), Policy-Making Bodies and Governance (IV); ³ Indicative values extracted from similar Smart City Projects and/or relevant targets set by EU and literature.

,

Table 8. Repository of social KPIs.

Name of KPI	Definition/Description	Unit	Level of Assessment 1	Relevant Stakeholders 2	Indicative KPI Threshold/Targets 3
4.1: Consumers’ engagement	The implementation of ICT solutions can also be related to the involvement of the users in the control over the energy use in the building. This KPI includes the number of final users involved and/or the number of people with increased capacity.	#	A, B, E, F	II, III, IV	n/a
4.2: Professional stakeholder involvement	The extent to which professional stakeholders outside the project team have been involved in planning and execution. In this context, relevant stakeholders may include: industry or business associations, local councils, government departments, politicians, environmental organisations, architects, project developers.	Five-point Likert scale (No unit)	E, F	III, IV	High involvement: stakeholders are actively involved on an almost day-to-day basis in developing the project plan and advising on its implementation.
4.3: Social Compatibility	The extent to which the project’s solutions fit with people’s ‘frame of mind’ and do not negatively challenge people’s values or the ways they are used to do things. If an innovation requires people to significantly think differently its implementation in society will be more difficult.	Five-point Likert scale (No unit)	E, F	II, IV	Very high: the solution does not differ from the usual way of doing things in operational sense and is fully consistent with existing norms.
4.4: Ease of use for end users of the solution	The extent to which the solution is perceived as difficult to understand and use for potential end-users. It is presumed that a smart city solution that is easy to use and understand will be more likely adopted than a difficult solution.	Five-point Likert scale (No unit)	E, F	II, III, IV	Very easy: the solution is easy to understand and use.
4.5: Advantages for end-users	The extent to which the project offers clear advantages for end users. The advantage can take many forms, for instance cost savings, improved quality and increased comfort. It is presumed that solutions which have a higher level of advantages to end users will be more likely to be adopted than solutions which have negative or no advantages.	Five-point Likert scale (No unit)	E, F	II, III, IV	Very high advantage: The project offers a very high advantage to end users as the applied technologies have a direct and a positive effect on them.
4.6: Advantages for stakeholders	The extent to which the project offers clear advantages for stakeholders. This advantage could, for example, be ease of management or reduced maintenance costs.	Five-point Likert scale (No unit)	ALL	I, III, IV	Very high advantage: The project offers a very high advantage to stakeholders as the applied technologies have a direct and an extremely positive effect on them.
4.7: People reached	Percentage of people in the target group that have been reached and/or are activated by the project. A Smart City project is usually most successful if the entire target group of a service participates.	%	E, F	II, III, IV	>75% (own estimations)
4.8: Thermal comfort	The quality of the delivered heating/cooling service is certainly a matter of technical aspects that can be measured with quantified technical indicators, but also a matter of the opinion of the service receivers.	Five-point Likert scale (No unit)	A, B, E	II, III	Very satisfying thermal comfort
4.9: Increased environmental awareness	Awareness of environmental problems is important for creating support for environmental projects and programs. This indicator assesses the extent to which the project has used opportunities for increasing environmental awareness and educating about sustainability and the environment.	Five-point Likert scale (No unit)	E, F	II, III, IV	Excellent: relevant were taken into account in the project communication; the project utilized every possibility to address this issue.
4.10: Increased consciousness of citizenship	The extent to which the project has contributed in increasing consciousness of citizenship. Consciousness of citizenship is the awareness (consciousness) of one’s community, civic rights and responsibilities and as such contributes to the sense of community.	Five-point Likert scale (No unit)	E, F	II, III, IV	High: increasing civic consciousness was (one of) the main goals of the project and has done substantial effort to enhance it.
4.11: Increased participation of vulnerable groups	Vulnerable and other groups whose opinions or contributions are not reflected well enough in our society (like women, minorities and the disabled), require special attention to be included in the community, thereby enhancing social cohesion and diversity.	Five-point Likert scale (No unit)	E, F	II, III, IV	Excellent: Participation of groups not well represented in society has clearly been improved due to the project.
4.12: Local job creation	One of the pillars of the smart city projects is to improve the economy by reducing costs and energy, but also by fostering the local economy and the local eco-systems. This indicator assesses the creation of direct jobs from the implementation and operation of the respective solutions.	#	A, B, E, F	II, IV	+CITYXCHANGE: 900 SMARTER TOGETHER: 1400GROWSMARTER: 1500
4.13: Local community involvement in the implementation phase	The extent to which residents/users have been involved in the implementation process.	Five-point Likert scale (No unit)	E, F	II	Community self-development: the project planners have empowered community actors to manage the project implementation and evaluate the results.
4.14: Increased citizen awareness of the potential of smart city projects	Measures the increased citizen awareness of the socio-cultural potential of smart city projects.	Five-point Likert scale (No unit)	E, F	II	n/a
4.15: Number of city officials and urban experts trained to conduct the meaningful and ethical engagement of citizens	Measures the number of city officials and urban experts trained to conduct the meaningful and ethical engagement of citizens.	#	E, F	IV	n/a
4.16: Provision of a localised multi stakeholder co-creation and co-production Field Guide for Citizen Engagement activities	Measures the provision of a localised multi stakeholder co-creation and co-production Field Guide for Citizen Engagement activities (number of co-creation objects added to field guide). This is the direct aim of the Citizen Engagement approach.	#	E, F	II	n/a
4.17: Participation of citizens, citizen representative groups and citizen ambassadors in the co-creation of local/micro KPIs for Citizen Engagement for Smart Cities	Measures the participation (number) of citizens, citizen representative groups and citizen ambassadors in the co-creation of local/micro KPIs for Citizen Engagement for Smart Cities	#	E, F	II	n/a

¹ Building (A), Set of Buildings (B), Energy Supply Unit (C), Set of Energy Supply Units (D), Neighbourhood (E), City (F); ² Distribution System Operators (I), Consumers (End-Users) (II), Technology and Services Providers (III), Policy-Making Bodies and Governance (IV); ³ Indicative values extracted from similar Smart City Projects and/or relevant targets set by EU and literature.

,

Table 9. Repository of ICT KPIs.

Name of KPI	Definition/Description	KPI Formula	Unit	Level of Assessment 1	Relevant Stakeholders 2	Indicative KPI Threshold/Targets 3
5.1: Peak load reduction	Peak load is the maximum power consumption of a building or a group of buildings to provide certain comfort levels. With the correct application of ICT systems, the peak load can be reduced on a high extent and therefore the dimension of the supply system.	$P{L}_{REDUCTION}=\left(1-\frac{P_{peak,R\&I}}{P_{BaU}}\right)\ast 100$	%	A, B, E, F	I, III	n/a
5.2: Number of customers that are positive about how energy systems are controlled	All the end-users involved in the demonstrations are asked whether they are satisfied with the provided services including the ICT systems. This is done with a yes/no question and the value of the indicator is given by the percentage of satisfied end-users.		%	A, B, E, F	II, III, IV	n/a
5.3: Reliability	With the application of ICT measures it is possible to correct a potential misbehaviour of the system and avoid unexpected stops.	$Reliability=\frac{Numberoffailuresavoided}{Totalnumberoffailures}+numberoffailuresavoided·100\%$	%	C, D, E, F	II, III	n/a
5.4: Increased system flexibility for energy players	The ability of the system to respond to—and stabilize/balance—supply and demand in real-time, as a measure of the demand side participation in energy markets-energy efficiency intervention.	$\mathsf{\Delta }SF=\frac{S{F}_{R\&I}-S{F}_{BAU}}{P_{peak}}$ 𝑆𝐹 is the amount of load capacity participating in demand side management.	%, W/€	E, F	I, III	n/a
5.5: Increased hosting capacity for RES, EVs and other new loads	This KPI is intended to give a statement about the additional loads that can be installed in the network, when R and I solutions are applied, and compared to the BAU scenario.	$EH{C}_{\%}=\frac{H{C}_{R\&I}-H{C}_{BAU}}{H{C}_{BAU}}·100\%$ EHC: the enhanced hosting capacity of new loads when R and I solutions are applied with respect to BAU scenario. HC: the additional hosting capacity of new loads applied with respect to currently connected generation (GW or MW).	%	E, F	I, II, III	n/a
5.6: Impact of ICT apps into mobility	Impact of ICT apps into switching from non-sustainable mobility into sustainable mobility, this is, change on modal split.		%	E, F	III, IV	n/a
5.7: Developer engagement	Measures the use of open datasets by developers (Number of API calls per month). Developers are important stakeholders in the open data market. It is important to gain insight in the variety, importance and value of data used and not used by the developers.		#	F	III	n/a
5.8: Data safety	Number of blocked malicious hacking attempts. From DDoS to someone taking control of the servers, the risks are diverse.		#	F	II, III, IV	n/a
5.9: Data loss prevention	Lost data points in a period. Managing data brings a lot of opportunities but also some safety issues. To know if data has been stolen, leaked or otherwise distributed it is important that monitoring is in place.		#	F	II, III, IV	n/a
5.10: Usage of open source software	The use of open source software means less possibilities of vendor lock-in and more space for communities to develop together smart city solutions. It also lowers the software costs.		Five-point Likert scale (No unit)	F	III, IV	n/a
5.11: Expiration date of open data	Percentage of out-dated datasets on a city platform per timeframe. Open data can become out-dated and obsolete, which acts negatively on the attractiveness of using data from platforms. By monitoring the expiration dates of the data, the owner gets a message to renew or remove the datasets.		%	F	III, IV	n/a
5.12: Quality of open data	Percentage of data that uses DCAT standards. The quality of open data is better if standardized. Processes get easier when data standards are applied. The DCAT standard allows municipal employees to produce data in a standardized way.		%	F	III, IV	n/a
5.13: Platform downtime	Downtime per timeframe. To run a stable platform, monitoring is required to fix bugs and quickly improve the software environments.		Minutes/(h, d, w, m)	F	III, IV	n/a
5.14: Open data based solutions	Number of services based on open data. To gain insight of the use of open data, mapping the applications developed based on the open data is vital.		#/(month, year)	F	II, III, IV	n/a
5.15: Number of active ‘touch points’ identified	Measures the number of active ‘touch-points’ identified where citizens have a degree of interaction with solution. This is the basis for distinguishing communication and CE activities and for prioritising and mapping suitable activities to each IS.		#	E, F	II, IV	n/a

¹ Building (A), Set of Buildings (B), Energy Supply Unit (C), Set of Energy Supply Units (D), Neighbourhood (E), City (F); ² Distribution System Operators (I), Consumers (End-Users) (II), Technology and Services Providers (III), Policy-Making Bodies and Governance (IV); ³ In the case of ICT KPIs, relevant baseline and/or BaU values that could potentially be used as threshold values could not be extracted from literature. Available targets highly depend on the applied solutions. Future research to address this gap is thus of great significance.

and

Table 10. Repository of legal KPIs.

Name of KPI	Definition/Description	Unit	Level of Assessment 1	Relevant Stakeholders 2	Indicative KPI Threshold/Targets 3
6.1: Green Building self-consumption Legal Framework Compatibility	The level of suitability of the legal framework for the integration of self-consumption RES generation solutions in buildings.	Five-point Likert scale (No unit)	F	ALL	Full legal support: The legal framework fully approves the integration of the proposed solution.
6.2: Symbiotic waste heat Legal Framework Compatibility	The level of suitability of the legal framework for the integration of symbiotic waste heat solutions.	Five-point Likert scale (No unit)	F	ALL	Full legal support (see above)
6.3: Energy flexibility policies Legal Framework Compatibility	The level of suitability of the legal framework for the integration of energy flexibility policies such as incentives for peak-shaving.	Five-point Likert scale (No unit)	F	ALL	Full legal support (see above)
6.4: Smart EVs Legal Framework Compatibility	The level of suitability of the legal framework for the integration of private EVs and public transport EVs in the city mobility policies.	Five-point Likert scale (No unit)	F	ALL	Full legal support (see above)
6.5: City platform Legal Framework Compatibility	The level of suitability of the legal framework for the integration of a web city platform for the energy management and citizen engagement. This takes into account not only whether the platform is permitted, but also what measurements are taken in order to maintain system security and privacy.	Five-point Likert scale (No unit)	F	ALL	Full legal support (see above)
6.6: Change in rules and regulations	It shows the extent to which the project is able to change the context in which they were applied, by providing a different interpretation of existing rules and regulations. The change in local rules has an important signalling function which can inspire a new interpretation of the rules in other locations.	Five-point Likert scale (No unit)	F	ALL	High impact: Project has led to a public discussion, leading to a change in rules and regulations.
6.7: Measure extent to which privacy by design is ensured	Measures the extent (number of measures) to which privacy by design has been ensured. Trust is paramount to the adoption of smart city solutions, which must fully respect individual freedom and the right to privacy.	#	F	III, IV	n/a

¹ Building (A), Set of Buildings (B), Energy Supply Unit (C), Set of Energy Supply Units (D), Neighbourhood (E), City (F); ² Distribution System Operators (I), Consumers (End-Users) (II), Technology and Services Providers (III), Policy-Making Bodies and Governance (IV); ³ Indicative values extracted from similar Smart City Projects and/or relevant targets set by EU and literature.

, respectively. The repository consists of 75 KPIs falling under the six dimensions. It should be noted that the specific repository is an indicative example based on the evaluation that has been decided to be followed in the framework of the IRIS project, as a first approach-subject though to continuous updates. The repository can be adapted according to the specific needs of the project that is implemented. However, most of the KPIs proposed are setting a solid basis for the evaluation of the majority of smart city and energy efficiency related projects. Based on a screening statistical analysis of the information presented in Table 5, Table 6, Table 7, Table 8, Table 9 and Table 10, the following key observations can be made:

• Most of the KPIs fall under the technical (18 KPIs), social (17 KPIs) and ICT (15 KPIs) dimensions, which is to be expected, considering the variety and special characteristics of each technological solution included in smart city projects and the need for smart cities to be inclusive.
• The significant majority (72 out of 75) of KPIs can be assessed on a city level. The end-goal of smart city projects is to make an impact on a city level, thus this result is fully justified. Especially regarding environmental, economic, ICT and legal dimensions all KPIs are also addressed on a city level. On the contrary, only 18 KPIs are assessed on an ESU level, while most of them (eight KPIs) are falling under the technical dimension.
• The major group of stakeholders identified that are interested in the implementation of the proposed KPIs (61 out of 75 KPIs) corresponds to the TSPs. This further confirms the crucial role of private sector and prosumers in the successful implementation of smart city solutions. DSOs (37 out of 75 KPIs) are mostly interested in the technical, economic and legal dimensions, whereas consumers, including citizens (49 out of 75 KPIs) and policy-making bodies and governance (49 out of 75 KPIs) are paying significant emphasis on environmental, social and legal dimensions.
• A very high interrelation is observed between the building and set of buildings and ESU and set of ESU levels of assessment, since the same KPIs can be used for assessing both levels.

To further enhance the added value of this study, for a number of characteristic KPIs presented below, indicative threshold values and/or targets have been extracted from similar smart city projects and/or relevant targets set by EU and literature. These values provide a first insight of the ambition and requirements set under relevant initiatives. It should be noted though that the values presented may significantly change depending on the level of assessment and the special characteristics of the solutions demonstrated by each project.

4. Conclusions and Further Considerations

This study presents a holistic performance framework allowing the evaluation of the specific technical characteristics of smart city projects, their impact on the social and environmental surroundings, their feasibility from an economic point of view, their smart automation and interaction through an ICT platform and their availability concerning the legal infrastructure. The framework proposed includes six key steps: (a) Clustering of the technology/service solutions into Transition Tracks (five TTs are proposed); (b) definition of the main groups of stakeholders (four main groups are proposed); (c) definition of KPIs dimensions (six dimensions are proposed); (d) definition of KPIs repository per dimension; (e) definition of the scope of evaluation per KPI (six evaluation levels are proposed); and (f) threshold definition per KPI. The specific framework is generic on purpose, in order to satisfy various assessment requirements of a proposed technology solution, however specific recommendations and tips are provided based on its actual implementation and lessons learned from the IRIS smart city project.

The holistic evaluation of smart city solutions includes various, and sometimes competing, interests of the relevant stakeholders (e.g., profit for the market operator vs. cheap services for the consumer). The scalar quantification of solutions through the assessment criteria, being defined by the selected repository of KPIs, enables the comparison on a fair basis among conventional technologies and the application of innovative ones. The implementation of the proposed framework led to a repository of 75 KPIs that can serve as a great basis for similar projects to monitor and evaluate the performance of their solutions.

The process of evaluation using KPIs is of great importance, as it indicates the degree of success of either a research innovation project or even a commercial one. All interested stakeholders can just take a look at the KPI values and acquire a good impression of the progress that is made. In that respect, and to improve and strengthen the impact of the solutions demonstrated, starting from the project-limited boundaries and expanding to the EU level, the evaluation has to be done inductively (the part to whole approach). Such a route approach can also achieve the successful passage from the specific case studies to a more generalized scheme. That is the reason why the evaluations of each case study need to be generalized taking benefit of smaller-scale experience gained by similar to IRIS case studies towards a greater-than-IRIS scale.

In this light, everyone should have in mind to foresee an expanding character in the selected KPIs, so that the most important of them or appropriate consolidations of them into fewer can operate as a general framework for policy and business investment making, on a larger than each community level. A globalized evaluation of solutions, considering the needs primarily of the governance from the side of stakeholders’ perspective along with the inclusion of consolidated globalized KPIs in terms of the six (6) proposed KPI dimensions, should form the basis for a holistic globalized evaluation platform.

Above the solution-level of evaluation, an aggregation of the KPIs should take place in order to reach an evaluation at TT, city and project levels. There is not any specific solid based scientific methodology that can do such a calculation. However, for most of the environmental, social and ICT KPIs this can be done, as most of them are measured in Likert scales and, in the end, what counts is not absolute numbers, but the general feeling of the relevant stakeholders.

Although it is not among the objectives of the present study, the technology evaluation should be able to acquire more global characteristics. For example, the use of EVs as a method of storage and DSM in order to help the increase in RES penetration is firstly used in the specific pilots of each city. The collective experience by all the pilots could give the directions for the integration in a larger scale, such as a whole city. This could give additional experience according to its evaluation and show the way to a wider integration on larger grids, or even to the interconnected system. The final level of generalization is that of the EU who is close to a market grid unification.

This generalised evaluation cannot be done in the close barriers of a single project. IRIS and similar other projects are under observation by the EC since the conclusions can guide to tomorrow’s European policies concerning the state-of-the-art application and the market rules. The use of KPIs from the CITYKeys and SCIS projects facilitate the evaluation at the European level.