A Methodological Framework for the Selection of Key Performance Indicators to Assess Smart City Solutions

: Smart and sustainable cities are expected to form a cornerstone for achieving resource e ﬃ ciency and sustainability worldwide. In this speciﬁc study we introduce a holistic framework for determining a repository of key performance indicators (KPIs) that are able to evaluate both business-as-usual and novel technologies and services related to smart city solutions. The framework includes six steps: (a) Clustering of the technology / service solutions into groups called Transition Tracks; (b) deﬁnition of the main groups of stakeholders; (c) deﬁnition of KPIs dimensions (or domains); (d) deﬁnition of KPIs repository per dimension; (e) deﬁnition of the scope of evaluation per KPI; and (f) threshold deﬁnition per KPI. The implementation of the proposed framework led to the development of a repository of 75 KPIs categorized in six dimensions (technical, environmental, economic, social, ICT and legal KPIs) with the corresponding levels of assessment and stakeholders’ group of interest. The proposed repository can serve as a great basis for similar projects to monitor and evaluate the performance of their solutions. Tips and guidance based on the actual implementation and lessons learned from a smart city project are provided.


The Role of Cities towards Urban Sustainability
Cities are continuously and rapidly growing [1].In 2015, almost 75% of the EU population lived in urban areas and this share is expected to rise to over 80% by 2050 [2].Urban areas account for 60-80% of global energy consumption and about the same share of CO 2 emissions [3].In 2016, nine out of 10 people who lived in cities were exposed to air that did not comply with the World Health Organization air quality guidelines for fine particulate matter (PM 2.5 ) [4].Such detrimental effects can be primarily attributed to the actual energy mixture used in both the building and transport sector and the inefficiency of the building stock: an estimated 97% of the EU's building stock is considered to be energy inefficient, while up to 75-85% of these buildings will continue to be utilized until at least 2050 [5].
Smart and sustainable cities are expected to form a cornerstone for achieving resource efficiency in Europe [6].They can potentially deliver significant energy savings and increased resource efficiency, always in harmony with the cultural aesthetics of the urban and natural landscape.Technological and scientific advancements, especially when being integrated, are offering a rich pool of solutions that can help make a city a sustainable place to live.Such for instance, include: self-harvesting energy production and management (e.g., smart RES, district heating, smart metering, intelligent street lighting), smart storage (e.g., heat pumps, vehicle-to-grid (V2G) storage, innovative batteries), smart mobility (e.g., electric vehicles (EV), EV-charging infrastructure, solar V2G car sharing), city information platforms (e.g., smart open data city platform, urban monitoring) and citizen engagement and co-creation approaches and solutions (e.g., apps for visualizing energy consumption).In light of such a fast transitioning environment, the need for strategies that help cities to smartly integrate technological solutions becomes more and more apparent.This creates new sources of revenue for projects, new business models for value capture and new opportunities for investors [7].Accelerating the transition to a low-carbon competitive economy is both an urgent necessity and a great opportunity for cities in Europe [8].
The COP21 Paris Agreement (2015) recognizes the role of cities and calls on them to rapidly reduce greenhouse gas emission and adapt to climate change.The European Energy Union and the Energy and Climate Policy Framework for 2030, established ambitious commitments (a) to reduce greenhouse gas emissions by at least 40% by 2030 (compared to the 1990 levels); (b) increase the share of renewable energy consumed above 27%; and (c) set an energy savings target of 27% by 2030 [9].The EU is also committed to implement the 2030 Agenda for Sustainable Development [10].According to this agenda, by 2030 the EU should provide access to safe, affordable, accessible and sustainable transport systems, enhance inclusive and sustainable urbanization and reduce the adverse per capita environmental impact of cities.As a result, European policymakers are orchestrating efforts to make urban areas more sustainable by encouraging smart city initiatives.
Smart city projects play a vital role in this attempt.The key objective of smart city projects is to improve sustainability of the city and quality of life of its inhabitants by demonstrating solutions that are able to solve urban problems in an efficient way [11].Several smart city projects have been funded by the European Commission (EC) in recent years under the 7th Framework Programme (FP7) and Horizon 2020 Smart Cities and Communities (SCC) with a view to demonstrate integrated commercial-scale solutions with a high market potential that can address this objective.

The Role of KPIs and Smart City Assessment Frameworks
Although the interest for smart city projects and initiatives has been continuously growing, there has been less progress regarding the evaluation and measurement of their outcomes.Effective assessment is significant to prove the value of smart city projects and initiatives and the benefits delivered to city authorities and all city stakeholders [12].The implementation of smart city solutions still faces some serious challenges due to prodigious data processing demands and heterogeneity of connected smart components [13].To support the monitoring of relevant projects and initiatives, Key Performance Indicators (KPIs) can be a universal instrument to evaluate the progress of smart city strategies [14].
KPIs, in general, measure the effectiveness of a project towards the achievement of specific key objectives.The process of selecting KPIs also assists to clarifying the project's measures of success.In general, indicators (and even more so, KPIs) should express as precisely as possible to what extent an aim, a goal or a standard has been reached or even surpassed.Data that is not linked to standards or specific goals, can be used as quantitative background information (e.g., total investments), but is not suited for evaluative purposes.
The need for a uniform monitoring of the energy smartification throughout Europe has led to initiatives promoting the cooperation and exchanging of know-how among European cities.Such initiatives as the Smart Cities Information System [15] (SCIS) and CITYkeys [16] have created platforms of interaction along with a list of KPIs, each for the evaluation of systems and technologies demonstrated in smart city projects.
The SCIS focuses on the development of indicators to measure technical and economic aspects of energy related measures.These are applicable to European funded demonstration projects for SCC, energy efficient buildings and designated projects funded under the calls for energy efficiency.Launched with support from the EC, SCIS encompasses data, experience and stories collected from completed, ongoing and future projects.
Funded by the European Union HORIZON 2020 program, CITYkeys developed and validated, with the aid of cities, KPIs and data collection procedures for the common and transparent monitoring as well as the comparability of smart city solutions across European cities. CITYkeys was built on existing smart city and sustainable city assessment frameworks.The bases of the framework are the traditional sustainability categories of People, Profit and Planet, but the performance measurement framework includes specific smart city KPIs that go beyond the traditional division into these categories and measure the integration level and openness of the technological solutions.
Table 1 presents a number of indicative SCC EC-funded projects including some key characteristics of the assessment framework, they developed to evaluate and monitor performance.Apart from relevant projects, scientific studies are also available to introduce various assessment frameworks that attempt to evaluate smart city performance.Lombardi et al. [22] propose a model that involves the civil society along with universities, industry and government for classifying smart city performance indicators.Hara et al. [23] introduce a set of KPIs for smart cities building upon the Gross Social Feel-Good Index including six layers of indicators, i.e., those of environment, economy, comfort, health, safety and satisfaction.Studies are also available proposing indicator systems to measure the progress of low-carbon developments at the city level [24].Girardi and Temporelli [25] present a methodology called Smartainability that can estimate through quantitative and qualitative KPIs, to what extent smart cities are sustainable due to the deployment of smart technologies.The methodology has been tested only on a district level (Expo Milano 2015 site) so far, and the estimation can be performed before the technologies are deployed.Another interesting study is that of Dall'O' et al. [26] providing a method for assessing the smartness of a city through a set of indicators focusing on small and medium-size cities and communities.The indicators selected are consistent with the ISO 37120 standard and Sustainable Energy Action Plans under the Covenant of Mayors Initiative.Several studies introduce ranking systems to assess and compare the smartness of cities through indexes but involve the high risk of losing information on the complexity of smart cities [27].Li et al. [28] propose a systematic approach, utilizing a bi-index method, to identify stakeholders and KPIs for multi-level (from building to district) energy performance analysis.
It should be noted that various definitions of the term "smart city" are available [27] that are moving beyond the inclusion of ICT aspects also referring to quality of life.This lack of uniformity of smart city definitions can lead to diverse results and poses challenges to the target setting of cities.Indeed, Ahvenniemi et al. [29] analysed 16 sets of city assessment frameworks comprising 958 indicators indicating noticeable differences between smart cities and urban sustainability frameworks especially regarding their tendency to highlight environmental, social and economic aspects.Another interesting conclusion presented in this study is that, although decreasing energy should be an important goal for smart cities, the use of energy related KPIs is limited in the smart city frameworks.

Methodology
The objective of this study is to present a methodological framework for determining a repository of KPIs that are able to evaluate both conventional and novel technologies and services related to smart city solutions.In parallel, this framework needs to be citizen-centric, since the citizens' benefits should be the aim of any type of solutions rendering the city smart and more sustainable.Realizing citizen engagement, co-creation of inclusive information services for citizens are crucial drivers and enablers in the urban energy transition.Therefore, citizens play an important role in the smart city demonstration activities.
The proposed framework is adopted under the EU-funded smart city project IRIS [30].One of the main goals of IRIS is to evaluate and optimize the operation of smart systems, operating on a district level and potentially cover the needs of a whole city, building upon RES-based technologies.RES solutions can feed both thermal/cooling and electrical needs, with interconnected grids, promoting the idea of synergy among the three main energy vectors.However, the current knowledge on how such grids should be designed to operate in a synergetic manner is limited and technological advancements and demonstration activities need to be carried out before the level of technological maturity reaches a readiness level of nine (TRL9) and the technology can be commercialized.Through demonstration activities in three Lighthouse Cities (Utrecht, The Netherlands; Nice, Cote d'Azur, France; and Gothenburg, Sweden), both thermal and electricity grid performance evaluation results will be used to mature the technologies.
The framework for selecting the KPIs has been finalised in collaboration with key partners from the three Lighthouse Cities; thus it reflects the actual experiences and needs of the cities.Our approach differentiates from other similar frameworks for three key reasons: (a) specific emphasis is given on integrating all relevant stakeholder points of view concerning the deployment of smart city solutions; (b) an out-of-the box thinking is adopted through the inclusion of targeted technological and legal KPI dimensions, in addition to the standard (economic, environmental, social, ICT) ones usually applied; (f) threshold definition per KPI.The next steps for aligning the results with monitoring and evaluation planning on a project level are also presented.The specific framework is generic on purpose, in order to satisfy various assessment requirements of a proposed technology solution.A detailed description of the framework is presented in the following sections.

Clustering Solutions in Transition Tracks
Smart city solutions can be clustered under several Transition Tracks (TT) that cover the majority of the issues that need to be addressed, while aiming at the energy transition of urban systems.The term TT is used to group interdisciplinary and complementary integrated solutions (IS) in terms of the need they address under a common umbrella.The assessment of new technology solutions is a very important step towards the further development of smart cities; thus, the approach on this should be as holistic as possible.To do that, allowing a smooth coordination, management and monitoring of various solutions on a city level and to decrease the complexity of that as much as possible, the IRIS project adopts a categorization under five (5) TTs (provided below), thus covering the whole spectrum of solutions according to their orientation.However, such a categorization cannot be limited to this approach, since other types of schemes can be followed.Specifically, the first three TTs enable the transition towards reduced energy demand and increased shares of renewables and e-mobility in the urban energy and mobility systems, aiming at: ICT play a pivotal role as enabler of smart integration, unlocking the synergy potential of divergent energy and mobility solutions and offering new meaningful insights and services thanks to the data generated by the integrated solutions.Given this condition and the fact that cities can act as large-scale demonstrators of integrated solutions, we propose a fourth TT as well namely, the City Innovation Platform.This includes: A common architecture, harmonized data models and a sustainable data governance plan ensure the interoperability and replicability of the solutions, transferring them from city to city.The city data market and the service marketplace manage access to all data and services, with appropriate licenses and flexible pricing models in and across cities and allowing real-time KPI monitoring and benchmarking of smart energy and mobility performances.
However, except for the technical sector, smart city projects should start from people-by focusing on citizen needs, embracing citizen-centric design and their search for an integral quality of life.To this end, we propose one additional TT, which focuses on citizen engagement, named as:

•
TT#5: Citizen Engagement and Co-Creation: This orients to design and demonstration of feedback mechanisms and inclusive services for citizens to achieve that they are intrinsically motivated to (a) save energy; (b) shift their energy consumption to periods with redundant renewables; (c) use electric vehicles; and (d) change the vehicle ownership culture towards a use or common mobility assets culture.These solutions include game-theory based engagement methods and instruments ranging from co-creating infotainment apps, local school campaigns, offering training on the job to students living in the district by partaking in the demo activities, competitive energy games using the home energy management system, energy ambassadors creating local energy communities, to crowd-funding creating a sense of being part of the solution.
It should be mentioned that the proposed categorization is not absolute and was developed with a view to provide a holistic evaluation covering all necessary smart city aspects.Other projects may need to adapt the proposed TT based on their special characteristics and the technologies they aim to demonstrate.

Definition of the Main Groups of Stakeholders
The inclusion of relevant stakeholders' opinion in decision-making, is considered to be of high significance-no one knows better the needs and the other parameters of a problem than the people affected by and affecting it.For smart city projects, a sensible stakeholder categorization can include the following groups, so that most of the stakeholders can be actively participating/represented in the evaluation of the solutions (from a first level) and of the city (to a final level): (a) Distribution System Operators (DSOs), (b) consumers (end-users), (c) technology and services providers and (d) policy-making bodies and governance.Citizens and/or representative citizen groups can also be distinct groups of stakeholders, but in this study we will consider them as part of the consumers group (assuming that all citizens will eventually become consumers of the services provided).The four defined stakeholder groups attempt to represent all the stakeholder points of view concerning the deployment of smart city solutions.

Distribution System Operators (DSOs)
A DSO is responsible for the management and operation of the distribution network of electricity.In addition, depending on the legislation of each country, a DSO or a DNO (Distribution Network Operator) might be responsible for energy consumption requests for reduction.Sometimes, in the competitive electricity market, the distribution of electricity is usually a monopoly controlled by the regulating authorities.It is of high interest for smart city projects to evaluate system performance from the DSO's point of view.The main aim of a DSO is the sustainability, reliability and flexibility of the system, the ability of the distribution grid to reciprocate to the various consumer needs every single moment (industry and domestic-scale), or the ability to modify the load curve via peak shaving techniques.Similar to the DSOs (a term mostly used for electricity distribution), are the distributors of heating/cooling or other types of energy vectors (e.g., natural gas).Since this specific category of distributors are not named with a standard format, we include them as part of the DSOs.When we refer to DSOs we actually imply either the electricity or the heating/cooling providers, dependent on the type of smart city solution.

Consumers (End-Users)
The citizen engagement has an enhanced role in the modern struggle for increased energy efficiency.Many smart city projects try to ensure and promote the active participation of end users-citizens in the market and grid operations; thus, special focus is granted to the evaluation of the end users' performance within the context of the projects.Concerning TT#1 and TT#2, the consumers can be classified as residential and non-residential, if someone wants to examine end-user's role in the grid level in more detail: (a) Residential consumers: Their main interest is the reduction in the energy consumption, as well as in the energy price, with a probable environmental care about the electricity mixture.Residential consumers are willing to renovate their residences with energy solutions that lower the energy bills; and (b) non-residential consumers: Their main interest is grid security and sustainability, as well as the provision of energy for a low price, with a care for a socio-economic improvement concerning the local energy consumption.They include factories, facilities, offices and generally non-residential buildings, municipal or private, with high energy demand, usually in a fixed daily timetable.
Due to the variety of solutions in TT#3, various consumers are identified in the mobility system.The first group of consumers are the citizens, who in the electro-mobility and car-sharing solutions aim at making use of a common pool of electric vehicles for satisfying their mobility needs instead of using their own car.In the other services (e.g., electric buses) the citizens receive the final benefit in the form of less travel time or pollution.The second group of consumers are the public transport operators, which can upgrade their fleet of vehicles to electric ones or have priority at intersections.Concerning TT#4 and TT#5, all citizens of the city should be considered, since they can potentially be end-users of the services provided.

Technology and Service Providers (TSPs)
In this category, the private sector composed of industry, technological companies and service providers, including SMEs, have a crucial role by connecting smart cities eco-system and supporting the provision of the solutions in different ways.ESCOs, aggregators and retailers are interested in monitoring and analysing the behaviour of the end-users, in validating the operational credibility of the technological installations supporting alternative DR schemes, in identifying potential profile deviations, and in evaluating the impact of the benefits generated by the applied policies.Towards this direction, it is essential for smart city projects to evaluate the impact of the different strategies (demand response, storage and EV management) to the different market stakeholders.
Furthermore, the term 'prosumers' refers to agents that both consume and produce energy at local level.The growth of small and medium-sized agents using solar photovoltaic panels, smart meters, vehicle-to-grid electric vehicles, home batteries and other 'smart' devices, induces the increase in flexibility of the electricity networks.As the number of prosumers increases, the electricity sector is likely to undergo significant changes over the coming years, offering possibilities for the greening of the system.Prosumers could be alternatively included in the end-users category.On the other hand, they invest on energy, sometimes even having profit instead of paying for the energy they consume; thus, they tend to behave more like a market operator.The main interest of a TSP is the profit in an energy venture, a fast payback period of the initial capital cost and a large investment lifetime.Various market operators should be asked for their opinion, beginning from the ones that own the largest share in the electricity mixture in each city, to small prosumers.
In TT#1 and TT#4, the TSPs are responsible for executing and supervising the implementation of the solutions.In some cases, their role is also to promote citizen engagement (TT#5) in order to reach the appropriate business models.At the district level there are various types of market operators, such as housing corporations, who have experience in testing combined energy efficient solutions in buildings and companies manufacturing and supplying electrical equipment who deal with the implementation and exploitation of advanced devices and applications.In TT#2, the traditional utility operators and their expected new business roles should be considered.ESCOs and DR aggregators are the responsible parties to manage the technology to perform DR and negotiate on behalf of their customers with the operator for the provided services.In TT#3, the role of the TSPs is to implement, maintain and run the solutions.They are responsible for both the development and the commercial exploitation of the solutions in the market.They range from traffic management providers and vehicle manufacturers (usually large companies) dealing with the priority service and the electric vehicles, respectively, to service providers (usually SMEs) able to provide car-sharing services or dedicated apps.

Policy-Making Bodies and Governance
The current regulators represent an important stakeholder group that needs to be considered, too.They are responsible for a normal and steady operation of the energy market, its gradual privatization, and they provide the basis of the regulatory framework, which is responsible for the determination of the quality standards and the basic rules.A clear and consistent vision for the smart city has not been adopted by legislators or regulators.Even though there is a great discussion about individual technologies, such as renewables or about specific energy issues, little progress about the overall vision for a modernized smart city and grids is detected.
In the TT#1, the policy making and municipal authorities are responsible for providing installations and services towards the implementation of energy efficient solutions with main objective being the socio-economic development of the district and the reduction of emissions.In TT#2, the municipality is partly responsible concerning the citizen engagement regarding the application and success of the policies developed for the increase of the flexibility of the grid.In TT#3, the policy making and governance authorities are responsible for providing mobility services to the citizens and keeping the pollution levels under desired thresholds.Smart city solutions are supporting them in these objectives by reducing emissions in the urban regions (electro-mobility, car-sharing schemes) while providing a communication channel with the citizens (urban pulse) for increasing their awareness and sensibility towards them.Policy-making bodies should also make sure that the vast amount of data generated during the implementation and monitoring of smart city solutions are organized and utilized in such a way that enhances their decision-making capacity (TT#4).In TT#5 the governance should be tested in its ability to get in touch and motivate a considerable number of end-users, mainly domestic and SMEs, in order to get engaged in the proposed solutions.

Definition of KPIs Dimensions
A basic step of the proposed framework lies on the definition of KPIs dimensions, namely technical, environmental, economic, social, ICT and legal.These dimensions are complementing each other to set the holistic performance framework and are defined as:

•
KPIs measuring Technical Performance, such as the energy consumption, the RES generation ratio, the peak load reduction, etc.

•
KPIs measuring Environmental Performance, such as CO 2 emissions reduction.

•
KPIs measuring Economic Performance, such as the average cost of energy consumption, the average estimation of cost savings, etc.

•
KPIs measuring Social Performance, such as the degree of users' satisfaction.

•
KPIs measuring the Performance of ICT, such as people utilizing apps which enable the residents to monitor and analyse their energy consumptions, home energy management systems, etc.

•
KPIs measuring Legal Performance, such as the level of adaptation of electricity/heat integration in the legal framework, legal barriers for usage of biofuels for energy exploitation purposes, etc.
The current proposed dimension categorization is not the only one that can be adopted.There are other relevant frameworks, either close to the one presented (e.g., SCIS) [15], or quite different (e.g., CITYKeys) [16].We propose the one presented as a more holistic option in studies for systems operation characterized by a medium to high TRL.The legal dimension is a new aspect that is presented in this study and many stakeholders demand it nowadays, given the condition that the current EU legislative framework is not uniform, but fragmented across the various EU countries.Table 2 shows an example of the relevance between the specific dimensions' categorization with the main questions that have to be posed for the evaluation of a technology solution (e.g., second-life batteries).

Questions for the Evaluation of the 2nd Life Batteries Implementation Dimension
Do they need maintenance often?Technical Are there any CO 2 savings because of their implementation?Environmental Is their cost per year higher than that of brand-new batteries?Economic Is the idea publicly accepted or are they not trusted?Social Do they correlate well with other components in a smart grid?ICT Is their use accepted by the legal framework?Legal

Technical Dimension
KPIs in the technical dimension measure the effectiveness of a given solution with respect to the operating parameters and technical constraints acting on electricity/thermal grid and active/passive users, as well as the effectiveness of technology solutions concerning heating/cooling, electrification and mobility, on both a building and a district level.Representative technical KPIs can be obtained by gathering the electrical metrics of the network (e.g., voltages/currents collected along feeders, number of e-charging stations and V2G vehicles deployed in the area, proportion of RES integration) and of customers and producers consumption profiles (e.g., active/reactive energy/power exchanged with the network, usage of the car-sharing vehicles, energy consumption for cooling, heating and hot water).In some cases, the KPIs need to be supported by numerical simulations on the basis of a thermal or an electricity grid model, representing the operation of a building or a district and/or possible actual measurements collected during the grid operation.
The interest in these KPIs changes depending on the perspective of the various stakeholders, such as DSOs, who are mainly concerned about KPIs related to the MV/LV network operation, while customers are focused on KPIs assessing the performance of a new approach/strategy at their premises.However, other factors exist that could affect the relevance of the KPIs considered in the different situations, for example the regulatory framework in force, which could promote an improvement of the quality of service with reference to specific technical indices (SAIDI/SAIFI), or business cases applying in each particular scenario, also in relationship with the target performances defined in the economic dimension.

Environmental Dimension
KPIs in the environmental dimension are important for understanding and evaluating the environmental impact of energy/storage, smart grid distribution, heating/cooling and mobility related solutions and are important for a smart system planning and operation.The environmental KPIs can be used to evaluate the efficiency of the solutions demonstrated in the cities from the viewpoint of the expected environmental impact.For example, there are KPIs that refer to the operational phase (noise and pollen pollution exposure), as well as to the end-of-life phase (EROI).The main focus is on operational phase evaluation through the definition of KPIs that set the framework for day-to-day evaluation, while a life cycle analysis (LCA) methodology can be applied for the determination of environmental aspects and potential impacts of a product or system from raw material extraction through production, use and disposal, while evaluating possible recycling routes following a cradle-to-cradle approach (e.g., second-life batteries).

Economic Dimension
The economic performance evaluation takes into account the business efficiency of each application and usage scenario from the market stakeholder perspective.Different demonstrators offer different value propositions to stakeholders and thus, special focus should be delivered to the definition of KPIs that reflect this specific viewpoint.Among the objectives of smart city projects is to provide market-viable solutions, defining business-oriented KPIs to evaluate the day-to-day performance of the applied tools and applications.For example, the residents of apartments would like to have a view of the economic benefit produced by their flexible consumption behaviour.They may be willing to sacrifice part of their comfort to achieve lower energy bills and they would like to know what the cost/benefit ratio is.Likewise, the business stakeholder (DR aggregator) will like to know the actual benefit from the implementation of DR strategies in a portfolio of customers.Concerning the closed-loop energy positive districts, the local communities try to promote and support energy efficient measures and solutions targeting to economic and business development by reducing the electricity bills and engaging consumers to an energy sensitive attitude.With regards to mobility, the city is willing to reduce congestion and pollution as well as parking places, while the consumers are willing to increase the usage of the vehicles (the system operator) and to increase the availability of shared vehicles (the citizens).Once again, the overall business and economic analysis is closely related to the definition of business stakeholders in the project, along with the selection of business models and associated scenarios to be examined at the demonstration sites of each project.

Social Dimension
The social aspects of energy projects were found to be the less popular among the employed KPIs in previous similar studies.The chosen indicators reveal that attitudes towards energy are interrelated with demand response mechanisms [31] and such KPIs can be used to evaluate the extent to which the end-users (citizens in most cases) are willing to participate and be self-motivated for further demonstration and application of the demonstrated solutions.Generally, the social performance domain visualizes the impact of a technology, scheme or policy to social factors like local wealth, unemployment, satisfaction, or even more specific, like the effect on the use of public transport, the health care system, etc.A popular approach used in literature for expressing the social KPIs is the Likert scale, since it is a sensible way for quantifying a qualitative value.Partners responsible for such KPIs should determine target groups among the various stakeholders and pose them a question that needs a Likert answer.

ICT Dimension
The ICT dimension could be concerned as one of the technology pillars of smart city projects.A smart city tends to connect the various energy operations, including generation and consumption, with a central energy management platform that interacts with citizens and generally all stakeholders.ICT is used as a KPI dimension because it indicates the interoperability of the technology solution presented, its ability to correlate with the rest components of the energy grid, its capability for two-way interaction with the citizens.The KPIs listed in the ICT dimension refer to the City Innovation Platform mainly, regarding the monitoring and sensible control of the proposed technology solutions.This dimension is appropriate for smart city projects regarding the implementation of new components in an existing smart grid.

Legal Dimension
KPIs in the Legal dimension mainly monitor the legislative background concerning the application of the proposed solutions.The specific dimension is not commonly used, but it is of great importance in the R and I, since law-making bodies are often not flexible enough to follow the progress of technology, especially when these are related to strongly regulated/protected markets (energy and mobility).This is a serious problem, especially in EU, since most of the already mature technologies cannot be actually implemented and operate in real-life conditions, because there is not the necessary legal background, allowing their actual life operation.Even more important are the economic results.An immediate legislative support of a new technology can give a serious handicap for its developer and end-user in a world-wide market, where the exploitation of innovations is one of the most serious sources of profit.Generally, market operators (including DSOs and prosumers) need a steady legislation concerning their invested capital, and fast response concerning the legislative background of innovations.The legal KPIs mainly evaluate the governance in terms of legislative flexibility.This flexibility is difficult to be objectively quantified, so the subjective point of view of several stakeholders is needed, usually in the form of a percentage scale.

Definition of KPIs Repository
The use of quantitative indicators is valuable not only to describe/assess as accurately as possible individual characteristics of a technology, but also to evaluate them, in a simple and on a fair basis way, against other solutions of the same characteristics serving the same role.Such an approach facilitates the direct comparison of available technologies, designed for the same scope.A filtering procedure according to predefined criteria is necessary to narrow down the vast number of potential indicators that can be included in the repository [32].To achieve having a shortlist of indicators and following bilateral discussions with key partners from the three LH cities, a set of criteria was used in IRIS project and are proposed in this study, using as a basis the CIVITAS framework [33], according to which each set of KPIs should be characterized by:

•
Relevance: Each indicator should have a significant importance for the evaluation process.The indicators should be selected and defined in such a way that the implementation of the smart city project provides a clear signal in the change of the indicator value.Familiarity: The indicators should be easy to understand by the users.

•
Non-redundancy: Indicators within a system/framework should not measure the same aspect of a subtheme.

•
Independence: Small changes in the measurements of an indicator should not influence preferences assigned to other indicators in the evaluation.However, as the current energy systems in many cities are still largely based on fossil fuels, there is a direct relation between a reduction in the use of energy and the reduction of the emission of carbon dioxide.This will lead, to a certain extent, to double-counting the impact.
In the case of the IRIS project the definition of KPIs was conducted in accordance with other projects enhancing the way towards the energy smartification of European cities (Figure 1).The majority of the proposed KPIs were mainly taken by the CITYKeys and SCIS KPI pools.Some KPIs, mostly the Legal ones, did not exist in previous literature.Specifically, the legal KPIs were firstly used in the SMILE project [35].The list was discussed among the demonstrators in order to make the appropriate additions and adjustments according to the project needs.The first and most important criterion for the definition of KPIs by the demonstrators should be if this KPI can actually be measured or calculated.


Measurability: The identified indicators should be capable of being measured, preferably as objectively as possible.For the majority of indicators in the ICT, social and legal dimensions, quantitative measurability is limited.Social sciences provide approaches to deal with qualitative information in a semi-quantitative way [34]. Reliability: The definitions of the indicators should be clear and not open for different interpretations.This holds for the definition itself and for the calculation methods behind the indicator.


Familiarity: The indicators should be easy to understand by the users. Non-redundancy: Indicators within a system/framework should not measure the same aspect of a subtheme. Independence: Small changes in the measurements of an indicator should not influence preferences assigned to other indicators in the evaluation.However, as the current energy systems in many cities are still largely based on fossil fuels, there is a direct relation between a reduction in the use of energy and the reduction of the emission of carbon dioxide.This will lead, to a certain extent, to double-counting the impact.
In the case of the IRIS project the definition of KPIs was conducted in accordance with other projects enhancing the way towards the energy smartification of European cities (Figure 1).The majority of the proposed KPIs were mainly taken by the CITYKeys and SCIS KPI pools.Some KPIs, mostly the Legal ones, did not exist in previous literature.Specifically, the legal KPIs were firstly used in the SMILE project [35].The list was discussed among the demonstrators in order to make the appropriate additions and adjustments according to the project needs.The first and most important criterion for the definition of KPIs by the demonstrators should be if this KPI can actually be measured or calculated.The collection of primary (measurement-based) data by the different demonstrators is crucial for the calculation of the KPIs, as well as its overall evaluation in terms of the different pilots and its replication ability.In most cases, the data is described by its units and the time point/period it refers to.The data source directs to the methodology used for the data collection.Except for the raw measurements associated with the real-time operation of the city information platforms, many additional parameters, not easily measured, will need to be determined for the The collection of primary (measurement-based) data by the different demonstrators is crucial for the calculation of the KPIs, as well as its overall evaluation in terms of the different pilots and its replication ability.In most cases, the data is described by its units and the time point/period it refers to.Except for the raw measurements associated with the real-time operation of the city information platforms, many additional parameters, not easily measured, will need to be determined for the calculation of KPIs.These secondary (model-based) data consist of the configuration parameters and normalization factors that will enable the model-based KPIs calculation.In some cases (e.g., retailer or market prices), dynamically updated values can be considered and, thus, interfaces with external service providers (e.g., energy markets) need to be defined.
The second criterion is the importance of the specific KPI for their opinion as an ecosystem (taking into consideration the opinion of their relevant ecosystem stakeholders).These two parameters are enough for their prioritization, based on our experience (see the Results section for the complete KPI repository).

Definition of the Levels of Evaluation Per KPI
The evaluation of smart city solutions with the utilization of KPIs entails different levels of spatial aggregation which goes from a single building to a whole district or city.Table 3 presents the different levels of aggregation that can be evaluated using the proposed framework.The level of evaluation per KPI needs to be clearly defined to increase the transparency of the results.Table 3. Different levels of evaluation (adapted from SCIS [15]).

Building
This concerns the energy performance balance of: (a) the delivered energy required to meet the energy needs; and (b) the exported energy.The delivered energy is to be expressed per energy carrier.If part of this delivered energy is allocated to energy export, it also needs to be specified in the data collection where the electricity produced is not used in the building.In this case the corresponding amount of gas allocated to electricity production shall be specified in order to be able to calculate the energy performance of the building.
At the building level the data required is (calculation procedure goes from the energy needs to the primary energy): (a) Energy needs per area of application; (b) energy technologies supplying these energy needs; (c) energy storage units; and (d) delivered energy to each energy supply units expressed per energy carrier.

Set of Buildings
The assessment for a set of buildings is done by the aggregation of building units.
The indicators can then be calculated for the sum of the buildings as a group.

Energy Supply Unit (ESU)
At the ESU level, the approach to be followed is similar to the building level.Delivered energy per energy carrier and output energy allocated to energy carrier need to be specified.Additionally, and depending on the energy supply unit, different indicators can be calculated.This evaluation level refers to building integrated energy supply units as well as large-scale energy supply units.

Set of Energy Supply Units
The assessment for a set of ESU is done by the aggregation of energy supply units.
The indicators can then be calculated for the sum of the energy supply units.

Neighbourhood
This level of evaluation (area or neighbourhood) is composed by the aggregation of different entities.The energy flows at this point need to be defined.The following information is required to define the energy system: (a) Energy carriers used at the implementation area level and the primary energy factors corresponding to this area; (b) demonstration units involved (buildings, energy supply units, storage units and distribution systems); (c) delivered energy to each ESU and building allocated to the corresponding energy carrier; (d) output energy of each ESU and, if applicable, output energy exported out of the boundary allocated to the amount of delivered energy carrier; and (e) energy flows between technologies and buildings (which ESU is supplying which building or ESU).Due to the complexity of these systems, indicators can only be calculated if a full set of data is available.

City
Most smart city projects demonstrate solutions in city environment (the approach to be followed is similar to the neighbourhood but on a wider scale).Evaluation at a national level may be needed for the legal performance indicators.

Threshold Definition Per KPI
After the final definition of the KPI repository, the threshold definition is an important and sometimes difficult task, since it sets the quantified objectives of the project.Each KPI will finally acquire a value calculated throughout the monitoring of the project.The actual evaluation of the presented technology solution has to be done with the comparison of the KPI final value with a threshold that separates success to failure.This separation line can have the form of:

•
Baseline: Baseline is a measurement taken in the beginning of the project.If the threshold is the baseline, then the scope is to check the difference in the actual result because of the implementation of the proposed technology solution.

•
Business as Usual (BaU): BaU is a more complex threshold, since it takes into consideration the change in the value of the KPI throughout the time period of the project, without the implementation of the tested technology solution.It takes into account the general tendency of the change in the KPI value.The BaU threshold comprises a more realistic view on the tested technology impact on its environment but is more difficult to be estimated.

•
Other threshold: A threshold value could be defined by the evaluator, without it being either a baseline or a BaU.This could apply to KPIs that have not been estimated in the past, such as the legal KPIs or some social KPIs that are measured with the Likert scale.
Either way, the threshold is defined regarding the necessary literature survey.The demonstrators have the last word in the threshold determination since they are able to take into consideration the most aspects influencing the performance of the tested technology solutions in their city environment.

Align with Monitoring and Evaluation Planning
The proposed framework deals with the selection of the KPIs that can be used for the evaluation and monitoring of the IS being grouped under the TT (see Section 2.1) that will be actually demonstrated during a smart city project.As a next step there is a need to define the list of KPIs, not only for the solution demonstration activities, but to go a step forward towards defining the necessary KPIs that will be used for the evaluation of each of the smart cities in a time period, (including an adequate period after the end of the project).This is important as the outcome (positive, negative and its quantification) of a smart city project will be made through this specific list of KPIs (project/city-oriented).It is also significant to describe how the monitoring activities of the solutions will be done, with the use of the selected KPIs and describe the full evaluation approach of the project.In addition to that, some representative KPIs (e.g., those belonging to the economic dimension) can be consolidated to be used for the evaluation of the smart city project itself.To do that, one needs to think of the project structure.Following a top-bottom approach, this starts from the project level, continues with the city level first and the Transition Tracks subsequently, ending to solutions level (Figure 2).Thus, we suggest to start the evaluation following the exactly opposite pathway, in the sense that if we evaluate each solution, then we can evaluate each of the Transition Tracks and then, through them, each smart city and end with an evaluation of the project under examination compared to other similar on-going EC projects.

Scalability and Replicability Analysis
An additional issue that needs to be taken into account while developing the KPIs repository is that the wide-scale rollout of smart city solutions' needs to comply with two key requirements; scalability and replicability.Scalability refers to the possibility of increasing the geographical area of a project-piloted technologies without compromising its efficiency and effectiveness, whereas replicability refers to the possibility of applying the same solution/technology to achieve the same objective in a different city/geographical area [36].Actually, many smart city projects fade out after the pilot stage and/or when the project subsidy dries up, failing to generate scalable solutions [37].Although the literature on smart cities is growing, much less progress is observed regarding how we can achieve the up-scaling of smart city solutions.Currently, most studies available in literature put an emphasis on the analysis of the scalability and replicability of smart grid implementations.Some characteristic studies are those of Calvo et al. [38] and Sigrist et al. [39].
KPIs have been identified as a required tool for scalability and replicability analysis (SRA) [40].While there is not a universal way to address smart cities solutions' scalability and replicability, the main elements impacting them have been identified by experts and researchers [41] and are summarized in Table 4.
Table 4. Indicators relevant to replicability and scalability in SCC projects [41].

Scalability and Replicability Analysis
An additional issue that needs to be taken into account while developing the KPIs repository is that the wide-scale rollout of smart city solutions' needs to comply with two key requirements; scalability and replicability.Scalability refers to the possibility of increasing the geographical area of a project-piloted technologies without compromising its efficiency and effectiveness, whereas replicability refers to the possibility of applying the same solution/technology to achieve the same objective in a different city/geographical area [36].Actually, many smart city projects fade out after the pilot stage and/or when the project subsidy dries up, failing to generate scalable solutions [37].Although the literature on smart cities is growing, much less progress is observed regarding how we can achieve the up-scaling of smart city solutions.Currently, most studies available in literature put an emphasis on the analysis of the scalability and replicability of smart grid implementations.Some characteristic studies are those of Calvo et al. [38] and Sigrist et al. [39].
KPIs have been identified as a required tool for scalability and replicability analysis (SRA) [40].While there is not a universal way to address smart cities solutions' scalability and replicability, the main elements impacting them have been identified by experts and researchers [41] and are summarized in Table 4.The elements mentioned in Table 4 have to be reflected on as much as possible during the development of the KPIs repository.Smart city projects usually put together a set of different solutions with a view to enable the wide scale deployment.This increases the complexity but also favours modularity (i.e., potentially only part of the project can be replicated).Including integrated solutions consisting of mostly well-established and standardized technologies can help minimize technical and financial risks.
Socio-cultural issues (e.g., differences across regions and countries) may reduce replicability potential and, thus, social KPIs need to be carefully selected to ensure that citizens act as co-creators of the solutions.As already mentioned, the proposed framework follows a citizen-centric approach that inherently requires strong participation from the local residents which can limit the scalability to larger environments.The selection of KPIs that can measure the effectiveness of the dissemination strategy for citizen engagement can lay down a solid basis for its potential scalability.
Since smart city projects respond to a pressing need (low carbon economy and climate change adaptation), political commitment although needed, is pretty much ensured as can be reflected by the EU, national and local strategies that put energy transition and climate change mitigation/adaptation at the forefront of sustainable development (i.e., more than 7700 municipalities are participating at the Covenant of Mayors Initiative).The regulatory environment is more and more favouring solutions introduced by smart city projects (e.g., the new EPBD promoting e-mobility and deep energy renovation of buildings) but major constraints (e.g., possibility to sell energy in some countries) are still there.Knowledge of the regulatory landscape in which the project is positioned is an important factor for success.Adding the legal dimension (and relevant KPIs) in the proposed framework is of significant added value since it enables the evaluation of the regulatory environment, a critical element for both scalability and replicability.

Implementation
The implementation of the proposed framework leads to the development of a repository of KPIs categorized per dimension with the corresponding levels of assessment and stakeholders group of interest.The technical, environmental, economic, social, ICT and legal KPIs are presented in the Tables 5-10, respectively.The repository consists of 75 KPIs falling under the six dimensions.It should be noted that the specific repository is an indicative example based on the evaluation that has been decided to be followed in the framework of the IRIS project, as a first approach-subject though to continuous updates.The repository can be adapted according to the specific needs of the project that is implemented.However, most of the KPIs proposed are setting a solid basis for the evaluation of the majority of smart city and energy efficiency related projects.Based on a screening statistical analysis of the information presented in Tables 5-10, the following key observations can be made:

•
Most of the KPIs fall under the technical (18 KPIs), social (17 KPIs) and ICT (15 KPIs) dimensions, which is to be expected, considering the variety and special characteristics of each technological solution included in smart city projects and the need for smart cities to be inclusive.

•
The significant majority (72 out of 75) of KPIs can be assessed on a city level.The end-goal of smart city projects is to make an impact on a city level, thus this result is fully justified.Especially regarding environmental, economic, ICT and legal dimensions all KPIs are also addressed on a city level.On the contrary, only 18 KPIs are assessed on an ESU level, while most of them (eight KPIs) are falling under the technical dimension.

•
The major group of stakeholders identified that are interested in the implementation of the proposed KPIs (61 out of 75 KPIs) corresponds to the TSPs.This further confirms the crucial role of private sector and prosumers in the successful implementation of smart city solutions.DSOs (37 out of 75 KPIs) are mostly interested in the technical, economic and legal dimensions, whereas consumers, including citizens (49 out of 75 KPIs) and policy-making bodies and governance (49 out of 75 KPIs) are paying significant emphasis on environmental, social and legal dimensions.

•
A very high interrelation is observed between the building and set of buildings and ESU and set of ESU levels of assessment, since the same KPIs can be used for assessing both levels.Indicative KPI Threshold/Targets 3

1.5: Energy Demand and Consumption
The energy entering the system in order to keep operation parameters (e.g., comfort levels).The energy demand is based on the calculated figures and the energy consumption is based on the monitored data.This indicator can be used to assess the energy efficiency of a system.If: Highly depends on applied solutions-technologies  [48] 1.14: Free Floating subscribers The successful implementation of a free-floating car-sharing system mostly depends on the use of the vehicles, which is highly related to the service subscribers.This indicator assesses the increase in the number of subscribers to the free-floating car-sharing service.
# A, B, E, F III, IV 60,000 per city (rough estimation considering that there are currently 3 M subscribers corresponding to 50 cities) [49] 1.15: Yearly km made through the e-car sharing system instead of private conventional cars The key element of a car-sharing system is the usage of the system, not only in terms of users but in terms of kilometres.This indicator will assess the number of kilometres done using the car-sharing service.Indicative KPI Threshold/Targets 3

2.4: Reduction in annual final energy consumption
The final energy consumption of the project taking into account all forms of energy (e.g., electricity, gas, heat/cold, fuels) and for all functions (transport, buildings, ICT, industry, etc.).The final energy consumption is the energy actually consumed by the end-user.Indicative KPI Threshold/Targets 3

3.1: Payback
The payback period is the time it takes to cover investment costs and is usually considered to assess risks.Investments with a short PBP are considered safer than those with a longer one.Highly depends on applied solutions (e.g., >20% for PVs [56]).Typical values are: 2-5% STARDUST: 9-44% Indicative KPI Threshold/Targets 3

3.5: Total Annual costs
The total annual costs are defined as the sum of capital-related annual costs, requirement-related, operation related costs and other costs.The total annual costs are related to the considered interval of time (year).To make different objects comparable the same types of costs have to be included in the calculation.

4.3: Social Compatibility
The extent to which the project's solutions fit with people's 'frame of mind' and do not negatively challenge people's values or the ways they are used to do things.If an innovation requires people to significantly think differently its implementation in society will be more difficult.
Five-point Likert scale (No unit) E, F II, IV Very high: the solution does not differ from the usual way of doing things in operational sense and is fully consistent with existing norms.
4.4: Ease of use for end users of the solution The extent to which the solution is perceived as difficult to understand and use for potential end-users.It is presumed that a smart city solution that is easy to use and understand will be more likely adopted than a difficult solution.
Five-point Likert scale (No unit) E, F II, III, IV Very easy: the solution is easy to understand and use.

4.5: Advantages for end-users
The extent to which the project offers clear advantages for end users.The advantage can take many forms, for instance cost savings, improved quality and increased comfort.It is presumed that solutions which have a higher level of advantages to end users will be more likely to be adopted than solutions which have negative or no advantages.Indicative KPI Threshold/Targets 3

4.6: Advantages for stakeholders
The extent to which the project offers clear advantages for stakeholders.This advantage could, for example, be ease of management or reduced maintenance costs.Indicative KPI Threshold/Targets 3

5.1: Peak load reduction
Peak load is the maximum power consumption of a building or a group of buildings to provide certain comfort levels.With the correct application of ICT systems, the peak load can be reduced on a high extent and therefore the dimension of the supply system.The level of suitability of the legal framework for the integration of a web city platform for the energy management and citizen engagement.This takes into account not only whether the platform is permitted, but also what measurements are taken in order to maintain system security and privacy.To further enhance the added value of this study, for a number of characteristic KPIs presented below, indicative threshold values and/or targets have been extracted from similar smart city projects and/or relevant targets set by EU and literature.These values provide a first insight of the ambition and requirements set under relevant initiatives.It should be noted though that the values presented may significantly change depending on the level of assessment and the special characteristics of the solutions demonstrated by each project.

Conclusions and Further Considerations
This study presents a holistic performance framework allowing the evaluation of the specific technical characteristics of smart city projects, their impact on the social and environmental surroundings, their feasibility from an economic point of view, their smart automation and interaction through an ICT platform and their availability concerning the legal infrastructure.The holistic evaluation of smart city solutions includes various, and sometimes competing, interests of the relevant stakeholders (e.g., profit for the market operator vs. cheap services for the consumer).The scalar quantification of solutions through the assessment criteria, being defined by the selected repository of KPIs, enables the comparison on a fair basis among conventional technologies and the application of innovative ones.The implementation of the proposed framework led to a repository of 75 KPIs that can serve as a great basis for similar projects to monitor and evaluate the performance of their solutions.
The process of evaluation using KPIs is of great importance, as it indicates the degree of success of either a research innovation project or even a commercial one.All interested stakeholders can just take a look at the KPI values and acquire a good impression of the progress that is made.In that respect, and to improve and strengthen the impact of the solutions demonstrated, starting from the project-limited boundaries and expanding to the EU level, the evaluation has to be done inductively (the part to whole approach).Such a route approach can also achieve the successful passage from the specific case studies to a more generalized scheme.That is the reason why the evaluations of each case study need to be generalized taking benefit of smaller-scale experience gained by similar to IRIS case studies towards a greater-than-IRIS scale.
In this light, everyone should have in mind to foresee an expanding character in the selected KPIs, so that the most important of them or appropriate consolidations of them into fewer can operate as a general framework for policy and business investment making, on a larger than each community level.A globalized evaluation of solutions, considering the needs primarily of the governance from the side of stakeholders' perspective along with the inclusion of consolidated globalized KPIs in terms of the six (6) proposed KPI dimensions, should form the basis for a holistic globalized evaluation platform.
Above the solution-level of evaluation, an aggregation of the KPIs should take place in order to reach an evaluation at TT, city and project levels.There is not any specific solid based scientific methodology that can do such a calculation.However, for most of the environmental, social and ICT KPIs this can be done, as most of them are measured in Likert scales and, in the end, what counts is not absolute numbers, but the general feeling of the relevant stakeholders.
Although it is not among the objectives of the present study, the technology evaluation should be able to acquire more global characteristics.For example, the use of EVs as a method of storage and DSM in order to help the increase in RES penetration is firstly used in the specific pilots of each city.
and (c) a clear definition of the level of evaluation per KPI is pre-determined.The definition of KPIs is conducted in accordance with other European projects.The framework proposed includes the following six steps: (a) Clustering of the technology/service solutions into groups called Transition Tracks; (b) definition of the main groups of stakeholders; (c) definition of KPIs dimensions (or domains); (d) definition of KPIs repository per dimension; (e) definition of the scope of evaluation per KPI; and

Figure 1 .
Figure 1.The proposed framework for KPIs definition.
The most common cases in smart city projects are: (a) existing web services; (b) smart meters; (c) plug-level meters; (d) utility bills; (e) battery management systems and EV charging platforms; (f) grid power quality analyser; and (g) supervisory control and data acquisition (SCADA).

Figure 1 .
Figure 1.The proposed framework for KPIs definition.
The data source directs to the methodology used for the data collection.The most common cases in smart city projects are: (a) existing web services; (b) smart meters; (c) plug-level meters; (d) utility bills; (e) battery management systems and EV charging platforms; (f) grid power quality analyser; and (g) supervisory control and data acquisition (SCADA).

Figure 2 .
Figure 2. The bottom-top KPI list aggregation of the IRIS evaluation framework.

Figure 2 .
Figure 2. The bottom-top KPI list aggregation of the IRIS evaluation framework.
Type A static: EPP = EPIBR m m can be calculated as average annual costs in use savings (€/a) m = TAC a f ter − TAC be f ore EPIBR (€) = Energy-related investment Dynamic Types can be also applied.evaluation of the feasibility of an investment or the comparison between different possible investments.It is defined as the ratio between the total incomes/net profit and total investment of the project.ROI T = T t=1 (lnt−TAafter t )−(IBR−IER) IBR−IER ROI T = Return on Investment (%) T = Duration of the economic analysis period: T = 10, 15 and 20 [yr]

Five
project offers a very high advantage to end users as the applied technologies have a direct and a positive effect on them.

Five
Change in rules and regulations It shows the extent to which the project is able to change the context in which they were applied, by providing a different interpretation of existing rules and regulations.The change in local rules has an important signalling function which can inspire a new interpretation of the rules in other locations.Five-point Likert scale (No unit) F ALL High impact: Project has led to a public discussion, leading to a change in rules and regulations.6.7: Measure extent to which privacy by design is ensured Measures the extent (number of measures) to which privacy by design has been ensured.Trust is paramount to the adoption of smart city solutions, which must fully respect individual freedom and the right to privacy.
The framework proposed includes six key steps: (a) Clustering of the technology/service solutions into Transition Tracks (five TTs are proposed); (b) definition of the main groups of stakeholders (four main groups are proposed); (c) definition of KPIs dimensions (six dimensions are proposed); (d) definition of KPIs repository per dimension; (e) definition of the scope of evaluation per KPI (six evaluation levels are proposed); and (f) threshold definition per KPI.The specific framework is generic on purpose, in order to satisfy various assessment requirements of a proposed technology solution, however specific recommendations and tips are provided based on its actual implementation and lessons learned from the IRIS smart city project.

Table 1 .
Indicative assessment frameworks by EC-funded projects.
[34] for the indicators should be easily available.As the inventory for gathering the data for the indicators should be kept as limited as possible, in time and effort, the indicators Social sciences provide approaches to deal with qualitative information in a semi-quantitative way[34].•Reliability:The definitions of the indicators should be clear and not open for different interpretations.This holds for the definition itself and for the calculation methods behind the indicator.• should be based on data that either: (a) are available from the project leader or others involved in the innovation case that is being evaluated; (b) or can easily be compiled from public sources, and c) or can easily be gathered from interviews, maps, or terrain observations.• Measurability: The identified indicators should be capable of being measured, preferably as objectively as possible.For the majority of indicators in the ICT, social and legal dimensions, quantitative measurability is limited.

Table 5 .
Repository of technical KPIs.

Table 5 .
Cont.Charging infrastructure development is critical for the promotion of electromobility and the deployment of EVs.This indicator assesses the level of service with regards to charging capabilities offered by measuring the number of electric vehicles charging stations deployed in the area.Additionally, it measures the number of solar powered V2G stations comparing it with the total number of stations.
1 Building (A), Set of Buildings (B), Energy Supply Unit (C), Set of Energy Supply Units (D), Neighbourhood (E), City (F); 2 Distribution System Operators (I), Consumers (End-Users) (II), Technology and Services Providers (III), Policy-Making Bodies and Governance (IV); 3 Indicative values extracted from similar Smart City Projects and/or relevant targets set by EU and literature.

Table 6 .
Repository of environmental KPIs.
Making Bodies and Governance (IV); 3 Indicative values extracted from similar Smart City Projects and/or relevant targets set by EU and literature.

Table 7 .
Repository of economic KPIs.

Table 8 .
Repository of social KPIs.

Table 8 .
Cont.Technology and Services Providers (III), Policy-Making Bodies and Governance (IV); 3 Indicative values extracted from similar Smart City Projects and/or relevant targets set by EU and literature.

Table 9 .
Cont.The use of open source software means less possibilities of vendor lock-in and more space for communities to develop together smart city solutions.It also lowers the software costs.Percentage of out-dated datasets on a city platform per timeframe.Open data can become out-dated and obsolete, which acts negatively on the attractiveness of using data from platforms.By monitoring the expiration dates of the data, the owner gets a message to renew or remove the datasets.

Table 9 .
Cont.Percentage of data that uses DCAT standards.The quality of open data is better if standardized.Processes get easier when data standards are applied.The DCAT standard allows municipal employees to produce data in a standardized way.Downtime per timeframe.To run a stable platform, monitoring is required to fix bugs and quickly improve the software environments.Number of services based on open data.To gain insight of the use of open data, mapping the applications developed based on the open data is vital.Measures the number of active 'touch-points' identified where citizens have a degree of interaction with solution.This is the basis for distinguishing communication and CE activities and for prioritising and mapping suitable activities to each IS.

Table 10 .
Repository of legal KPIs.The level of suitability of the legal framework for the integration of private EVs and public transport EVs in the city mobility policies.