Review of Smart City Assessment Tools

Abstract

Today’s cities are estimated to generate 80% of global GDP, covering only about 3% of the land, but contributing to about 72% of all global greenhouse gas emissions. Cities face significant challenges, such as population growth, pollution, congestion, lack of physical and social infrastructures, while trying to simultaneously meet sustainable energy and environmental requirements. The Smart City concept intends to address these challenges by identifying new and intelligent ways to manage the complexity of urban living and implement solutions for multidisciplinary problems in cities. With the increasing number of Smart City projects being implemented around the world, it is important to evaluate their strengths and weaknesses for their future improvement and evolution track record. It is, therefore, crucial to characterize and improve the proper tools to adequately evaluate these implementations. Following the Smart City implementation growth, several Smart City Assessment tools with different indicator sets have been developed. This work presents a literature review on Smart City Assessment tools, discussing their main gaps in order to improve future methodologies and tools. Smart City Assessment can deliver important performance indicators monitoring for the evaluation of multiple benefits for different actors and stakeholders, such as city authorities, investors and funding agencies, researchers, and citizens.

1. Introduction

Today’s cities face significant challenges, such as increasing population, pollution, congestion, resource usage, lack of adequate physical and social infrastructures, sustainable economic growth, and increasingly stricter energy and environmental requirements [1,2].

According to the United Nations [3], more people live in urban areas than in rural areas. In 1950, only 30% of the world’s population was living in an urban environment, but in 2018, this number increased to 55%. In 2030, it is expected to increase to about 60% [4] and by 2050, the projections point to about 68% of the world’s population being urban [3]. The United Nations [3] states that “the growth in the urban population is driven by overall population increase and by the upward shift in the percentage living in urban areas”. Together, these two factors can add about 2.5 billion to the world’s urban population by 2050, with almost 90% of this growth happening in Asia and Africa [3]. The urbanization process is occurring at different rates, being considerably faster in developing regions than in the developed ones. Africa is expected to be the fastest urbanizing region and according to Joint Research Centre [5], in the last 25 years, the urban population has more than doubled in almost all countries across sub-Saharan Africa, although meanwhile, in many parts of North America and Europe, the urban population has been declining. In 2016, there were, globally, about 512 urban centers with at least 1 million inhabitants and 31 megacities with over 10 million inhabitants [6]. By 2030, these numbers are projected to increase to 662 urban centers and 41 megacities, most of which in developing regions [6].

Nowadays, cities are estimated to generate 80% of all economic growth [7], covering only about 3% of the land, but producing about 72% of all global greenhouse gas emissions [8]. All the continuous population growth, and especially in urban areas, put a lot of challenges in the design of cities and exceptional sustainability challenges, both on infrastructures and the environment [9].

The development challenges of cities have attracted many interdisciplinary fields of study. Computer Science is one of the multiple fields addressing city development challenges. The interest of Computer Science began at the end of the 20th century when cities were idealized to become a network of computers and other electronic based devices [10]. Nowadays, cities are composed of even more complex systems in constant evolution, in which the combination of multiple research areas is needed in order to overcome the arising endless challenges [10]. In order to make cities and communities more inclusive, safe, resilient, and sustainable, as per the “Sustainable Development Goal 11” of the 2030 Agenda for Sustainable Development [11], it is crucial to understand the key trends in the future of urbanization [4], and to evaluate city growth development and implementation measures to fulfill the multi-dimension human and sustainable requirements.

The “Smart City” concept is intended to address the referred challenges by identifying new and smart ways to manage the complexity of urban living and implement solutions for multidisciplinary problems ranging from energy consumption, resource management, environmental protection, security, quality of life, the efficiency of urban operation, and the availability of a wide variety of services [9,12,13].

The estimated number of smart city implementations varies depending on the definition and extent of the elements included in the analysis. A Navigant Research global market report [14] estimated that in 2017, there were more than 250 smart city project implementations in 178 cities worldwide. Europe is the top-performing geographical area with 12 cities ranking among the top 25 smart cities ranked by the IESE Cities in Motion Index 2018 [15]. Europe is aiming to have about 300 smart cities by the end of 2020 and India plans to build about 100 smart cities by 2022, impacting a population of almost 1 billion people [16].

“Smart City” as a new research concept began in 1992 with the book entitled “The Technopolis Phenomenon: Smart Cities, Fast Systems, Global Networks” [10,17]. Since then, this research area has not only attracted different research areas, but also many stakeholders, from governmental organizations to industries and social society, supporting the growing interest of the “Smart City” concept, within the global sustainable development framework.

According to Mora [10] the “Smart City” research established itself as a new scientific field in the year 2009, but despite all the growing number of publications, the concept is far from having a clear and established definition. Many definitions for the concept have been developed, but so far none has been commonly accepted by the scientific community. It is common to see different interpretations according to the context of the publications [13,18]. The absence of such a definition has been pointed out in several publications, namely [1,10,19]. According to the bibliometric analysis of the literature published between 1992 and 2012 performed by Mora [10], there is a tendency among publications to differentiate themselves, being difficult to acquire a common agreement.

Some publications mostly see Smart City as a technological innovation whereas others see it as social innovation. There is the idealization of a Smart City being supported by a technological combination of different IoT devices forming a wireless sensor network (WSN), for numerous control and monitoring applications fields namely for utilities, security, asset tracking, and smart metering [20]. The bibliometric analysis of the literature developed by Mora [10] points out that “researchers seem to agree in picturing the smart city as an urban environment in which an ICT-driven approach to urban sustainability is activated”. The same analysis states that there are although different research paths being followed. The two most cited groups of publications describe the smart city as a more techno-centric approach and the other, one of the most cited and influential documents in the area published between 1992 and 2012, Giffinger [21], as a balanced combination of human, social, cultural, economic, environmental, and technological factors [10]. Technology should be seen as a means to achieve the outcomes of a Smart City implementation according to the needs of its location (environment, energy, people, business, governance, etc).

The Smart City concept is still evolving and there is not yet a commonly accepted worldwide definition, but it is quite clear that Smart Cities leverage information and communication technologies to enhance service levels, citizen well-being, sustainability, and economic development. Expanding the Smart City concept will, for instance, improve the existing infrastructures, increase the digitalization and integration of facilities and services, expand the use of collaboration tools and automated service management platforms, increase the safety of citizens and transportation, making cities more attractive for stakeholders, and living up to citizens’ expectations. According to an OECD recent report [22], when well connected with inclusive growth objectives, Smart City implementations can, besides many other advantages, offer powerful tools to support the shift from in-person to remote service delivery, mitigating the fallout of the crisis on urban residents and businesses, including the most vulnerable ones, and empower new forms of local governance. However, the same report states as a possible threat to society the “increased inequality among digitally marginalized groups” and “possible abuse of citizen data, privacy and safety”.

Different and distinct Smart City concepts and architectures have emerged during the past decades, but all of them visualize it as being composed of different inter-related components, such as data, economy, people, governance, mobility, environment, and living [13,23]. All of the components are related to different aspects of urban living, such as communication, industry, education, e-democracy, logistics and infrastructures, efficiency and sustainability, security, and quality [23]. All this diversity of components and related aspects has attracted multi-disciplinary approaches and science fields to the Smart City developments, which has contributed to the enrichment of scientific knowledge and increase and scale-up of innovative technology developments.

Despite all the different research paths and lack of commonly accepted definition in the trending Smart City concept, one point is clear; the impact evaluation of Smart City implementations is extremely valuable and strongly needed. Every impact evaluation has the principle of permitting the characterization of achieved levels of performance according to a certain target goal. This is crucial for the planning of the city further developments and upgrades, to assess the strengths and weaknesses, for comparison with other implementations and to inform the different interested stakeholders about the implementation status.

This paper is focused on the assessment tools for Smart City implementation evaluation, which can also be mentioned as “Smart City Assessment (SCA)”. In the literature, it is possible to find different names for such evaluation tools, such as “rankings”, “ratings”, “indicators”, “key performance indicators (KPI)”, “indexes”, “frameworks”, and “assessment tools”. In the scope of this publication, the mentioned “Smart City Assessment (SCA)” will refer to an assessment process or tool that can evaluate the performance of a certain indicator in the scope of a Smart City concept implementation.

The remainder of the paper is structured as follows: Section 2 presents the method used for the literature review. Section 3 and Section 4 presents the results of the literature review in terms of the smart city concept and the smart city assessment, respectively. Then, the existing research and gaps in SCA are assessed in Section 5. Finally, Section 6 discusses the findings and presents the main conclusions.

2. Literature Review

In the literature, it is possible to find “Smart City” analogous concepts referred to in relatively different ways. Several different wordings appear, with “Green City”, “Sustainable City”, and “Intelligent City” being the most common [10,18,24,25,26,27,28]. Using the “Web of Science” search engine with the string: AK = ((“smart” OR “green” OR “sustainable” OR “intelligent”) AND (“city” OR “cities”))], it was possible to find 8,518 publications with the referred keywords since 1992. The searches on Web of Science (webofknowledge.com) were made on the 31 July 2020, and revealed that the number of publications has been constantly increasing over the years reaching a total of 1599 in 2019. In 2020, about 730 publications had already been published.

The wording is also quite diverse when referring to tools or studies that can provide an evaluation of Smart City implementations. In order to find and refine the search of articles related to the subject of this article, the following string was inserted in the search engine: AK = ((“smart” OR “green” OR “sustainable” OR “intelligent”) AND (“city” OR “cities”) AND (“tool” OR “assessment” OR “indicator” OR “key performance indicators” OR “KPI” OR “framework” OR “toolkit” OR “index” OR “rating” OR “ranking”)). In the last 20 years, a total of 372 publications were found, from which 298 (80%) were published in the last five years (since 2015). The years with the greatest number of publications were 2017 and 2018, with respectively 63 and 62 yearly publications. In 2020, there are already 41 publications captured by the search string. The countries with most publications are the Popular Republic of China, with about 79 publications (21%), Spain with 34 (9%), and Italy with 31 (8%). A more refined search was done to this preliminary screening of results, based on the publication’s number of citations and impact factor. Afterward, to further refine the selection an abstract reading was conducted that was able to identify the most adequate publications in the scope of this paper. This methodology was believed to identify the results that are most satisfactory possible for the review. Other publications could be read and cited, but with potentially less added value to the final result. The conclusions regarding the existing tool gaps were collected from the analyzed publications, preferring the ones that clearly stated a satisfactory and reasonable evaluation methodology.

It is clear by the demonstrated increasing number of articles being published over the years, that there is a growing research interest for the subject. This paper intends to present an updated review on existing SCA tools and their main gaps, being a starting point for those initiating their research on the subject. With SCA being a nascent field with a growing potential to evolve [13], having a good review of the strengths and weaknesses of existing tools can be important for future developments.

3. Smart City Concept

As already mentioned, in the literature, different expressions can be found referring to similar concepts of “Smart City”. The use of combined wordings can also be found, suggesting or definition new concepts, sometimes for differentiation purposes, like for instance the “Smart Sustainable City” concept [29]. Although sometimes not exactly being in the same focus, the use of all these different expressions has been creating terminological confusion [26,30,31]. Today, “Smart City” is probably the most widespread and accepted wording among citizens, media, investors and companies, and public authorities. It is being used by public authorities and the business sector, as it is a buzz word understandable by most of the targeted audiences [29].

According to Grabys [32], references to Smart City similar concepts can be traced back from the 1960s, when the literature refers to “informational or cybernetically planned cities” and after that, in the 1980s, when references appear to the “networked cities” or “computable cities”. The Smart City concept was introduced in 1994 [33]. Later, by 2010, the European Union (EU) began to use the “smart” wording to refer to sustainable projects and actions related to the urban environment [33]. With the appearance of Smart City project implementations supported by the European Union (EU), the interest regarding the subject began to increased significantly [34].

The literature review performed in 2014 by Mosannenzadeh [35], showed that the concept of Smart City was being developed in three main areas: Academic, Industrial, and Governmental, creating different points-of-view by each domain area. At the time, it was pointed out that there was no common definition for the concept and that the disparity of wordings used in different definitions was a sign of controversy in the concept [35]. Nowadays, and despite all the growing number of publications since then, there is still not yet a common and acceptable definition for the Smart City concept, although researchers seem to agree in the significant contribution of the use of technologies for urban development [10,35].

However, there seems to be huge differences and confusion between the concept definitions and what actually is being done in real-world implementations and even in the scope of the main definitions for most reputed international institutions. The literature refers to “Sustainable City” as being the umbrella where other concepts like “Smart City” are under. “Sustainable City” is referred as including a harmonized integration of social, economic, environmental, and institutional aspects, whereas “Smart City” is pointed out as having the main goal of improving the sustainability of cities through the application of ICT, the collaboration of key stakeholders, and integration of domains [36]. On the other side, other publications refer that one of the many attributes or dimensions of Smart Cities is “Sustainability” [9,13,28,37], and that “a general goal of Smart Cities is to improve sustainability with the help of technologies” [34].

The vision of Smart Cities according to the European Commission (EC) is beyond the use of ICT. It is seen as interactivity between different infrastructures with the main goal of providing benefits in all sectors. According to the EC [38] Smart Cities are “Cities using technological solutions to improve the management and efficiency of the urban environment”. The EC [38] refers to Smart City as “a place where traditional networks and services are made more efficient with the use of digital and telecommunication technologies for the benefit of its inhabitants and business”. It is also stated that “a smart city goes beyond the use of information and communication technologies (ICT) for better resource use and fewer emissions. It means smarter urban transport networks, upgraded water supply and waste disposal facilities and more efficient ways to light and heat buildings. It also means a more interactive and responsive city administration, safer public spaces and meeting the needs of an aging population” [38].

IEEE also has this broader understanding of the Smart City concept where technology is seen as an enabler for a better quality of life and to decrease environmental impacts [34]. IEEE Smart Cities Initiative refers that “a Smart City brings together technology, government and society and includes but is not limited to the following elements: A smart economy, Smart energy, Smart mobility, A smart environment, Smart living, Smart governance” [39].

The United Nations, on their “Smart Cities and infrastructure report” [40] also refer that there is no standardized commonly accepted definition for Smart City. In the report [40], it states, as a reference, the definition provided by the International Telecommunication Union (ITU) after analysis in 2014 about 100 different definitions, and presented at the ITU-T Y. 4900 recommendations [41]: “an innovative city that uses information and communication technologies (ICTs) and other means to improve quality of life, the efficiency of urban operation and services, and competitiveness, while ensuring that it meets the needs of present and future generations with respect to economic, social, environmental as well as cultural aspects”. Although this definition provided by ITU [41] is actually for “Smart Sustainable City”, the United Nations use it as a reference when mentioning to the definition of Smart City, stating also that “Governments and stakeholders need to work together to develop a common understanding of what Smart City means in their specific national and city-level contexts” [40]. Nowadays, the United Nations refers to the wording “Sustainable Cities and Communities” as a means to “make cities and human settlements inclusive, safe, resilient and sustainable” [42].

4. Smart City Assessment

As referred by Sharifi [13] Smart City Assessment (SCA) “is a nascent field with much potential for future development”. There is limited research analyzing SCA tools that Sharifi [13] attributes to the fact that SCA is a relatively new field of research and practice. The evaluation of Smart City implementations should be seen as critical, in order to historically document their strengths and weaknesses, for the scope of future improvements and to inform interested stakeholders about the level achieved in different target goals. SCA tools can also be used to present city-rankings, revealing the best (and the worst) places for certain activities, which is pointed out by literature to be a central instrument for assessing the attractiveness of urban regions [21,43,44]. The SCA and resulting rankings can thus be used by cities to improve their position in the competition with other regions. A highly reputed city ranking helps to improve the international image of a city and can, therefore, play a central role in its marketing strategy [21]. Giffinger [21] mentioned a detailed analysis and comparison of 10 German rankings. The city rankings referred to in (Giffinger et al., 2007), which have been published by the magazines focus, capital, impulse, and Bizz between 1994 and 2003, point out the following benefits for city rankings:

• City rankings draw public attention to major issues affecting the quality of life of the citizens and promote healthy competition;
• City rankings stimulate a broad discussion on regional development strategies;
• Regional actors are forced to make their decisions transparent and comprehensible;
• Positive changes are also registered outside the region;
• The results in detail may initiate the learning effects of local actors;
• On the other hand, the same report identifies the handicaps of city rankings;
• City rankings tend to neglect complex interrelations in regional development;
• The discussion is mainly focused on the bare rank;
• Long-term development strategies may be threatened;
• Existing stereotypes may be strengthened.

Based on the literature review presented by Sharifi [13], the next

Table 1. Potential multiple benefits of Smart City Assessment (SCA) for different stakeholders [13].

Stakeholder	Potential Benefits
Cities and city authorities	Monitoring performance to improve the international image and competitive position of the city in the eyes of investors, as well as creative citizens and the public [21,43,44,45,46] Justify the value of smart city investments and interventions [47] Identify strengths and weaknesses and guide smart city planning [21,43,44,48,49,50] Track progress in achieving pre-defined goals and targets and identify the position of the city in its efforts towards smartness [48,51] Understand the socio-economic and environmental implications of smart city projects [52] Understand the technical requirements of smart city projects [52] Learn from the experiences of peers (when assessment involves benchmarking) [21,44,50,53] Identify and showcase best-practice cases to learn lessons from [21] Enhance governance transparency [21,44,50,53] Stimulate discussions among various stakeholders that may result in improved mobilization of resources [21,44,54]
Investors and funding agencies	Evidence-based evaluation of the completed or ongoing projects [47] Scientific means for prioritizing funding allocation [45] Enhanced ability to make decisions regarding the best sites for future investment [44,46] Ability to identify and capitalize on new business opportunities [46]
Researchers	Develop new strategies for improvement of smart city performance [47] Simplify the complexities of the smart city concept [9,24]
Citizens	Enhanced awareness about the benefits of smart city projects [47] Ability to make informed decisions when it comes to future investments [45] Motivation to engage in smart city development activities and to communicate their desires and priorities to city authorities [45,46,53]

presents the benefits of SCA for the different stakeholder’s categorization.

Table 1. perfectly summarizes the existence of multiple benefits for the SCA activities and that they may offer different perspectives for each one of the actors and stakeholders involved.

5. SCA Research Review and Identified Gaps

The assessment of citie’s smartness has received much more attention in recent years, but very few studies have analyzed SCA tools and their strengths and weaknesses [13]. According to Sharifi [13] there are two major approaches to the study of SCA tools: One, focused on providing an overview of the tools, and another one involving more detailed analyses of the tools to better understand their thematic focus and the typology of their indicators.

Albino [28] provided a brief overview of frameworks, rating systems, and a number of indicators, for existing smart city assessment until 2015. The analysis is performed in a more quantitative overview of the indicators used with very few qualitative details, thus it is not possible to evaluate the potential gaps of the presented assessment tools. The tools and frameworks presented are, for instance, smart ranking systems developed by the University of Vienna, the Intelligent Community Forum’s Smart 21 communities, the Global Power City Index, the Smarter Cities Ranking, the World’s Smartest Cities, the IBM Smart City, and the McKinsey Global Institute rankings.

The main objective of the CITYkeys European project [55] that started in February 2015 and ended two years later (2017) was to develop and validate a framework for transparent performance evaluation, including the definition of key performance indicators (KPIs) and guidelines for data collection and a performance measurement system prototype. The CITYkeys project consortium performed a survey of existing KPI systems for project evaluation in order to create an inventory of existing assessment frameworks or tools. The inventory included relevant H2020 and FP7 projects, as well as relevant projects from the countries of the three research organizations involved: Finland, the Netherlands, and Austria. It also included the status of relevant initiatives that were under development at the time, like the smart city framework of CEN/CENELEC and the Smart City Information System [56]. A total of 43 indicator frameworks were scanned for indicators that could be related to the CITYkeys pre-selected subthemes and thus potentially be used for the CITYkeys framework. Based on this inventory analyzed on Neumann [57], it is reported that in general terms, the analyzed frameworks suggested that the availability of the KPI was saturated. Neumann [57] refers that numerous indicators that existed at the time and where being used across different sectors in cities and districts in Europe, as well as globally. It also reported the following gaps in terms of indicator availability: Multilevel governance, education, employment, scalability, and replicability. The report suggests that there was a significant variation in the coverage of the different subthemes, including for instance the ”Energy and Mitigation” and ”Environment” subthemes [57]. It should be noted that the gap analysis was carried out only by linking subthemes (and their associated indicators) of existing frameworks with the selected subthemes of the CITYkeys framework, thus not being an exhaustive analysis of the existing indicators individually.

There have been also several approaches to standardize the indicators from which the frameworks or rankings can provide an assessment for Smart City implementation. Recently, Huovila [58] provided a quite extensive comparative analysis of existing standardized indicators for Smart City Assessment. Such publication performs an evaluation of seven recent existing standardized frameworks with 413 indicators.

Table 2. Summary of international standardized indicators analyzed in [58].

Name, Type and Abbreviation	Main Categories Addressed
ISO 37120:2018 sustainable development of communities – indicators for city services and quality of life International non mandatory standard Abbreviation: ISO 37120 [59]	Economy, education, energy, environment and climate change, finance, governance, health, housing, population and social conditions, recreation, safety, solid waste, sport and culture, telecommunication, transportation, urban/local agriculture and food security, urban planning, wastewater, water
ISO/DIS 37122:2019 sustainable development in communities - indicators for Smart cities DIS = draft international standard Abbreviation: ISO 37122 [60]
ETSI TS 103 463 key performance indicators for sustainable digital multiservice cities TS = technical specification Abbreviation: ETSI indicators [61]
ITU-T Y.4901/L.1601 key performance indicators related to the use of information and communication technology in Smart sustainable cities Type: Recommendation Abbreviation: ITU 4901 [41]	ICT, environmental sustainability, productivity, quality of life, equity and social inclusion, physical infrastructure
ITU-T Y.4902/L.1602 key performance indicators related to the sustainability impacts of information and communication technology in Smart sustainable cities Type: Recommendation Abbreviation: ITU 4902 [62]	Environmental sustainability, productivity, quality of life, equity and social inclusion, physical infrastructure
ITU-T Y.4903/L.1603 key performance indicators for Smart sustainable cities to assess the achievement of sustainable development goals Recommendation Abbreviation: ITU 4903 [63]	Economy, environment, society and culture
Sustainable Development Goal 11+ monitoring framework UN Inter-Agency Expert Group definition Abbreviation: UN SDG 11+ indicators [42]	UN SDG targets 11.1, 11.2, 11.3, 11.4, 11.5, 11.6, 11.7, 11a, 11b, 11c, 1.4, 6.3

shows the standards analyzed and their main categories addressed.

The analysis provided by Huovila [58] indicated that there is a lack of balance between the different indicators, namely between the indicators related to sustainability and smartness. There is also a strong emphasis on the smartness indicators [13,59]. Huovila [58] affirms that ISO, ETSI, and Sustainable Development Goal 11 are well documented, but the ITU standards have a short definition of the indicators, which may lead to different their calculation methods, leading to different results in each assessment. It is even stated, when referring to the lack of documentation, that “without provision of sufficient metadata and guidance, they might have limited implementation in practice and lack reliable cross-city comparability” [58].

The assessment tools use different indicators. These indicators should be selected taking into consideration multiple complex factors, but the developers of SCA tools need to take into account that the methods should follow simple and applicable guiding principles in order to approach city managers, that are interested in the big picture and not on technical or methodological details [58]. Huovila [58] refers that “further developments of indicator frameworks need to be embedded in the analysis of cities needs”. This is an important note since, sometimes, the indicators used in the cities evaluation are not in line with the city goals. This can, for instance, contribute to a poor evaluation of future city developments, that are evaluated according to indicators that have nothing to do with what was idealized by the city managers or planers.

Ahvenniemi [34] performed an analysis of a set of 16 SCA tools. In the analysis, it is stated that “while environmental sustainability is an essential target of smart cities environmental indicators are clearly underrepresented in the smart city frameworks analyzed”. Ahvenniemi [34] recommends that Smart City assessment “should not only use output indicators that measure the efficiency of deployment of smart solutions but always also impact indicators that measure the contribution towards the ultimate goals such as environmental, economic or social sustainability”.

A more detailed critical review about SCA tools was recently developed by Sharifi [13] where 34 selected SCA tools were analyzed was presented. This publication presents a rather interesting literature review about the Smart City concept and existing research on SCA tools, where the author identifies two main gaps: The first is that the majority of the studies conducted so far have mainly analyzed a limited number of tools, and the second is that the existing research is mainly focused on thematic focus and indicator type, neglecting many other important issues related to assessment tools [13].

Table 3. Summary of SCA tools analyzed in [13].

Tool	Year	Primary Developer(s)
Lisbon ranking for smart sustainable cities	2019	[9]
Smart Sustainable Cities China	2019	[64]
Cities in Motion Index	2018	Center for Globalization and Strategy and IESE Business School’s Department of Strategy [65]
Global Power City Index	2018	The Mori Memorial Foundation’s Institute for Urban Strategies [66]
Innovation Cities™ Index	2018	China Academy of Telecommunication Research and China Communications Standards Association [67]
EasyPark	2018	EasyPark Group [68]
IoT-Enabled Smart city framework	2018	National Institute of Standards and Technology [69]
Smart Cities Council’s tools and frameworks	2018	Smart Cities Council, Australia and New Zealand [70,71]
What Works Cities	2018	Bloomberg Philanthropies [72]
Code for Smart Communities	2018	Smart Cities Council Australian New Zealand and the Green Building Council of Australia [73]
China Smart City Performance	2018	Shen [74]
Smart City Governments	2018	Eden Strategy Institute and ONG&ONG Pte Ltd. [75]
Assessing Smart City Initiatives for the Mediterranean Region	2017	Universidad Politécnica of Madrid (UPM) [76,77]
Smart Cities Index- India	2017	Indian School of Business [45]
Juniper Research smart city frameworks	2017	Juniper Research [77]
UK Smart Cities Index	2017	Navigant Research [78]
CITYkeys	2016	Netherlands Organization for Applied Scientific Research (TNO) [55]
Networked Society City Index	2016	Ericsson in collaboration with Sweco [79]
Cities of Opportunity	2016	PricewaterhouseCoopers (PwC) [80]
Community KPIs for the IoT and Smart Cities	2016	Future Everything [81]
Gulf States Smart Cities Index	2016	Navigant Research [82]
European Digital Cities Index	2016	Nesta [83]
Smart City Strategic Growth Map	2016	ESPRESSO, European Commission [84]
City IQ Evaluation System	2015	Wu [85]
International Data Corporation (IDC) Smart City Analysis	2015	IDC [86]
Telecommunication and Standardization Sector of International Telecommunication Union (ITU)	2015	ITU-T Focus Group on Smart Sustainable Cities [87]
United Nations Economic Commission for Europe-ITU Smart Sustainable Cities Indicators	2015	UNECE Committee on Housing and Land Management, Environment Agency Austria, and ITU [88]
Smart Cities Ranking of European Medium-sized Cities	2014	TU Vienna, in cooperation with the University of Ljubljana and the TU of Delft [21]
Boyd-Cohen Smart City Index	2014	Boyd-Cohen [89]
Mapping Smart Cities in the EU	2014	RAND Europe, European Union (EU) [90]
Smart City Maturity Model and Self-Assessment Tool	2014	The Scottish Government and Scottish Cities Alliance [51,91]
Smart City Profiles	2013	Austrian Climate and Energy Fund and Environment Agency Austria [92]
United Cities and Local Governments (UCLG) smart cities study	2012	City of Bilbao and Committee of Digital and Knowledge-based Cities of UCLG [93]
Smart Cities Benchmarking in China	2012	China Academy of Telecommunication Research and China Communications Standards Association [94]

presents the 34 selected SCA tools analyzed by Sharifi [13].

The analysis performed by Sharifi [13] is quite extensive and presents the results of his analysis of all the 34 SCA tools according to the following sub-sections:

• Comprehensiveness:

  a.

  The extent of inclusion of indicators in the selected tools;

  b.

  Distribution pattern of indicators

• Stakeholder engagement;
• Context sensitivity;
• Strategic needs;
• Uncertainty management;
• Interlinkages and interoperability;
• Temporal changes;
• Flexibility;
• Feasibility;
• Presentation and communication of the results;
• Action plans.

Several strengths and weaknesses are pointed out to each one of the SCA tools according to the main purpose of the previous sub-sections. In terms of weaknesses, which can be considered a gap in the assessed SCA tools, it is worth mentioning at least the following [13]:

• There is a lack of balanced distribution of indicators;
• The most tools have not taken measures to engage stakeholders in their development and implementation processes;
• The majority of the selected tools do not consider locally specific conditions;
• Overall, the issue of feasibility has not been well addressed across the tools;
• Only about 25% of the tools provide recommendations on linking results to action plans.

After evaluating the selected SCA tools, using the developed framework, Sharifi [13] states that in “overall, limited success has been achieved and several improvements are needed as the tools continue to evolve over the coming years”. The author also affirms that in order to anticipate future changes and to avoid being overwhelmed by constantly changing dynamics of urban systems, SCA tools should consider the use of separate and scenario-making techniques for dealing with future uncertainties [13].

According to Sharifi [13] extensive analysis “more research is needed to gain further details about the suitability of individual SCA tools for guiding cities towards smartness” and that “such research should be informed by empirical findings from case studies to better understand the complexities of smart city assessment”. This statement shows the importance of testing the SCA tools in real scenarios through the evaluation of real-world Smart City implementations.

6. Discussion and Conclusions

With an increasing world population and the shift towards urban living environment [3], cities face multidisciplinary challenges that require innovative and intelligent solutions. Smart City implementations can help to overcome the limitations of traditional urban development models, improving Citie’s sustainability and resilience.

Even if no consensual definition for the Smart City concept is available, all of the existing ones have at least the same common goal: Improve the quality of urban life [95]. There are limited concept definitions provided by international organizations, and researchers and SCA tool developers tend to follow different paths. An international agreement on such definitions should be provided by an agreement between international sustainable development organizations (e.g., United Smart Cities program from UNECE – United Nations) and/or international standardization bodies (e.g., ISO, IEC and ITU).

The Smart City concept can be seen as a balanced combination of human, social, cultural, economic, environmental, energy, and technological developments, commonly integrated for the same purpose, aiming to overcome the main challenges of urban living. Technology should be seen as a means to achieve the outcomes of a Smart City implementation according to the needs of its location (environment, energy, people, business, governance, etc). We see Smart Cities in a broader approach, with the use of technologies facilitating the interaction between different utilities, infrastructures, services, and systems, with the main purpose of supporting the sustainable use of resources and increasing city administration efficiency, benefiting inhabitants, governance, and economic growth.

Smart City Assessment tools are important for the proper evaluation of Smart City implementations, bringing benefits for all the different involved stakeholders [13]. Such benefits can be very diverse, namely: Improvement of the international city image [21,43,44,45,46], prioritization of funding allocation [45], development of new strategies for performance improvement [47], and motivation for engagement in development activities and to communicate desires and priorities. The tools should enable the evaluation of the implementations regarding their impacts on the local objectives, but also on global established goals [45,46,53].

The main purpose of the Smart City Assessment is to give feedback and guidance for decision-making, enabling the assessment of whether the implementations are proceeding towards the wanted direction [34]. Many of the existing SCA tools are nowadays mainly used for promotional purposes and very few for an evaluation of what actually should be done in order to increase the performance of future developments.

The diverse approaches followed by different SCA tools can be partly explained by the fact that there is no consensus on the actual definition of the Smart City Concept [18]. This fact may introduce confusion into the main stakeholders involved. Even the main international organizations and standard bodies sometimes use differentiated wordings when addressing similar concepts. Therefore, an international effort should be made so that a common definition for the different concepts can be agreed.

This paper carries out an updated literature review of the main different approaches of the Smart City concept and of the different SCA tools being developed in the last decades. The different perceptions of the Smart City concept are somehow generating distinct SCA tools that exacerbate different components and aspects, according to the perspectives theirs developers. The following gaps, regarding the available SCA tools, were identified:

• There is a lack of balance distribution of indicators;
• The great majority of the SCA tools are static assessments or snapshots;
• They present limitations when comparing cities with different scales;
• In general, they do not measure or evaluate the impacts on the specific local needs or goals of the implementation;
• The engagement of stakeholders in Smart City developments and implementation processes is not evaluated;
• They do not evaluate the contribution to the United Nations Sustainable Development Goals (SDG) or other worldwide targets/goals;
• The feasibility of the implementations is not evaluated.

The development of a single tool, able to overcome all the mentioned gaps of the existing SCA tools, is considered to be an quite ambitious task. Efforts should be made to at least develop an assessment tool capable of performing an evaluation track record, with, for instance, automated acquisition of several important indicators, and the possibility to compare implementations with different scales (namely in terms of area and financial dimensions).

The lack of balance between the indicators can be solved by the combination of different indicators sets already defined by the literature, and the inclusion of several others of which it is possible to obtain. Sometimes, this unbalance is related to the SCA tool developers that would like to emphasize specific outcomes of the implementations.

The proliferation of wireless sensor networks (WSN) capable of acquiring data in an automated and synchronous way [20] could provide real-time or quasi-real-time data for several of the indicators that compose a SCA tool. This would contribute to a constant update on the assessment of certain domains (i.e., water and energy consumption) of a Smart City implementations, that could provide insights about their evolution track record, for better definition of future improvements and actual verification of their efficiency. In certain domains, this constant monitoring of data could also provide relevant data to better evaluate the feasibility of the implemented measures.

The literature review presented in this paper provided insights about the existing gaps in the available SCA tools. Future research needs to be carried out on the analyses of indicators sets that could be included in a SCA tool, capable of overcoming several of the existing identified gaps. Also, it will be useful for the evaluation that, as much as possible, several of these indicators could be automatically acquired using automated information systems like sensing and metering devices and other data collection platforms, namely smart energy meters, smart water meters, outdoor air quality stations, traffic monitors, and waste management platforms. The authors are researching the development of such a SCA tool that can create added value in relation to the existing ones, contributing to overcome some of the existing identified gaps. The SCA tool under development will not only provide an overall rating of Smart City implementations, with all the multiple benefits for different stakeholders, but also will be able to provide different size implementation comparisons, historical overview and evolution, and different ratings for the main involved stakeholders, using a multi-criteria/multidimensional analysis. The different ratings will not only establish an overall rating of the city, but also a rating reflecting the different stakeholder’s points of view (i.e., city authorities, investors, researchers, citizens). This framework is believed to contribute significantly to the knowledge, evaluation, and empowerment of the stakeholders, increasing their willingness to actively contribute and participate in future implementations. Such an SCA tool can also be used in validation methodologies of both large- and small-scale pilot implementations, since it will consider the scale of the implementation.