Analysis of the Building Smart Readiness Indicator Calculation: A Comparative Case-Study with Two Panels of Experts

Abstract

The last release of the Energy Performance of Buildings Directive 2018/844/EU stated that smart buildings will play a crucial role in the future energy systems. Consequently, the Directive introduced the Smart Readiness Indicator in order to provide a common framework to highlight the value of building smartness across Europe. The methodology for the calculation of the Smart Readiness Indicator is currently under development and therefore not yet officially adopted at the European Union level. In this context, the current research analyzed the second public release of the proposed methodology, discussing the feasibility of its implementation and the obtained results through a practical application. Specifically, the methodology was applied to a nearly zero-energy office building located in Italy, and the evaluation was carried out in parallel by two different expert groups composed by researchers and technical building systems specialists. With the aim of analyzing the impact of subjective evaluations on the calculated indicator, a two-step assessment was adopted: in a first phase the two groups worked separately, and only in a second phase they were allowed to compare results, discuss discrepancies and identify the difficulties in applying the methodology. As the main outcome of this research, a set of recommendations are presented for an effective broad implementation of the Smart Readiness Indicator, able to increase the relevance of its evaluation and effectiveness, as well as to enhance the comparability of smart readiness of buildings through the definition of benchmarks and to integrate with other measurable key indicators, especially concerning energy flexibility.

1. Introduction

1.1. Smart Buildings in the European Union Strategy

Smart buildings have been assuming an increasing interest in the European Union in the last years. Indeed, more than 40 Horizon 2020 projects have dealt with the topic of smart buildings and building automation from different perspectives [1], a variety of smart technologies and services are already available in the market [2], and features and potential of smart buildings have been explored in the literature [3,4]. This trend was recognized by the European Commission as well, which, through the revised Energy Performance of Buildings Directive (EPBD) 2018/844/EU [5], acknowledged smart buildings as key enablers of future energy systems characterized by a large share of renewables, distributed supply and, in particular, demand-side energy flexibility [6]. In order to make the added value of building smartness more tangible for building users, owners, tenants and investors, the Directive introduces an optional Common Union scheme for rating the smart readiness of buildings, i.e., the Smart Readiness Indicator (SRI). As stated in the Directive, such an indicator is meant to measure the capability of a building to “use information and communication technologies and electronic systems to adapt the operation of buildings to the needs of the occupants and the grid and to improve the energy efficiency and overall performance of buildings”. In further detail, the European Directive brings out key smart readiness functionalities:

• “The ability to maintain energy performance and operation of the building through the adaptation of energy consumption for example through use of energy from renewable sources”;
• “The ability to adapt its operation mode in response to the needs of the occupant while paying due attention to the availability of user-friendliness, maintaining healthy indoor climate conditions and the ability to report on energy use”;
• “The flexibility of a building’s overall electricity demand, including its ability to enable participation in active and passive as well as implicit and explicit demand response, in relation to the grid, for example through flexibility and load shifting capacities”.

Thus, building, occupants, and the interaction with the energy grid are the focus points of the European Smart Readiness Indicator, and the Directive drives towards high performance buildings characterized by low energy demand, high levels of indoor environmental quality, high flexibility, large use of renewable energy sources, and a high degree of integration with the grid. The proposed SRI enforces energy efficiency at the building level and aims to measure, on the one hand, the adaptation of building energy consumption in response to both availability of renewable sources and needs of the occupants, and, on the other hand, the flexibility of the power demand of a building, expressed as participation in demand response strategies in relation to the grid requirements [7]. Indeed, this kind of new Smart Readiness Indicator is expected to play a supporting role for the development of intelligent networks as well, facilitating energy flexibility and power-to-heat solutions driven by renewable energy sources [8]. Finally, the Smart Readiness Indicator is meant to raise awareness amongst building owners and occupants of the optimal use of the building system [9], and of the value behind building automation and electronic monitoring of technical building systems, giving confidence to occupants about the actual savings achieved thanks to the new enhanced functionalities [10]. Consequently, a reduction of the performance gap between the current building energy performance certificates and the actual behavior is expected as well [11].

1.2. Smart Readiness Indicator as Key Performance Indicator( KPI)

The Smart Readiness Indicator, SRI, as intended by the European Directive 2018/844/EU, can be considered a top-level Key Performance Indicator, KPI. Indeed, as observed by [12,13,14], KPIs differentiate from simple metrics since they are both “predictive” and “persistent” and can be defined as “a high-level performance metric that is used to simplify complex information and point to the general state or trends of a phenomenon”.

In order to achieve the goals exposed in the previous section and successfully inform building owners and occupants, a clear definition of the SRI KPI is crucial. In particular, it is important to consider the target group of the indicator, in order to select a proper level of detail, compatible with the users’ set of competences, goals, focus, and tasks. For example, while building operators can benefit from the access to raw data and simple metrics for diagnostic purposes on an existing building system, this source of information can be misleading or scarcely useful for occupants, who often lack the necessary knowledge for correct building data analysis and interpretation. Indeed, although a whole-building level KPI provides generally limited insight into how buildings perform in detail, this kind of top-level indicators are frequently adopted for rating and certification purposes [15]. Other key aspects are related to the way in which the SRI KPI is defined, such as required inputs for the KPI calculation and temporal and space boundaries, as well as presentation of the calculated indicator, which is supposed to follow the specific energy policy of a given territory.

KPIs for smart buildings present in the literature can be distinguished according to the approach adopted to describe the building performances. In particular, it is possible to differentiate among:

• The type of indicator, which can be either (A) qualitative (e.g., [16]) or (B) quantitative/physical (e.g., [17,18,19]);
• The considered building performance, which can be focused, for instance, on (A) CO₂-emission, (B) cost efficiency (e.g., [20]), or (C) energy efficiency (e.g., [12]);
• The boundaries of the assessment, ranging from analyses on (A) individual buildings (e.g., [17,21]) to those on (B) clusters of buildings or districts (e.g., [22]).

Clearly, the SRI KPI can be defined in different ways and composed by more indicators belonging to the different categories. For instance, Rachman [23] observed that smart-ready buildings have both to show high performances (KPI category 3A) and to be grid-friendly (KPI category 3B). As regards the former performance criterion, the author proposed a characterization in terms of energy efficiency, indoor environmental quality, and exploitation of renewable energy sources, while, as far as the interaction with the grid was concerned, to consider the impact of the building system on the grid. Arteconi et al. [24] instead adopted a different methodological approach and proposed a single dimensionless indicator (the Flexibility Performance Indicator), calculated by means of a weighted average of four KPIs characterizing building flexibility: response time, committed power, recovery time, and actual energy variation.

1.3. The European Strategy for the SRI Definition

In order to support the establishment of the SRI in terms of possible scope and characteristics of the indicator, a first technical study has been commissioned and supervised by the European Commission services (Directorate-General for Energy). This study proposed a methodological framework for the SRI and the definition of smart services that such an indicator can build upon. The first study was conducted by a consortium of VITO, Waide Strategic Efficiency, Ecofys and OFFIS and finalized in August 2018. Its results were published in [16]. The consortium addressed its attention towards the development of a ‘European SRI’ as top-level qualitative KPI, able to embrace the different aspects mentioned in the Directive 2018/844/EU and to consider both the performance of single buildings and their behavior in a smart grid.

At the end of December 2018, a second technical support study was launched [25] with the aim to provide further technical inputs to refine the ‘European SRI’ scheme previously proposed. Building on the outcomes of the first technical study, this second ongoing study will deliver a consolidated calculation methodology of the SRI and will explore possible pathways for the implementation of the indicator on EU level.

The number of studies regarding the ‘European SRI’ methodology, test cases, and examples of applications is currently limited in the literature. Nevertheless, the published papers presented some considerations useful to highlight the potential and limitations of the proposed methodology, allowing to discuss and implement possible improvements in these last development stages and adaptations for the national application. Horák and Kabele [26] applied the SRI methodology to three residential buildings and to an educational building in Czech Republic. They concluded that the methodology presents some limitations, for instance the impossibility of properly accounting for combined heat sources, as well as an insufficient characterization of some impact scores such as ‘Health and Wellbeing’, which can easily reach the maximum points, because of the small number of included services. Markoska et al. [27] observed that assessment of buildings’ intelligence solutions typically requires extensive manual analyses and proposed an algorithm to automatize the process for the SRI calculation, based on the descriptions in metadata models in a Performance Testing (PTing) framework. As previously done by Volkov and Batov [28] regarding the application of the ‘Building Intelligence Quotient’ [19], they stressed the importance of input data. As the case study, they analyzed a university campus smart building in Odense, Denmark, and performed a sensitivity analysis assessing the impact of the number of considered services and weighting systems proposed by the ‘European SRI’ methodology. Janhunen et al. [29] implemented the SRI methodology [16], with the aim of discussing its suitability to northern Europe cold climates. Specifically, the authors analyzed two educational buildings of different construction periods and an office building in Helsinki, Finland. Janhunen et al. [29] concluded that the methodology presented some limitations in recognizing the specific features of cold climate buildings, in particular regarding advanced district heating systems. Consequently, the authors raised concerns on the feasibility of a homogeneous application in the different European Member states. Indeed, it was underlined how some steps of the procedure are characterized by choices that can be subjective, with significant impacts on the final result. In order to mitigate the effects of these aspects, Janhunen et al. [29] proposed and tested two alternative variants of the methodology. Similar considerations were expressed by Märzinger and Österreicher [19], who analyzed the ‘European SRI’ and observed that the methodology relies on potentially subjective expert judgment, it should be better applied within operational buildings, and the whole procedure could be time-consuming. As an alternative to the ‘European SRI’, the authors proposed a quantitative approach based on building load shifting potential and active interaction with the grids, distinguishing the electrical, the thermal, and the natural gas grid, and adopting an approach with similarities with the IEA Annex 67 approach [12,17].

As a whole, the analyzed contributions in the literature raised the attention to some critical points, in particular regarding the impact of input information and of subjective choices that can be made by the SRI assessors, which might affect the efficacy of the proposed indicator in promoting the adoption of smart solutions in buildings as planned by the European Union strategy. In order to discuss those aspects in further detail and to propose possible mitigation solutions, the latest available release of the ‘European SRI’ methodology [25] was in this research applied to a nearly zero-energy building (nZEB), the ‘Black Monolith’ at the NOI TechPark in Bolzano, Italy. The choice of a nZEB case-study is due to the fact that in Italy all new and renovated buildings have to be in compliance with nZEB criteria since 2021 [30] and this category of buildings generally presents a higher number of smart services and solutions compared to the rest of the existing building stock, allowing in such a way a deeper investigation of the ‘European SRI’ methodology. With the very aim of analyzing the impact of subjective assessment and interpretation of inputs, as well as to collect feedback from different assessors, a multi-step approach with independent working teams was adopted. Specifically, two expert groups were asked to twice apply the procedure for the SRI evaluation in the framework of a two-phase analysis, designed to explore and discuss the motivations of the different choices made during the assessment. In further detail:

• First step: to perform an independent assessment, each group referred to one of the two sets of information available for the building, respectively provided by different sources (i.e., facility management and system integrator);
• Second step: a cross-comparison of all collected information was carried out by the two groups to formulate a shared result.

The different SRI evaluations were examined and discussed, emphasizing the role played by input collection and availability and the most critical steps of the assessment which can bring to deviations in the characterization of the smart readiness of a building.

Stated the novelty of the methodology and the current lack of practical implementation from wide test campaigns, this paper provides an initial insight on the potential issues occurring during the application of the SRI evaluation. The involvement of two expert groups and the two-step analysis allowed to investigate the level of uncertainty of the SRI assessment and the difficulties in the evaluation of a complex case study. The discussed results represent relevant statements to identify more detailed specifications to be adopted in the further development of the methodology and within the future national implementation by the European Member States.

2. Methodology

This study was designed to evaluate the applicability of the currently proposed SRI methodology on a sample building. The SRI detailed calculation method was applied to a building case study by two teams of experts, in order to provide significant feedback on the feasibility of the proposed approach and outline possible improvements of the provisional method. Since the work focused on the aspect relating the practical application of the methodology from the assessor’s perspective, the implementation of customized weightings for the multicriteria evaluation was not developed in this study, but the standard ones reported in [25] were applied.

2.1. Methodological Framework

As explained before, the approach developed in [16,25] can be considered qualitative in nature (1A), dedicated to multiple performance domains (2), but essentially focused on the individual building system (3A), integrated with some criteria assessing the readiness of the building towards the grid. In particular, the European SRI methodology is multi-level and based on impact criteria, domain services, and functionality levels.

The first European SRI scheme [16] included eight main impact criteria (energy savings, flexibility to grid, self-generation, comfort, convenience, health, maintenance and fault prediction, information to occupants) and 10 domain services (heating, cooling, domestic hot water, controlled ventilation, lighting, dynamic building envelope, on-site renewable energy generation, demand side management, electric vehicle charging, monitoring and control). For each service, several functionality levels are defined, each representing a different degree of smartness. For instance, a functionality level 0 refers to a not-smart service implementation, while the highest level (which can vary from a minimum of 2 to a maximum of 5 depending on the service) indicates an advanced functionality, where the control is based on demand. Since the number of levels can vary, a direct comparison between the different services is not allowed. As an example, the service ‘heat emission control’ presents five different functionality levels from 0 to 4, accounting respectively for the configurations with ‘no automatic control’, ‘central automatic control’, ‘individual room control’, ‘individual room control with communication between controllers and to the building automation control system BACS’ or ‘individual room control with communication and presence control’. On the contrary, the service ‘thermal energy storage’ shows just three levels, from 0 to 2, corresponding to ‘continuous’, ‘time-scheduled’ or ‘load prediction based’ storage operation.

Each smart service can have several impacts on the occupants, the building itself and the grid, which are grouped according to the impact criteria mentioned above. However, some domain services can be irrelevant for some specific criteria and others can be mutually exclusive. Furthermore, the identification of relevant services can be performed through a triage process, establishing a priority scheme. Besides the services included in the domains already mentioned, an additional domain named ‘various’ contains services which are currently out of scope or insufficiently mature to be included. In total, Verbeke et al. [16] proposed a catalogue of 112 smart services, further reduced to 52 actionable smart-ready services.

In the second study for the definition of the European SRI for buildings [25], some simplifications have been introduced. The number of impact criteria has been reduced to 7, by including the ‘self-generation’ into the ‘energy savings’ category. As regards the domain services, they have been reduced to 9 by removing the ‘demand side management’ as an independent domain. An excerpt of the overall structure of the second European SRI service catalogue is presented in Figure 1.

Besides the different number of impact criteria and domains in [16] and [25], the procedure for the SRI assessment remains the same. First, an inventory of the smart ready services that are present in the analyzed building has to be performed using a simple check-list approach. Then, in order to cope with this multitude of domains and impact categories, the Smart Readiness Indicator SRI is calculated according to a multi-criteria assessment method.

Weightings can be attributed to the different functionality levels of each smart ready service, according to the climate conditions and the context of the building. Then, impact scores of the individual services can be summed up for each of the smart-ready domains and divided by theoretical maximum individual scores in order to obtain a domain impact score. For each impact criterion, a total impact score can be calculated as a weighted sum of the domain impact scores. The SRI score is then derived as a weighted sum of the total impact scores and it expresses how close (or far) the building is from its theoretical maximum smartness (i.e., 100%). Specifically, according to the default weighting system [25], for each of the ‘energy savings’, ‘maintenance and fault prediction’ and ‘energy demand flexibility’ impact criteria, all combined domains except ‘dynamic envelope’ and ‘monitoring and control’ are attributed an overall weight of 75%. As regards all other impact criteria, the sum of all domains except ‘monitoring and control’ is given an overall weight of 80%. Finally, considering the ‘monitoring and control’ domain, it always has a 20% weight, regardless of selected impact criterion. In addition to the overall SRI score, the sub-scores generated at both the domain level and the impact category level can also be communicated as part of the SRI. The assigned weights can be customized at national or regional level, in order to allow for flexibility in the different countries and territories of the European Union.

2.2. Case Building

The SRI assessment was completed for an office building located in Bolzano, in South Tyrol region in northern Italy. The case building, as shown in Figure 2, is a new construction called ‘Black Monolith’ and is part of the NOI TechPark, an intervention of urban requalification of a disused former aluminum factory site that obtained the LEED v4 Neighborhood Development Gold sustainability certification and the Class “A” according to the Klimahaus Certification, i.e., the local protocol for building energy labelling adopted in South Tyrol. The building houses a wide range of innovation companies, university research institutions and training centers in office spaces, seminar rooms, research laboratories, and common areas. The key characteristics of the building are shown in

Table 1. Key characteristics of the case building.

Name	Location	Year of Construction	Usage	Net Floor Area	Heated Net Floor Area	Heated Volume	S/V Ratio	Energy Performance
Black Monolith	Bolzano, Italy	2017	Office	10,252 m2	8509 m2	50,168 m3	0.26 m−1	nZEB

below.

The building owns a regular form consisting of two overlapping parallelepipeds and the façade is designed as a curtain wall composed of different layers to satisfy high energy requirements. The surface area to volume (S/V) ratio is 0.26 m⁻¹ and the average global transmission coefficient of the building envelope is 0.38 W m⁻² K⁻¹.

The ‘Black Monolith’ is conceived as a nearly zero-energy building (nZEB) according to the Italian law [30], with standard annual primary energy uses for space heating, sanitary hot water, space cooling, indoor lighting and auxiliary systems equal, respectively, to 99,850, 24,623, 48,304, 49,145, and 184,954 kWh. For the onsite renewable electricity generation, the building is equipped with a photovoltaic (PV) system covering the roof, with an overall capacity installed of 217.08 kW_p and a yearly energy production of 23,841 kWh.

Heating and cooling primary generation of the technological park is provided through different water to water heat pumps and three boilers are used as secondary heat generators. As a whole, the heating thermal capacity is 1802 kW while the cooling one is 1092 kW. The source side of heat pumps is connected to a water tank supplied by ground water. In the office spaces of the ‘Black Monolith’ building, radiant ceiling panels are used as heating and cooling systems, while several air handling units equipped with heat recovery systems are used to control fresh air, indoor air temperature, and humidity.

2.3. SRI Case Assessment

A beta testing of the SRI calculation approach was conducted on the ‘Black Monolith’ case study, by selecting, as assessment option, the detailed calculation method and applying the default weighting factors and ordinal scores for the multicriteria assessment framework, as proposed in the second technical support study [25]. The detailed method is based on the detailed service catalogue, it is applicable to all building types and the assessment is to be performed by a third-party qualified expert or through self-assessment by a non-independent expert [25].

For the public testing of the SRI, the technical support study provided an Excel-based calculation tool [25,31], requiring as input information:

• A description of the case building;
• The indication of the building systems available in the building with a triage process;
• The selection of the functionality levels for each smart ready service.

The SRI assessment took place in separated workgroups, organized in October 2019 and held in the case study building’s meeting rooms. The assessment for the case study building was performed by expert groups consisting of both technical building systems specialists and researchers. The technical building systems specialists were in charge to qualitatively check the availability of the smart ready services available in the detailed service catalogue and indicate the implemented functionality levels for the applicable smart ready services, while the researchers collected information and filled in the scores into an Excel-based calculation tool.

The evaluation process was subdivided in two working steps:

• Step 1

In the first step, two separate expert groups performed the SRI assessment of the building case study, with the objective being to check the influence of the assessors’ subjectivity and sources of information on the results. The composition of the two groups was constituted respectively by researchers of EURAC Research and Free University of Bozen-Bolzano (experts on the theme of smart buildings and working in offices based in the ‘Black Monolith’) and by different technical building systems specialists, as explained below.

The first team, expert group A, was composed by two researchers—the first and the second authors of the paper—interacting with the energy manager of the case study building. A workshop was organized by the researchers to analyze the smart ready service catalogue with the energy manager and together fill-in the Excel-based calculation tool. Based on a qualitative approach, the available services were selected in the triage process and the facility manager, given his high knowledge of the building, provided indications of the functionality levels scores for the applicable smart ready services for each domain, relying on the Building Management System (BMS). The desk meeting took approximately three hours.

The second team, expert group B, was composed by two researchers—the third and the fourth authors of the paper—collaborating with the provider of the building management and automation system of the ‘Black Monolith’. The researchers filled in the Excel calculation tool, based on the available technical documentation dealing with the automation and control logics of the energy building systems, as shared by the design team. The whole process lasted around five hours.

The members of the two expert groups presented an equivalent technical background (both groups were composed by engineers and architects), and the main difference was related to the source of information adopted for the SRI evaluation. Both groups collected a comprehensive documentation and information for SRI evaluation, but from different actors involved in the design and management of the building. This situation is representative for the assessment of complex constructions, where usually not all the actors involved in the design, construction and management could be contemporarily available to support the SRI evaluation.

• Step 2

In the second step, the two independent expert groups of the first step got together to compare the obtained results and work jointly to develop a shared outcome. The discussion took place in two meetings of two hours each, in which the differences in the available services of the building and in all the functionality levels have been pointed out.

3. Results

The results of the detailed evaluation of the building case study were achieved through the completion of the Excel-based tool. An overview of the completion choices and, accordingly, the results obtained in steps 1 and 2 are reported below.

3.1. Description of the Case Building Section

Regarding the first Excel-tool section about the description of the case building, there were no differences among the two expert groups in the choice between the available options for each required input information. In

Table 2. Excel-based calculation tool. Choice of input information (in bold in the table) to describe the case study building.

Input Information	Available Options
Building type	Residential; non-residential
Building usage	Single-family house; small multi-family house; large multi-family house; office; educational; healthcare; other
Climate zone	Northern Europe; Southern Europe; Western Europe; North-Eastern Europe; South-Eastern Europe
Net floor area of the building	<200 m2; 200–500 m2; 500–1000 m2; 1000–10,000 m2; 10,000–25,000 m2; >25,000 m2
Year of construction	<1960; 1960–1990; 1990–2010; >2010; not yet constructed
Building state	Original; renovated

, an overview of the input information is displayed and the groups’ selections are marked in bold.

As presented in the case building section (Section 2.2), the ‘Black Monolith’ is a non-residential office building, built in 2017, located in Bolzano (Italy, Southern Europe), and with a net floor area of 10,252 m².

3.2. Triage Process Section

In the second section of the Excel-based tool, a triage process was carried out to identify which services should be taken into account for the final score.

Table 3. Excel-based calculation tool. Choice of input information in the triage process (in bold in the table when assessed differently) to indicate the available building systems in the case study.

Domain	Input Information	Available Options	Step 1		Step 2
			Expert Group A	Expert Group B	Joint Evaluation
Heating	Emission type	Thermally activated building system; other hydronic system; non-hydronic system	Other hydronic system	Other hydronic system	Other hydronic system
	Production type	District heating; heat pump; central heating-combustion; central heating-other; decentral heating	Heat pump	Heat pump	Heat pump
	Thermal energy storage	Storage present; no storage present	Storage present	No storage present	No storage present
	Multiple heat generators	Single generator; multiple generators	Multiple generators	Multiple generators	Multiple generators
Domestic hot water	Production type	Non-electric; electric	Electric	Electric	Electric
	Storage present	Storage present; no storage present	No storage present	Storage present	Storage present
	Solar collector	Solar collector present; no solar collector present	No solar collector present	No solar collector present	No solar collector present
Cooling	Emission type	Thermally activated building system; other hydronic system; non-hydronic system	Other hydronic system	Other hydronic system	Other hydronic system
	Thermal energy storage	Storage present; no storage present	Storage present	No storage present	No storage present
	Multiple generators	Single generator; multiple generators	Multiple generators	Multiple generators	Multiple generators
Controlled ventilation	System type	Mechanical ventilation; controlled natural ventilation	Mechanical ventilation	Mechanical ventilation	Mechanical ventilation
	Heat recovery	Heat recovery; no heat recovery	Heat recovery	No heat recovery	Heat recovery
	Space heating	Used for space heating; not used for space heating	Used for space heating	Not used for space heating	Not used for space heating
	System sub-type	Combined air-water; all air; other	Combined air-water	-	-
Dynamic envelope	Movable shades, screens or blinds	Present; not present	Present	Not present	Not present
Electricity: renewables and storage	On-site renewable electricity generation	On-site renewable electricity generation; no on-site renewable electricity generation	On-site renewable electricity generation	On-site renewable electricity generation	On-site renewable electricity generation
	Storage of on-site generated renewable electricity	Storage present; no storage present	No storage present	No storage present	No storage present
	Combined heat and power (CHP)	CHP; no CHP	No CHP	No CHP	No CHP
Electric vehicle (EV) charging	On-site parking spots	On-site parking; no on-site parking	On-site parking	On-site parking	On-site parking
	Electric vehicle charging spots	EV charging; no EV charging	EV charging	EV charging	EV charging

reports the input information required and the available options, with the indication of the selected options reported for both step 1 by the two expert groups (expert group A and expert group B) and step 2 by the final joint evaluation team. The differences between groups are marked in bold.

In step 1, the main differences were related to the diverse interpretation of the services available in the building, in particular the availability of thermal energy storages for heating and cooling systems and the controlled ventilation functioning in terms of heat recovery and space heating. Unlike expert group B, expert group A considered as available the thermal energy storages for heating and cooling systems because they took into account the water tanks serving the heat pumps/chillers. Regarding the controlled ventilation functioning, group A verified with the energy manager the presence of heat recovery and use for space heating while group B could not find a technical scheme to prove the heat recovery option and evaluated the use of space heating as absent since it concerned only part of the building.

The evaluation of the dynamic envelope domain in the triage process raised discussion in both expert groups, since there was not a shared position whether the domain referred to a single option between presence of automated windows and presence of automated solar shadings or both technologies were required in the evaluation of this domain. The two groups confirmed the absence of automated shades or blinds—however, differently from expert group B, expert group A noticed with the energy manager the presence of some automated windows in the buildings and accordingly judged as present the services related to the dynamic envelope domain.

In step 2, the differences identified in the results of the two distinct assessments drove the joint discussion of the expert groups to outline a common evaluation. The first discordant point was the presence/absence of the storage systems. As a result of the debate, the two groups decided to consider the storage systems not present since the water tanks could be regarded as outside the boundaries of the heating systems and, in addition, they were not directly controlled by the heating system itself. For the service ‘Controlled ventilation’, expert group B agreed in evaluating the heat recovery available and taking it into account as a service. On the other hand, the use of ventilation for space heating was assessed as absent by expert group A as well, since it operated only in a portion of the building and did not cover the whole building area. In this regard, it would be crucial for the SRI methodology to clearly state the level of relevance needed to include a service in the assessment.

Similarly, concerning the presence of dynamic envelope, the two groups agreed in evaluating it as not present, since movable shades, screens and blinds were not installed and the available automated windows involved only part of the building and not the whole envelope. Table 3 shows the main information used in the triage process by the two expert groups, highlighting in bold the discrepancies in the data collections.

3.3. Selection of the Functionality Levels for Each Smart Ready Service Section

In the third section of the Excel-based tool, the assessors had to indicate the functionality level for each smart ready service available for the case study building, based on the selected options in the previous section of the triage process.

Table 4. Excel-based calculation tool. Choice of functionality levels for each smart service available in the case study building (in bold in the table when assessed differently).

Domain	Smart Ready Services	Functionality Levels
		Step 1		Step 2
		Expert Group A	Expert Group B	Joint Evaluation
Heating	Heat emission control	3	3	3
	Control of distribution fluid temperature	2	1	2
	Control of distribution pumps in networks	3	4	4
	Intermittent control of emission and/or distribution	2	3	2
	Thermal energy storage for building heating	1	-	-
	Building pre-heating control	1	1	1
	Heat generator control	2	2	2
	Heat system control according to external signal	1	0	1
	Sequencing of different heat generators	2	2	2
	Report information regarding heating system performance	3	2	2
Domestic hot water	Control of DHW storage charging	-	1	1
	Report information regarding domestic hot water performance	0	0	0
Cooling	Cooling emission control	3	3	3
	Control of distribution network chilled water temperature	2	1	1
	Control of distribution pumps in network	3	4	4
	Intermittent control of emission and/or distribution	2	3	3
	Interlock between heating and cooling control of emission and/or distribution	0	2	2
	Control thermal energy storage operation	1	-	-
	Generator control for cooling	2	2	2
	Sequencing of different cooling generators	0	1	1
	Report information regarding cooling system performance	3	2	2
Controlled ventilation	Supply air flow control at the room level	3	3	3
	Adjust the outdoor airflow or exhaust air rate	3	3	3
	Air flow or pressure control at the air handler level	3	4	4
	Heat recovery control: prevention of overheating	1	-	1
	Supply air temperature control	2	-	-
	Free cooling with mechanical ventilation system	2	3	3
	Reporting information regarding indoor air quality	1	1	1
Lighting	Occupancy control for indoor lighting	1	1	1
	Control artificial lighting power based on daylight levels	3	4	3
Dynamic envelope	Window solar shading control	0	0	0
	Window open/closed control combined with HVAC system	2	0	0
	Reporting information regarding performance	1	0	1
Electricity: renewables and storage	Reporting information regarding energy generation	2	2	2
	Storage of locally generated energy	0	0	0
	Optimizing self- consumption of locally generated energy	1	0	0
Electric vehicle (EV) charging	EV charging capacity	2	1	2
	EV charging grid balancing	0	0	0
	EV charging information and connectivity	0	0	0
Monitoring and control	Run time management of HVAC system	1	2	2
	Detecting faults of technical building systems and providing support to the diagnosis of these faults	1	1	1
	Occupancy detection: connected services	2	0	0
	Central reporting of technical building system performance and energy use	3	3	3
	Smart Grid integration	1	0	0
	Reporting information regarding Demand Side Management	1	0	0
	Override of Demand Side Management control	2	0	0

shows the list of available smart ready services with the functionality levels assigned in both step 1 by the two expert groups (expert group A and expert group B) and step 2 by the final joint evaluation team. The differences between groups are marked in bold.

In step 1, comparing the assignment of the smart services’ functionality levels of expert group A and expert group B, small differences were detected for a considerable number of smart services for all the domains, but the main differences were a consequence of the choices made during the triage process, i.e., concerning the thermal energy storages for heating and cooling systems, the controlled ventilation functioning in terms of heat recovery and space heating and the presence of dynamic envelope. Nevertheless, the most important inconsistency between the two groups came out for the ‘Monitoring and control’ domain, for the smart services related to the Smart Grid integration and the demand side management. The expert group B chose ‘zero’ as functionality level of these services, while the expert group A took into account the presence of a microgrid serving the NOI TechPark and enabling the net metering among buildings of the energy produced by the PV systems of the technological park.

In step 2, the functionality levels were determined according to the final choices done during the triage process by the joint evaluation group, based on the discussion and the cross check of the available documentation. The most significant differences in the selection of functionality levels in step 2, compared to step 1, are described in the following. In agreement with the final decision to consider the thermal energy storage systems for heating and cooling domains as absent, a ‘zero’ functionality level was assigned to the related smart ready services. Conversely, the storage for domestic hot water was evaluated as present and thus a functionality level of ‘1′ was finally attributed to the related smart ready service. Considering the ‘Controlled ventilation’ domain, heat recovery was considered as available and a functionality level of ‘1′ was given to the related smart ready service, while the use of ventilation for space heating was assessed as absent in the joint evaluation and thus the smart ready service of the ‘Supply air temperature control’ was no more available. Regarding the smart services referring to smart integration and Demand Side Management in the ‘Monitoring and control’ domain, the two groups discussed the possible general meaning to attribute to the smart grid concept and, given the absence of an energy management from an external aggregator, it was not possible to shift energy consumption of the building in response to price signals or higher availability of renewables in the national energy mix. Therefore, despite the potentials of the building, the assessors decided to assign to these services the null functionality level. Table 4 reports the functionality levels assigned in step 1 and step 2 by the two expert groups, highlighting in bold the discrepancies in the evaluation and the most critical domains.

3.4. Results Section

In the final Excel-tool section, the total Smart Readiness Indicator score and the bar charts of the impact scores for each impact criteria were displayed. The impact criteria scores, obtained in step 1 by expert group A and expert group B and in step 2 by the joint evaluation team, are shown in Figure 3.

As a result of the independent assessment in step 1, the two expert groups obtained a quite different SRI final score, respectively, of 66% for the expert group A and 53% for the expert group B. The reasons of this discrepancy were obviously a consequence of the different choices performed in both the triage process and the attribution of the smart functionality levels to the building smart ready services. The results were quite similar for the criteria ‘Energy saving on site’, ‘Comfort’ and ‘Wellbeing and health’ while a small difference was noticed for the criterion ‘Convenience’. The main diversity arose for the criterion ‘Flexibility for the grid and storage’ (44% for expert group A and 7% for expert group B) followed by the criteria ‘Maintenance and fault prediction’ (59% for expert group A and 39% for expert group B) and ‘Information to occupants’ (56% for expert group A and 41% for expert group B).

As a result of the analysis in step 2, the total SRI score was 61%. Looking in Figure 3 at the updated impact criteria scores obtained in the common evaluation, it is visible that the adjustments led for each impact criterion to obtain a score that is nearly the average value between the values obtained in the separated assessments of the two expert groups in the previous step.

4. Discussion

This paper examined the applicability of the SRI for buildings calculation methodology, focusing on the whole evaluation process, from the technical information collection, and the triage process of the available services towards the result assessment.

Considering the starting phase of the evaluation, the first issue that can be pointed out was the problem to retrieve all the technical documentation needed as initial input for the detailed assessment of the case study. The complex nature of the building, in terms of equipment, stakeholders involved in the design and management, variety of functions and available technical building systems resulted in a lack of a unique reference person to interact for the data collection. Consequently, the analysis during step 1 highlighted that the source of information and its interpretation can highly affect the evaluation, based on one side on the technical suggestions of the energy manager (expert group A), and on the other side on the control schemes of the building automation (expert group B). The analysis of this paper demonstrates that the source of data has a significant impact on the accuracy of the evaluation. Therefore, the authors recognize the importance of defining a source hierarchy with the indication of the preferable figures to be involved in the data collection (e.g., designers, facility managers, smart system providers). Accordingly, there would be a unique source selection process by the assessors, reducing the uncertainties in the results.

The analysis of the expert panels showed a significant impact of the assessor’s interpretation on the SRI evaluation, especially in the process of selecting the relevant building services and the relative functionality levels. One of the possible reasons is the lack of a guidance document including an exhaustive description of the smart services installed by different providers. Moreover, it is crucial to clearly state how to translate each installed technology in a specific service and choose the proper functionality level, in order to achieve a univocal evaluation. Considering the difficulty in defining a European-wide guidance with all the possible building elements and Heating, Ventilation, Air Conditioning (HVAC) systems technologies, it is recommended to let the Member State to define National guidelines tailored to the specific contexts. The guidelines need to include the definition of the functionality levels for the most applied technologies, considered as essential reference for reducing the level of interpretation and the impact on the results.

Another difficulty that occurred within the evaluation was the lack of a shared position on how to take into account specific services or smarter functionality levels when they were available only in a portion of the building surface or when they were installed but not used in everyday operation. In this regard, it would be important to better clarify in the procedure how to consider a service relevant for the building smartness, if according to (i) the share of building floor area covered by the technology respect to the overall one or (ii) the energy use of the interested areas or (iii) the number of users interested by the service.

Looking at the results’ interpretation, the obtained impact scores for each criterion were more interesting to understand the smartness of the building rather than the whole SRI score. To support the proper interpretation of the results, it would be important to introduce a classification of the assessed score for each criterion, in order to enable the stakeholders in making decisions for improving the smartness level where it is more needed and convenient.

The current methodology enables the comparison only among buildings with similar smart ready technologies installed and with the same available domains. Therefore, the definition of performance benchmarks and targets for comparing the results of different buildings and different design variants seems a complex issue to be addressed. Nevertheless, having a clear statement of benchmarks and acceptable smartness level will ease the stakeholder interpretation of the results, and the relevance of the SRI at broad level.

To increase the relevance of the results, the criteria need to be defined accurately and, although the SRI is based on a qualitative evaluation, each criterion should be associated with a measurable parameter that should be monitored and assessed during the building operation, enabling, as an additional effect, the comparison of the performances of smart buildings. For example, the criteria ‘Comfort’ and ‘Wellbeing and health’ could be misled by the final user. In this regard, the indication of physical quantities such as the number of hours of thermal discomfort and indicators of indoor air quality such as the CO₂ concentration measured in sample rooms can clarify the result interpretation. Another example is the criterion ‘Convenience’ which, as currently defined, could not provide a clear message to the stakeholders. In addition to this case, the definition of indicators such as the yearly costs (including energy supply and maintenance costs) to be assessed during the building operation, represent an interesting information to be coupled with the SRI.

The authors recognize that the Smart Readiness Indicator should mainly target the key stakeholders in future energy distribution systems, namely the grid operators, aggregators and energy prosumers. In this regard, the criterion ‘Energy flexibility and storage’ includes information that is becoming crucial for the energy supply. Therefore, the evaluation of the building smartness need to be associated with the assessment of energy flexibility, considered as the amount of energy that can be offered by the building to handle loads. On the one side, this information will support the grid operators and the energy aggregators to provide sustainable solutions for better controlling the fluctuations of energy demand. On the other side, according to the flexibility that a building can offer, it would be possible to provide cost-effective energy supply solutions following the availability of renewables and enabling to improve the cost-effectiveness for the energy consumers. Therefore, the authors propose to couple to the SRI a quantitative indicator on how much energy flexibility can be offered by the building, as a key information for the future energy system.

5. Conclusions

This paper presents the practical application of the new methodology for evaluating the SRI, discussing the feasibility of implementation and the obtained results. The SRI methodological framework was applied to a case study, a nZEB office building in the Italian city of Bolzano, and the evaluation was carried out by two expert groups composed of researchers and technical building systems specialists. The two-step evaluation allowed for a detailed discussion between the experts and for highlighting several issues that an assessor can face during the SRI assessment and that can affect the results.

Although the application is focused on a specific sample case, the discussed results and outcomes of this experience can withstand the future development work of the SRI technical support study and the implementation at national level.

One of the main outcomes of the analysis is the importance of the data collection, that is significantly affected by the available source of information. The data collection has a direct impact on the definition of functionality levels of the services, which is the core of the SRI evaluation. In fact, it drives the interpretation leading to subjective decisions and non-unique results by different assessors, as observed for the two expert groups during the first step of this analysis. To cope with the issue, detailed national guidelines including a source hierarchy for the data collection and a comprehensive list of technologies with the relative functionality levels is crucial for an effective implementation. Moreover, there is the need to set-up a specific documentation on the technical systems and building automation devices that should be assigned to a specific technical figure (e.g., the facility manager of the building), in order to ease the assessment process and to reduce the level of uncertainty.

The analysis of the results from the SRI assessment highlights that comparing the smart readiness of different buildings or between design variants is feasible only in presence of the same number of smart services implemented and domains addressed. This issue can limit the relevance of the SRI application, since setting clear performance targets and reference benchmarks will be difficult.

Moreover, the adopted qualitative manual service-assessment approach limits the opportunity to consider the smartness in terms of performance-based and measurable indicators. This is mainly relevant for the energy flexibility that can be offered by a smart building, representing one of the key features becoming crucial in the future energy system.

Coupled with quantitative indicators to be assessed and monitored, the SRI methodology could become relevant not only as an evaluation of building smartness but can be adopted during different phases of the life cycle. In particular, both for the design phase as a decision-making support for comparing different technical solutions, and during the operation phase as a reference to identify possible domains where an increased smartness would be needed according to the building monitored performance.

To conclude, the qualitative nature of the SRI assessment ensures a more affordable calculation of the indicator. However, as introduced in the EPBD Directive, the smartness should be evaluated in terms of adapting the operation of buildings to the needs of the occupants and the grid and improving the building energy efficiency and overall performance. Therefore, in this perspective, coupling to the qualitative evaluation a quantitative indicator as well will represent a crucial issue.

As a future outlook of the research, it would be valuable to apply the detailed calculation to a wider sample of buildings, including different sizes and building typologies, in order to further test the SRI methodology and to generalize the results of the present paper. In addition, it would be relevant to validate, on real case studies, how to couple the SRI assessment with quantitative indicators for assessing the building performance in terms of energy flexibility and with monitored performance during the operation. This would be a support for a wide range of stakeholders, also enabling a clearer analysis of the benefits given by building smartness, in terms of monitored data dealing with the impact criteria of SRI evaluation.