A Review of the Evaluation of Building Energy Code Compliance in the United States

Abstract

Building energy codes are essential tools for achieving energy efficiency in buildings. However, the full energy savings potential of these codes can only be realized if buildings are constructed in compliance with them. Therefore, evaluating building energy code compliance is crucial in bridging the gap between the energy efficiency requirements set by energy codes and the actualized energy savings achieved. An energy code compliance evaluation serves as a mechanism to assess construction practices, evaluate the effectiveness of code enforcement, identify gaps in compliance, and guide strategies for improvement through training and education. Conducting code compliance evaluation activities involves field studies that require careful design and significant resources. Historically, more emphasis has been placed on developing and adopting building energy codes, while efforts to evaluate compliance have been relatively limited and lacking consistent approaches. The passage of the 2009 American Recovery and Reinvestment Act (ARRA), which mandated that states create plans for achieving 90% compliance within eight years, stimulated the need for an energy code compliance evaluation. As a result, federal, state, and local governments, and utilities have invested in the development of methodologies and tools for code compliance evaluation studies. This paper reviews the code compliance evaluation studies conducted in the United States over the past three decades. It describes and compares the methodologies and metrics used to assess building energy code compliance, summarizes the general elements and steps involved in the evaluation process, and discusses common issues in these studies. Over time, code compliance evaluation methodologies have evolved from isolated development within individual states, regions, and utilities, to widely accepted protocols applicable across different states and local jurisdictions. There has been a transition in compliance metrics, shifting from historical compliance rates to energy-consumption-oriented approaches.

1. Introduction

In 2021, buildings accounted for 30% of global energy consumption and 27% of total energy sector emissions [1]. According to the Energy Information Administration (EIA), buildings in the U.S. consume 75% of the electricity and 40% of the total energy, resulting in 36% of all carbon dioxide emissions [2]. Unfortunately, a significant amount of energy in buildings is wasted due to outdated construction practices, and inadequate system controls [3]. All these factors make buildings an essential target for cutting energy waste and emissions. Building energy codes, which govern building construction to meet minimum energy requirements, are recognized as one of the highly cost-effective means of reducing energy consumption and greenhouse gas emissions.

Broad energy efficiency regulation emerged in the 1970s in response to concerns about energy security following the oil embargo. Building energy codes were developed to improve performance in areas such as building envelope, lighting, mechanical systems, and other building components. The two main building energy code systems in the U.S. are developed by American Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE), in conjunction with the Illuminating Engineering Society of North America, and the International Energy Conservation Code (IECC), respectively. The U.S. Department of Energy (DOE) is required by federal statute to issue a determination as to whether the latest edition of ASHRAE Standard 90.1 (for commercial and multi-family high-rise residential buildings) or the latest edition of the IECC (for low-rise residential buildings) will improve energy efficiency compared to the previous edition of the corresponding standard or code. Once an affirmative determination is published, states are required to certify that they have reviewed the provisions of their commercial and residential building code regarding energy efficiency and decide whether to update their codes to meet or exceed the updated edition of ASHRAE and IECC. It should be noted that building energy codes are implemented at the state level or, in home-rule states, at the local jurisdiction. The periodically updated ASHRAE standard and IECC code serve as model codes until cities, states, and other jurisdictions adopt them.

Energy efficiency measures regulated by building energy codes can lead to significant energy savings and limit the building sector’s overall contribution to global carbon emissions. Aroonruengsawat et al. [4] constructed a timeline of when individual states first implemented residential energy codes and developed an empirical model to assess the impact of these codes on residential electricity consumption. They concluded that residential energy codes reduced electricity consumption by 3–5% in the year 2006. In addition to energy savings and greenhouse gas mitigation, the adoption and implementation of building energy codes can preserve scarce natural resources, contribute to the security of the national energy supply, reduce local greenhouse gas emissions, mitigate the impacts of adverse events, and improve energy resilience [3,5,6]. In a more recent impact analysis of building energy codes in the U.S. [7], it was projected that energy codes for residential and commercial buildings would save a cumulative $138 billion of energy from year 2010 to 2040, reduce 900 million metric tons (MMT) of avoided CO₂ emissions, and save 13.5 quads of primary energy.

The anticipated energy savings from the adoption of more stringent building energy codes are potential benefits and can only be realized if the buildings are constructed to comply with the energy codes. The evaluation of building energy code compliance is critical to understanding the current state of compliance and identifying specific areas of non-compliance. This information can inform states and jurisdictions about the need for training or attention from builders, architects, and code officials to improve future code compliance. Code evaluation studies serve as a quality assurance and quality control mechanism for the code compliance process.

Historically, more effort has been placed on the development and adoption of building energy codes, while resources for training and code compliance enforcement have been lacking. Evaluation efforts to assess building energy code compliance have been rare. Although there were some activities to evaluate building energy code compliance in certain states or geographic region prior to the 2009 American Recovery and Reinvestment Act (ARRA), there was lack of nationwide awareness regarding the importance of code compliance evaluation and the need for consistent and reproducible methodologies.

The 2009 ARRA stimulated the demand for energy code compliance evaluation. It was the first time that states faced building energy code compliance requirements from the federal government. To receive the 2009 ARRA stimulus funds, states were obligated to ensure that their adopted energy codes met the stringency requirements of the 2009 edition of the IECC for residential buildings or the ASHRAE 90.1-2007 edition for commercial buildings. Additionally, states were required to establish plans to achieve 90% energy code compliance for buildings constructed within eight years.

To assist states in meeting the 2009 ARRA requirement, the U.S. DOE, through its Building Energy Codes Program (BECP), developed a code compliance evaluation methodology, protocol, and tool [8] (BECP 2010 methodology). As the first methodology of this kind, the BECP 2010 methodology was widely adopted and tested in many states across the U.S., leading to valuable lessons learned and reported recommendations from those studies [9,10].

One of the main limitations of the BECP 2010 methodology is its use of the compliance rate as a metric, which is based on a binary pass or fail and did not correlate with the energy consumption of the buildings. In 2014 and 2016, the BECP issued a Funding Opportunity Announcement (FOA) for field studies focused on code compliance improvement for new residential [11] and new commercial construction [12], respectively. These FOAs aimed to develop, test, and finalize the compliance evaluation methodology, protocols, and tools that could serve as templates for state and local jurisdictions to adopt and tailor for their specific code compliance evaluation purpose.

Most code compliance evaluation studies in the U.S. have been organized by utilities, states, regional energy efficiency organizations, or local jurisdictions with support from the federal government. The results of these studies have primarily been presented as internal technical reports by the entities conducting the studies, making them less accessible as articles in peer-reviewed journals. While earlier code compliance studies have been reviewed, there is a lack of reviews on the most recently developed methodologies. This paper aims to bridge the gap between the academic research community and the code compliance evaluations conducted by government code programs and energy efficiency organizations.

The authors conducted a comprehensive review of code compliance evaluation studies in the U.S. over the past three decades, focusing on several key aspects including building types (residential vs. non-residential), sample design (population specification and selection), data collection and evaluation methods, metrics used in the analyses, and reported compliance results. The emphasis of the review is on examining the methodologies, procedures, steps, and metrics evolutions, rather than solely focusing on the actual compliance results.

This paper begins with an overview of building energy code compliance evaluation in Section 2, and then proceeds to review the individual activities of building energy code compliance evaluation in Section 3. The review of individual code compliance evaluation activities is presented in chronological order based on when the studies were conducted in relation to the 2009 ARRA. Section 4 discusses common components implemented across code compliance evaluation studies, the differences employed in these studies, and the progression of code compliance methodologies over time. Finally, the paper concludes with key findings and suggestions for future code compliance evaluation studies in Section 5.

2. Overview of Building Energy Code Compliance Evaluation

Code compliance evaluation is different from regular code compliance, although they may share common steps that determine whether a building is code-compliant or not. Code compliance checks are part of the code enforcement procedures and aim to ensure that an individual building under construction meet the code requirements. On the other hand, code compliance evaluation aims to assess the overall level of code compliance in a state or local jurisdiction. It involves examining a representative sample of the building stock to identify major gaps in code compliance. The results of the evaluation are used to help decision makers prioritize areas for compliance improvement [9]. Code compliance evaluation typically involves the use of statistical sampling and analysis methods.

Smith and Nadel [13] conducted a literature review of studies related to building energy code compliance and enforcement. At the time of the research, only a small number of states had completed energy code compliance studies. The review revealed that the energy code was perceived more as guidance rather than an absolute standard, such as life and safety codes. As a result, some leniency in enforcement was considered acceptable.

To evaluate the performance of utility residential new construction programs, Vine [14] examined six studies of building compliance in two regions: the Pacific Northwest and California. These studies involved three states: California, Oregon, and Washington. The findings indicated that homes participating in utility sponsored residential new construction programs demonstrated higher levels of building compliance compared to nonparticipating homes. Additionally, the study highlighted the importance of the definition of compliance. While overall building compliance was low, compliance with individual building components was higher. Interestingly, some studies showed that compliance measured on performance factors, such as heat loss rate, was higher than compliance measured based on prescriptive requirements.

Yang et al. [15] conducted a review of residential energy code evaluations carried out in 16 states and commercial energy code evaluations in 7 states. The emphasis of their review was on residential buildings due to their relatively lower complexity and the cost and time advantages associated with conducting evaluations for residential energy codes.

Among their findings and recommendations, it was observed that on-site evaluations were expensive, which posed limitations for the sampling methodology and data collection efforts. Therefore, they advocated for the development of a standardized approach that could measure real-world energy performance in a statistically rigorous manner while also reducing costs.

The American Council for an Energy-Efficient Economy (ACEEE) conducted a survey among state officials responsible for building energy codes. The purpose of the survey was to determine whether states had conducted studies on the status of code compliance. Responses were received from 34 state energy officials and other individuals involved in energy codes across 31 states. Misuriello et al. [16] reviewed the survey and reported the findings.

The review revealed that the majority of state energy code compliance efforts were focused on meeting the expected federal code compliance performance goals. Several common issues were identified, including discrepancies between as-built conditions and original plans, occurrences of non-compliant product substitution during construction, and the need for stronger training and education efforts to improve code compliance. Additionally, it was noted that the methods used for code compliance evaluation varied widely, lacking uniformity, which made it difficult to compare compliance studies across states. In light of these findings, there is a need for the development of standard methods for collecting, analyzing, and reporting data in order to facilitate meaningful comparisons of compliance rates between states and over time.

Stellberg [17] conducted an examination of 45 statewide and regional compliance evaluation studies. The purpose of the study was to provide an overview of the state of non-compliance with energy codes and estimate the energy savings potential at the state level if non-compliant buildings were brought up to code requirements. Despite the existing knowledge and data gaps regarding code compliance rates, the study assessed each state’s energy savings potential based on a range of hypothetical baseline compliance rates. Assumptions were made regarding future construction levels, code adoption, energy demand, and prices.

The employment of the BECP 2010 methodology in the code compliance evaluations was discussed in both the review by Misuriello et al. [16] and Stellberg [17]. The standardized protocol of the BECP 2010 methodology ensure consistency and uniformity in data collection, analysis, and reporting, allowing for a comparison of results across different locations and time periods. However, the compliance rate metric used in the BECP 2010 methodology, which is based on a binary fail/pass classification, does not effectively capture the energy savings potential associated with increased code compliance. In response to this limitation, the BECP conducted FOA field studies in 2014 with the aim of improving the code compliance methodology for residential new construction [18].

Following the methodology improvement achieved through the residential FOA studies for residential code compliance evaluation, the BECP also focused on enhancing the methodology for commercial code compliance evaluation [19]. Bartlett et al. [20] conducted a review of commercial compliance evaluation studies conducted in the past two decades of 2016, summarizing the lessons learned from these earlier studies. The findings from these reviews informed the development of methodology improvements for commercial code compliance evaluation [20].

3. Activities of the Evaluation of Building Energy Code Compliance

3.1. Prior to ARRA

3.1.1. Commercial Code Compliance Evaluation

In April 1991, Ecotope conducted an evaluation of commercial energy code compliance in Washington and Oregon. The study had four main objectives: (1) to characterize building activities and current construction practices by selecting a random sample of new commercial buildings, (2) to assess code compliance in this sample by reviewing building documents, conducting site visits, and comparing the results with code requirements, (3) to interview design professionals and building code officials to determine attitudes toward energy codes and identify perceived training needs, and (4) to review the energy code and suggest changes that could enhance compliance [21,22,23]. The survey instruments and methodology for the commercial building energy code compliance evaluation in Washington and Oregon were developed through a prior pilot study [24].

The methodology developed during the pilot study [24] consists of three stages: sample design, data collection, and compliance assessment, along with conducting interviews with design professionals and code officials. The Dodge database [25] was utilized for developing the sample frame, which refers to the population or set of buildings from which a representative sample is selected for analysis. The sample frame plays a crucial role in the study as it ensures that the sample accurately represents the larger population of buildings relevant to the energy code being evaluated. Buildings included in the final samples were those for which obtainable building documents were available, permission was granted for a site visit, and an interview with at least one member of the design team could be conducted. In total, 70 buildings (out of a sample frame with 468 buildings) in Washington and 71 buildings (out of a sample frame with 213 buildings) in Oregon, permitted in 1990, were evaluated [22,23]. It should be noted that the criteria employed for sample selection did not ensure randomness, which introduced self-selection bias and affected the representativeness of the sample.

The data collection process involved building plan reviews and field visits to verify whether the building components specified in the plans and specifications were installed in compliance with the energy code. Energy code compliance was assessed for three building systems: the building envelope, mechanical system, and lighting system. Compliance rates were determined for each building system type as well as for the whole building. The study revealed that there was no direct correlation between code compliance rate and the energy efficiency of the buildings [23].

The energy impact of non-compliant buildings identified from the code compliance studies conducted in Washington and Oregon [22,23,26] were assessed using Bonneville Power Administration new commercial building prototypes and DOE-2 [27], a building energy simulation tool [26]. Energy simulations were conducted for typical non-compliant building prototypes and compared to the same prototypes adjusted to comply with the energy code. The difference in energy used between the compliant and non-compliant building simulations represented the savings potential that would result from energy code compliance.

The findings from the energy code evaluation studies led to the revision of the Washington State Nonresidential Energy Code (NREC) through simplification. The revised NREC was adopted in 1994, accompanied by a Code Implementation Plan developed by the Utility Code Group, which was formed and funded by the state utilities [28]. In order to assess the compliance rates of the revised 1994 NREC code and the effectiveness of the Code Implementation Plan, an evaluation study was initialized in 1995, employing the same methodology used in the previous 1991 evaluation study [22] to allow for accurate comparisons. A total of 88 buildings were sampled from the sample frame, which consisted of 792 buildings permitted in 1995 [28,29,30]. The sampling approach used a three-stratum stratified sampling, as opposed to the two-stratum stratified sampling used in the previous 1991 study [22,23]. The Dalenius–Hodges stratification technology with a Neyman allocation [31] was employed to ensure an accurate characterization of building types. This procedure aimed to equalize the square footage of the predicted area across the strata, resulting in the best stratification design based on the cumulative distribution. Sample weights were derived and applied to each stratum to reflect the sampling probability represented by each building in the stratum.

In both evaluation studies, interviews with builders/designers and code officials were included as a critical step in the methodology, aiming to determine who was responsible for energy code compliance and to gauge the attitude toward the energy codes and their enforcement. In the 1991 code compliance study [22,23], the majority of professionals interviewed overwhelmingly indicated that the energy code was not enforced, not relevant, or was simply an irritation that could be circumvented with little effort. However, in the 1995 study [28,29], the general response and attitude toward the energy code were more accepting. The results also indicated that inspections were the critical weak link in the enforcement process [28]. Therefore, the recommended revision to the 1994 NREC were aimed at making the regulations more enforceable and easily understood by the building and design communities. The wide adoption of the prescriptive compliance path was due to its provision of an easier and enforceable avenue for builders and code officials [32].

It was discovered from the 1991 and 1995 studies in the state of Orgon and Washington that the code compliance rates increased only marginally. However, the energy savings potential that would have resulted from full compliance were much more dramatic between these two samples [29,32].

Baylon [29] reviewed three commercial energy code compliance studies of the Washington State Energy Code (WSEC), including an earlier sample of 25 large office buildings permitted in 1988, and the two random samples permitted in 1991 and in 1995 as described above. He concluded that the shift in building characteristics due to the impact of the WSEC has been both striking and reassuring, indicating the viability of using energy code and enforcement as a market transformation mechanism. The Implementation Plan was the first of its kind in the country. Caution is warranted when comparing the results from different code compliance evaluation studies, even if they occurred in the same states. For example, in the Washington code compliance evaluation studies, the 1991 sample [22] was dominated by offices and schools, while the 1995 sample [28] was dominated by retail buildings and warehouses.

In 2001, the Northwest Energy Efficiency Alliance (NEEA) funded a 16-month study [33] to collect detailed information about the standard practices and attitudes of the building and design community in each of its member states (Idaho, Montana, Oregon, and Washington). The methodology developed and applied in previous study was used. The study collected data on 144 samples from a sample frame of 1179 buildings from Oregon, Idaho, and Montana, which were combined with the 88 samples collected in Washington in the previous 1995 study [29]. The same sampling and analysis methodologies allowed the comparison of results among the states in the Pacific Northwest Region. The study revealed a considerable regional technology shift, such as the dominance of low-emissivity and tinted windows, the prevalence of T-8 fluorescent lamps with electronic ballasts and compact fluorescent or high-intensity discharge downlights, and the use of adjustable speed drives to control the fan motors associated with variable air flow. It also exposed differences in attitude towards energy efficiency among the four states in the region. For example, it was found that less than 10% of architects and engineers interviewed in Idaho and Montana had any contact with the energy code or energy code officials, in contrast to Oregon and Washington, where almost 50% had direct feedback from code officials and/or had modified designs to accommodate energy code requirements.

In 2006–2007, a two-year study conducted by NEEA [34] aimed to improve the understanding and characterization of the new commercial building stock in the Pacific Northwest Region. This study focused on buildings constructed between 2002 and 2004, providing a new regional baseline for current practices in commercial buildings. While the primary goal of the study was not to evaluate compliance with a specific energy code, it aimed to assess energy use intensity (EUI) in the buildings. To ensure utility billing information was available, the sample frame included buildings that had been occupied for at least one year. The buildings in the sample frame were permitted under different energy codes. For the purpose of code compliance evaluation, a single prevalent energy code in each state was selected to assess code compliance [34]. Since the study focused on occupied buildings, certain building features (e.g., insulation levels) that were easily observed in previous code compliance studies were not directly observable after occupancy. To address this, the study relied on information from construction documentation or specifications to account for unobservable data points. Previous studies indicated that building components did not differ significantly from plans and specification. It is importance to note that field studies rely on objective direct observation and attempts to substitute data should be avoided to maintain the impartiality and accuracy of direct observations.

Although random sample selection was one part of the sample design in previous studies, there was no target sample size determined based on statistical criteria. However, in this study, the sample size was determined based on a 90% confidence interval with a 10% significance level [34].

Out of the 346 buildings that underwent site visits, billing data was collected for 190 of them, and end-use EUI of the buildings was calculated [35]. The calculation of EUIs was performed through the analysis of the billing data using the EZ Sim software [36]. The calculated EUIs were compared to data from various sources, including the Commercial Building Stock Assessment (CBSA) database of the Northwest U.S. [37], the Commercial Building Energy Consumption Survey (CBECS) database from EIA [38], and the California Commercial End-Use Survey (CEUS) database of California [39].

Khawaja et al. [40] conducted a study on statewide energy codes and standards adoption and non-compliance rates for Southern California Edison (SCE) based on permits filed between 1 November 2005, and June 2006 in the state of California. The study focused on evaluating five specific building measures and several appliances. Unlike other studies that assessed compliance at different levels, such as building components, building systems, and whole buildings, this study employed a compliance scoring system that categorized compliance into three distinct categories: process, design, and field. Additionally, the study introduced the concept of partial compliance, and Bayesian statistical analysis was utilized to update the initial score from document review based on the site visit score.

The Britt/Makela group conducted an energy code compliance assessment study in the state of Indiana [41]. Due to the extended duration of the study, the code compliance assessment, originally intended to compare against the 2000 IECCC, was modified to be compared against the 2003 IECC instead. Plan reviews and on-site field inspections were performed on 55 new commercial buildings. The plan review was carried out in two stages, beginning with an initial plan review to record building data, followed by a second review to ensure completeness. The field inspections focused on reviewing building data that could not be obtained from the plans, such as missing insulation levels, glazing type, and light fixtures, as well as collecting information that must be obtained in the field, such as air sealing and duct sealing.

3.1.2. Residential Code Compliance Evaluation

In 1994, the state of Oregon conducted evaluations to assess compliance with the Oregon Residential Energy Code (OREC) [42]. The field study protocol was developed following a pilot study involving 21 buildings [43]. The comprehensive study included 283 homes that were permitted in Oregon during 1993 and early 1994 [42]. The sample size for the study was determined based on an estimated 14,000 residential permits issued per year in Oregon. The desired confidence interval was set at 95% for an assumed binomial distribution, with an assumed compliance level of 75% across the state.

In addition to evaluating energy code compliance with the OREC, the study also assessed the energy implications of non-compliance by utilizing computer simulation to predict heating energy consumption. The methodology employed in this aspect of the study followed similar steps to those established and used in commercial energy code compliance studies conducted in the Northwest region of the United States [22,23].

Constructing a sample frame proved to be a challenging task in the code compliance evaluation study. Many jurisdictions in Oregon did not have a comprehensive record of buildings under construction. The absence of a computerized building tracking system in some jurisdictions further complicated the identification of homes available for inspection. Additionally, the random sample selection process had to be customized to suit the requirement of the field review process. Consequently, the sample frame did not encompass all permitted homes but rather focused on homes that were accessible during the scheduled field survey time periods.

In the second Market Progress Evaluation Report, which aimed to track the advancements achieved in NEEA’s Codes and Standards Projects from 2005 to 2007, Seiden et al. [44] performed an energy code compliance analysis. The analysis involved a comprehensive review of previously funded residential studies by NEEA and assessed compliance distributions on state, housing type, and building components. The finding of the analysis revealed an overall compliance rate of 85%. As a result, recommendations were made to expand the training program and conduct additional field studies in specific regions, building types, and technologies where data indicated lower compliance.

The Phase 1 evaluation report of the Efficiency Vermont’s Residential New Construction Program provided a detailed overview of the residential new construction market in Vermont, as well as an assessment of the program’s achievements during the period March 2000 to November 2002 [45]. The report offered a comprehensive description of the program’s activities, accomplishments, and their impact on the residential construction sector in Vermont.

Estimating the number of new single-family homes constructed in Vermont within a one-year timeframe posed a significant challenge. Various data sources, including Vermont Department of Property Tax Valuation data, the U.S. Census Survey data, Grand List data collected by towns in Vermont for property tax purposes, and builder survey data, produced inconsistent results. The selection of one dataset over another was not driven by any significant reason related to data quality or methodology. In the code compliance evaluation studies, it was not uncommon to compile and consolidate multiple data sources during the preparation of the sample frame.

The methodology for baseline characterization in this study [45] involved various data collection methods. These included gathering data from builders regarding their recent and ongoing construction projects, conducting a survey of customers who had purchased new homes in 1999 and 2000, and conducting an on-site survey of 159 newly constructed homes. By employing these diverse data collection approaches, the study aimed to obtain a comprehensive understanding of the baseline characteristics of the residential new constructions in Vermont.

3.2. ARRA

3.2.1. Methodology Developed by BECP

The 2009 American Recovery and Reinvestment Act played a pivotal role in incentivizing a significant nationwide initiative to assess compliance rates with building energy codes across the United States. By measuring compliance rates with building energy codes, the ARRA aimed to promote energy conservation, reduce carbon emissions, and foster sustainable development practices to address the pressing energy and environmental issues faced by the nation.

According to the BECP 2010 methodology [8], a minimum sample size of 44 was derived to achieve a 90% compliance rate with a one-side 95% confidence interval. This sample size calculation was based on an assumed coefficient of variation (CV) for each of the four building construction types: new residential, new commercial, residential renovations, and commercial renovations. The goal was to ensure sufficient data to access compliance rates accurately within these construction categories.

In line with the BECP 2010 methodology, it was recommended to estimate building populations using relevant database. For commercial buildings, the most recent one to three years of the Dodge database was suggested, while the U.S. Census permitting data [46] for residential new constructions was recommended. To facilitate the process of generating random samples from the building populations, an online tool called the “State Sample Generator” was developed. This tool assisted states in distributing samples by county and further allocating them to specific jurisdictions within each county. By utilizing this tool, states could efficiently create representative samples or their code compliance evaluation studies.

To ensure systematic data collection, a checklist consisting of more than sixty code items were developed for both residential and commercial new construction. This was carried out by reviewing the pertinent ASHRAE 90.1 standard and IECC code. Each item in the checklist was assigned a value of one, two, or three points, based on its relative importance. To determine building level compliance, the total points earned for items marked as compliant in the checklist were divided by the total points for all checklist items excluding those marked as “Not Applicable” or “Not Observable”. The calculation provided a measure of compliance for each individual building.

Within a given state, the individual compliance ratings were averaged to calculate an overall compliance metric. To facilitate this process, a web application called Score + Store was made available to the states. This web app served as a platform for gathering compliance checklist from states, allowing them to gauge their progress towards achieving a 90% compliance rate.

3.2.2. Code Compliance Studies to Meet the ARRA Requirement

The Association of Professional Energy Consultants (APEC) conducted a field assessment study to determine the baseline compliance rate of residential and non-residential buildings in Illinois with the 2009 IECC using the BECP 2010 methodology [47]. Throughout the 120-day project duration, the APEC team evaluated a statistically significant sample of 44 residential buildings and a smaller sample of 10 non-residential buildings. These buildings were selected from 35 Illinois jurisdictions authorized to adopt building codes.

The authors of the study [47] observed that the BECP 2010 methodology appeared to prioritize product-related aspects over practices. They suggested that the checklist in the methodology should specify when it is used for the trade-off or performance-based compliance, rather than just the prescriptive path.

APEC, like many other code compliance evaluation studies, faced challenges in sample recruitment. They encountered limited accessibility to certain jurisdictions, which prevented them from achieving a true random sample. Additionally, time constraints prevented them from collecting a statistically valid sample of commercial buildings.

To estimate the cost of conducting code compliance evaluation studies, it was found that the average number of hours spent per visit was approximately 3.97 h. This duration encompassed activities such as plan review, onsite visit, and interview, with an estimated average cost of $496.62 per visit (excluding travel expenses). Furthermore, the APEC study [47] identified that 30% of jurisdictions had adopted building codes but lacked the necessary resources for implementation. Additionally, 60% of jurisdictions had both adopted building codes and possessed the resources for enforcement but chose not to enforce the codes due to political pressure.

The study conducted by APEC [47] indicated that compliance verification heavily relied on permit documentation. In instances where direct observation of the work was not feasible, experienced evaluators would indicate compliance with a “yes” entry based on factors such as overall consistency in the builder’s practices, confidence in the field inspector’s or building officials’ ability to assess compliance or supported evidence such as pictures and anecdotal notes in the project file. However, it is important to recognize that this approach to compliance verification may introduce bias and compromise the objectivity of the data collected in the field. To ensure the objectivity and reliability of field-collected data, it is crucial to prioritize direct observation whenever possible.

The BECP 2010 methodology, with a modification to the sample recruitment process, was employed to assess energy code compliance in residential and commercial buildings tin accordance with the Florida energy code. The study included a random sample of 43 single-family detached homes and 50 commercial buildings [48].

For commercial buildings, the recruitment process was a procedure established by the Florida Solar Energy Center (FSEC). It began with identifying buildings that had obtained permits in March 2011 within a specific county. Successful recruitment involved identifying buildings that had submitted energy code forms. If an insufficient number of buildings was found in a county, the search expanded to previous months until the required number of buildings was identified. If necessary, the search recommenced in a different county.

Regarding residential code compliance, the focus was on single-family detached buildings with a floor area ranging from 1500 to 2300 square feet. The recruitment process for residential buildings differed from the commercial approach. It involved generating a mailing list of buildings that met the specific criteria by researching public records. Energy code forms were requested from the identified buildings, and audits were scheduled for those homes that expressed confirmed interest.

In the Florida study, it was noted that nearly all the evaluated commercial buildings had already been completed and occupied [48]. The site visits during the study encountered certain challenges, such as owners granting limited time for the visits, restricted access to secured areas, and the inability to view fully assembled components. These factors imposed limitations on the extent of the observations that could be made during the site visits.

In the study, the evaluation of commercial code compliance was divided into 23 distinct categories, covering areas such as opaque and fenestration components, heating and cooling systems, and lighting power and control. On the other hand, residential code compliance was split into 14 focused categories, including mechanical systems, domestic hot water systems, and building envelopes. Compliance was determined by the submitted code forms and the actual construction observed during on-site inspections. In the case of performance-based code compliance, a comprehensive audit was conducted using the Florida code performance software called Energy Gauge USA [49] for 38 out of the 43 homes.

NEEA contracted with the Cadmus Group Inc. to conduct residential energy code compliance studies in the states of Montana, Idaho, Washington, and Oregon. The energy codes used in each state were specific to their respective jurisdictions: an amended version of the 2009 IECC for Montana and Idaho [50,51], the 2009 WSEC for Washington [52], and the 2011 Oregon Residential Specialty Code (ORSC) for Oregon [53]. To assess compliance, the studies utilized the BECP 2010 methodology with certain modifications. In Montana, the minimum sample size of 44 was applied for both local code jurisdictions and self-certifying areas, resulting in an actual visit of a total of 61 homes in local code jurisdictions and 65 homes within self-certifying areas [50]. In Idaho and Washington, the minimum sample size increased from 44 to 66 [52]. In the case of Oregon, the sample size increased from 44 to 88 [53]. The decision to increase the sample size was based on the experience gained from previous compliance studies. The study team recognized the limitations of collecting comprehensive data in a single visit and the inability to conduct multiple visits to individual homes within the study scope. Although there may not have been a specific statistical justification for the increase of minimum sample sizes, they were implemented to address the challenges of data collection and enhance the reliability and representativeness of the compliance evaluations.

During the residential energy code compliance study conducted in Montana [50], it was noted that the U.S. Census permitting data did not accurately reflect the complete building stock in the state. This discrepancy primarily arose from a significant number of the homes being constructed in self-certifying areas. While the U.S. Census bureau acknowledges that less than 5% of homes nationwide are built outside of permit-issuing jurisdictions, stakeholders in Montana estimated that around half the home construction in the state occurred in self-certifying areas. Therefore, relying solely on U.S. Census permitting data would not provide a fully representative picture of the building stock in Montana.

To obtain an independent and more accurate estimate of the residential building population in Montana, the Cadmus group utilized data from the Montana Building Industry Association (MBIA). The MBIA, as an industry trade group, tracks housing starts and serves as a primary source for housing construction data. In this study, the MBIA data was used to supplement the U.S. Census Bureau’s building permit data at the county level for both local code jurisdictions and self-certifying areas [50]. By combining data from multiple sources, including the MIBA and the U.S. Census Bureau, the study aimed to improve the accuracy and completeness of the building population estimates for Montana, considering the unique characteristics of self-certifying areas.

Under the framework of the BECP 2010 methodology, a checklist of building characteristics and code items was compiled to access code compliance in different states. The number of items in the checklist varied across the studies: 63 items in Montana [50], 61 items in Idaho [51], and 54 items in Washington [52] and Oregon [53]. During the analyses, Cadmus found that many items on the checklist had minimal impact on building energy consumption. To address this, they developed an alternate methodology known as the Significant Item methodology. This approach focused on a select few items that had the most significant effect on energy use and assigned equal weight to each of these items, giving them three points in the calculation of compliance rates.

In the Montana and Idaho studies [50,51] the Significant Items included the window glazing U-factor, duct sealing, ducts located away from building cavities, floor insulation, wall insulation R-value, air sealing, and high-efficiency lighting. In the Washington study [52], the credit options in Chapter 9 of the WSEC were considered as an additional Significant Item [52]. The Oregon study [53] utilized a set of ten different significant items.

In the energy code compliance studies conducted in Idaho, Washington, and Oregon [51,52,53], the estimation of the energy impact of non-compliance was an important aspect. To facilitate the analysis, additional data items related to building characteristics were added to the checklist and collected for building energy simulation purposes.

For the energy usage compliance analysis, the Simple Energy Enthalpy Model (SEEM) simulation tool [54] Version 94 was employed. SEEM is a simplified simulation model used to estimate energy consumption. In cases where specific building data required for running the simulation models were missing, it was assumed that the components just met minimum code requirements. This approach was chosen to avoid introducing a significant positive or negative bias in the analysis. However, it is important to realize that the substitution of missing data inevitably introduces biases into the analysis.

To quantify compliance based on building energy, a compliance metric was calculated by dividing the energy usage of the standard reference home which just met the code, by the energy usage of the as-built home. This metric provided a means of comparing the actual energy usage of the evaluated buildings with the energy usage of a reference home that fully complies with the energy code requirements. While this methodology aimed to generate code compliance based on building energy, it is essential to consider the potential biases introduced through data substitution and the limitations associated with simplified simulation models.

In the assessment of the code compliance of homes following the trade-off compliance approach, the field-verified values of homes were compared against REScheck [55] values rather than against prescriptive requirements. This approach was taken because builders opting for the trade-off approach typically rely on REScheck for compliance verification [51]. REScheck is a software tool commonly used by builders to demonstrate compliance with energy codes [55].

The studies conducted in Idaho, Washington, and other states revealed a strong correlation between the compliance estimates obtained from the checklist method and the Significant Item method. This finding suggested that it is possible to reasonably estimate compliance with the checklist by assessing compliance with only a subset of a few key items [50,52].

It is important to note that, in the Idaho study [51], no statistically significant relationship was found when regressing the percentage of energy savings derived from simulation modeling against the compliance rate. Similarly, in the evaluation of commercial code compliance in Oregon [56], no correlation was observed between energy use and code compliance. Some of the highest-performing buildings (lowest EUIs) did not comply with certain aspects of the code, while other buildings that met all aspects of the code had higher than average EUIs [56].

ADM Associates, Inc conducted a study in Illinois using the BECP 2010 methodology to assess the statewide level of code compliance for new commercial and residential buildings permitted to the amended 2012 IECC in Illinois [57]. The study included a targeted sample of 44 residential sites and 42 non-residential sites, generated using the BECP’s State Sample Generator Tool. However, the study faced challenges in obtaining a full sample set as recommended by the tool. These challenges included the unwillingness of individual jurisdictions to participate, failure of the jurisdiction to enforce any energy code, and project cancellation after selection. These challenges are common in code compliance studies and reflect the complexities of sample design, selection, and recruitment in such studies. To determine individual compliance rates, ADM Associates utilized the BECP’s “Score + Store” web tool. For residential buildings, the statewide compliance rate was calculated based on a straight average of the individual compliance rates. For commercial buildings, the statewide compliance rate was determined by taking a weighted average of the compliance rates within the three building size strata used in the stratification sampling.

In the study conducted by ADM Associates, the baseline energy consumption for the residential buildings was determined using the residential prototype models developed by the Pacific Northwest National Laboratory (PNNL). For non-residential buildings, site-specific eQuest models [58] were used to estimate the baseline energy consumption. It was found that the residential energy consumption was not directly related to the compliance rate. This finding suggests that certain sections within the checklists had little to no impact on the energy consumption of the homes [57].

In the 2017 Massachusetts Energy Code Compliance and Baseline Study conducted by DNV GL [59], the BECP 2010 methodology was adopted to assess compliance with the 2012 IECC for commercial buildings in the state of Massachusetts. For the study, a sample of 39 buildings was selected using a building-size-stratified random sample design. The code compliance evaluation primarily relied on the review of construction documents. Only nine buildings underwent site verification, as they claimed that previous studies had shown minimal deviation document review and site observations. DNV GL utilized its own Excel-based data collection tools, deviating from the BECP 2010 methodology recommended. They introduced the concept of partial compliance in the compliance evaluation process. Instead of a binary yes/no assessment, DNV GL implemented an ordinal scale with levels 0, 1/3, 2/3, and 1 to handle partial compliance. This allowed for a more nuanced evaluation of the compliance level. In the revised compliance evaluation, DNV GL employed a weighting system for items such as equipment efficiency or lighting power density. The ratio of actual performance to the specific efficiency or performance requirement was used to assign scores to these items, reflecting the degree of compliance. By introducing partial compliance and acknowledging varying levels of efficiency and performance, DNV GL aimed to capture a more accurate representation of compliance levels and the overall energy performance of the assessed buildings.

VEIC utilized the BECP 2010 methodology for the New York energy code compliance study [60]. However, due to practical and budgetary considerations, VEIC made adaptations to the BECP 2010 methodology, and these adaptations were necessary due to challenges encountered during the study, such as the lack of readily available data, resistance from code officials in certain jurisdictions, and difficulties in reaching and obtaining consent from building owners. One of the adaptations made was adjusting the sample size and the random sampling process. The study eventually involved conducting detailed plan reviews and field inspections on 44 new residential buildings and 26 new commercial buildings.

The ARRA code compliance goal applies not only to new construction but also to renovation projects. While this study [60] did not include renovation projects, it is worth noting that the study found that the lack of permit data for renovations posed a significant challenge in analyzing compliance for such constructions. According to interviews conducted with code officials, the number of renovation permits issued annually in the state of New York exceeds the number of permits issued for new home construction. However, the renovation permit accounted for only six percent of all residential permits in the available Dodge dataset.

In this study [60], data collection served multiple purposes beyond determining code compliance rate based on the BECP 2010 methodology protocol. The collected data also enabled the generation of a HERS score for new homes and the calculation of lost savings resulting from non-compliance in both residential and commercial new buildings.

In the analysis of the energy impact of non-compliance, composite commercial building models were developed using the eQuest simulation tool [58]. A composite building represents the typical features of buildings within the sample it represents. In this study, three composite building models were constructed to represent small, medium, and large commercial constructions, aligning with the three size strata established in the stratified sample design. To create the composite buildings, information from the sampled buildings was combined in a representative manner, including details on construction and mechanical system types. The analysis of lost savings for residential new construction was conducted using the REM/Rate models on an individual home basis. REM/Rate is a widely used software tool that allows for the calculation of energy ratings and analysis of energy consumption in residential buildings.

Based on the estimated lost savings resulting from non-compliance with energy codes, the study projected a total minimum of approximately $1.3 billion in savings encompassing both residential and commercial sectors. It is important to note that the savings estimate was based on assumptions of a similar levels of construction activities and lost savings per building, as well as 20 years of useful life of commercial buildings and 50 years of useful life of residential buildings.

In the study conducted by Wirtshafter et al. [10] to test the BECP 2010 methodology and assess energy code compliance in New York State, several issues were identified with the design of the BECP 2010 methodology. These issues included self-selection bias, potential inconsistency in protocol implementation among states and evaluators, and scores that may not align closely with the REScheck and COMcheck [61] approaches commonly used for establishing actual compliance to each state’s energy codes.

As a recommendation, the study [10] proposed that BECP reconsider the use of third-party evaluators to assess energy code compliance. Instead, they advocated for the establishment of a “Certified Energy Inspector” (CEI) accreditation or license as they recommended to the New York Department of State (NY DOS). This approach involves having first party entities, such as design professionals and CEIs representing the builder owner, verify code compliance. Meanwhile, second-party entities, including local code officials and NY DOS, would provide oversight. The authors believed that this first-party verification, coupled with second-party oversight, would lead to a more comprehensive assessment of code compliance compared to relying solely on independent third-party evaluators. By involving professionals directly associated with the code compliance process and incorporating oversight from regulatory bodies, the proposed approach aimed to address the identified issues and improve the overall efficiency and consistency of energy code compliance assessments.

The Energy Futures group developed a comprehensive Vermont Energy Code Compliance Plan with the objective of achieving 90% compliance with Vermont’s commercial and residential building energy codes by 1 February 2017 [62]. The plan was developed to meet the requirement for receiving 2009 ARRA funds and aligned with Vermont’s adoption of the 2011 residential and commercial building energy standards, which met or exceeded federal targets.

The Vermont Energy Code Compliance Plan outlined strategies and actions to support the goal of achieving high compliance rates. It identified four pillars that formed the foundation for a robust compliance platform: measurement and evaluation, leadership and policy, outreach and education, and resources and funding. These pillars aimed to address various aspects of energy code compliance, including data collection and analysis, stakeholder engagement, policy development, educational initiatives, and financial support.

3.3. Post ARRA

The BECP 2010 methodology, along with its variants and similar approaches, has significantly contributed to addressing challenges associated with code compliance evaluation studies. However, the methodology has some limitations [63,64].

One limitation is that the methodology primarily focuses on a prescriptive and measure-by-measure review. It assesses compliance based on the percentage of compliant measures in each building on an individual component basis. This approach does not differentiate between varying levels of non-compliance or evaluate the energy impact of specific requirements. A strict focus on absolute code item requirements might not capture the legitimate tradeoff and overlook alternate strategies or design choices that can achieve code compliance. Additionally, due to the single site visit principle, it is not feasible to evaluate every building component as the site visit may occur during different construction stages. As a result, only observable code items during the site visit are assessed, which may not capture the full range of building characteristics or design features that could impact energy efficiency. This approach requires significant resources and time to recruit and review a sample set of buildings, but it may not provide sufficient data to assess building-wide compliance or the overall efficiency of individual buildings.

Furthermore, there is no direct relationship between compliance rates and energy savings from non-compliance in the binary approach used by the BECP 2010 methodology. This approach fails to address the crucial question of the dollar value associated with increasing compliance with the energy code. Policy makers, funders, and program implementers are interested in understanding the potential energy cost savings achieved through better compliance. To address this concern, the BECP has developed new methodologies capable of estimating the potential energy cost savings for a sample of buildings by improving compliance with the code [65].

In summary, while the BECP 2010 methodology has made significant contributions to code compliance evaluation studies, there is a need for methodologies that go beyond a measure-by-measure assessment and provide a more comprehensive understanding of overall compliance and the energy savings associated with improved compliance.

3.3.1. BECP Residential Energy Code Field Study

The U.S. DOE initialized an FOA for residential energy code compliance field studies in 2014 [11]. These studies aimed to explore energy savings opportunities through enhanced code compliance in single-family residential construction [18]. The primary objectives were to establish a standardized methodology for quantifying the energy impacts of code-based measures, assess the effectiveness of education and training in improving compliance, and project the long-term savings from enhanced energy code compliance. To facilitate our discussion, we will refer to the methodology from the residential FOA study as the residential FOA methodology in this paper.

The residential FOA studies consisted of three phases. The first phase involved a baseline field study to evaluate the energy performance of newly constructed single-family residential buildings and identify opportunities for energy efficiency improvements. The second phase focused on education, training, and outreach activities specifically targeted at addressing code compliance issues identified in the baseline study. The state teams implemented various training intervention strategies, including classroom and online training, circuit riders, hotlines, and technical resources. These interventions were tailored to each state’s specific needs and were implemented over a period of approximately two years. Following the education and training phase, a second field study (Phase III) was constructed to measure the impact of the interventions on code compliance and the associated energy impacts. This study compared the results to the original baseline study to assess the effectiveness of the education and training activities in improving compliance and achieving energy savings [66,67,68,69,70,71].

The residential FOA methodology focus on key items, a subset of code requirements identified as having the largest direct energy impact in single-family homes. The list of the key items consists of envelope tightness, duct leakage, wall insulation, ceiling insulation, foundation insulation, window efficiency (U-factor and solar heat gain coefficient), and high-efficiency lighting. The concept of key items is like the concept of Significant Items [50,51] introduced in the methodologies of the Northwest region.

In the two field studies, i.e., Phase I and Phase III, the methodology required the project team to define a geographic area for data collection and then collect at least 63 observations of all key items. The minimum sample size was determined with the objective to ensure that the pre-intervention baseline study (Phase I) can be compared to the later post-intervention study (Phase III) with 90% statistical confidence, ensuring that any differences observed between the studies are within 10% of their true population values [72].

It is important to note that the specified sample size in the sample plan refers to the minimum number of observations for all key items, not the number of buildings. To minimize bias and ensure confidentiality, homes were only visited once, and the total number of homes visited was typically larger than the minimum sample size as not all key items would be observable during a single site visit. The total number of homes that would need to be visited was not predictable in advance.

Based on the data collected during field survey, the residential FOA methodology consists of simulation-based analyses to assess baseline energy efficiency in new single-family residential buildings and quantify savings potential due to the non-compliance of key items. There are two types of energy simulation and analysis conducted on the collected data. The first type of simulation and analysis is used to obtain the EUI distribution and to calculate a state average EUI of the new residential homes constructed with the state current construction practice compared to those from homes built to the energy code. The results of this type of simulation aims to reveal the status quo of the code compliance in term of energy consumption. The second type of simulation and analysis focuses on the non-compliance of the key items and its results are used to estimate the measure-level energy savings potential if all non-compliant key code items were improved to meet the code requirement.

The construction of energy models for the measure-level savings potential is straightforward. A model is created on each unique value of the non-compliant observations of a key code item while all key items other than the one under evaluation have the value of the code requirement. However, there are challenges to constructing an energy model for assessing the state average EUI because the single site visit principle prohibits the collection of a full dataset required for simulation on any individual home visited. It is inevitable to have one or another way of data substitution for key items which are not in place or not visible during the site visit if the energy model is to be created on the individual home basis [51,52].

In the residential FOA methodology [72], the field observations of key items are treated as empirical distributions of the code values expected in the time when the field surveys were conducted, assuming the site-visited homes are a representative subset of single-family homes under construction in the state. A so-called pseudo home, which is merely an instance of a residential prototype model developed by the PNNL for supporting building energy code development [73], can be created by preparing all necessary inputs through a random drawing of values from the empirical distributions of the key items. Repeating the random-drawing process many times, a population of a large number of pseudo homes are generated. While any single pseudo home does not resemble any visited home, collectively, the values of key items in the suite of pseudo homes represent the distributions of the key code items in the new homes of the state.

In both sets of simulations, baseline models were created with all key items set to the code requirements. The EUI distribution and the average EUI of the large number of pseudo homes versus the EUI of the code-compliant home provide an overall picture of the energy consumption of the new homes relative to homes built to the energy code. The EUI result of the second set of energy simulations estimates the savings potential if all non-compliant homes were enhanced to meet the minimum code requirements. These data could be used to guide the state or local jurisdiction on their education and training activities aiming to improve code compliance.

The credibility of a field study for the evaluation of code compliance relies on its objectiveness, consistency, and accuracy. The residential FOA methodology consists of systematic training for data collectors which emphasize that only observable information should be recorded, and no assumption is acceptable [72]. This distinguishes it from many other code compliance studies where one or another way of data substitution is embedded in their methodologies.

The residential FOA methodologies were employed in eight pilot states including Alabama [74,75], Arkansas [76] (Arkansas participated in Phase I but dropped out of Phase III), Georgia [77,78], Kentucky [79,80], Maryland [81,82], North Carolina [83,84], Pennsylvania [85,86], and Texas [87,88], and customized sampling plans were implemented for each pilot state to generate a random representative sample of new construction.

In addition to its use in code compliance studies in the eight pilot states, the residential FOA methodology has been applied in code compliance studies of other states such as Idaho [89], Michigan [90], Montana [91], Nebraska [92], Oregon [93], Tennessee [94], and Virginia [95]. The U.S. DOE encouraged states to conduct these studies periodically, typically every 3–5 years, to validate the impacts of codes and other energy-efficiency programs and benchmark technology trends in residential construction [18].

3.3.2. BECP Commercial Energy Code Field Study

To transition from the binary assessment of commercial building code compliance in the BECP 2010 methodology to a methodology focused on estimating energy cost savings lost due to non-compliance, the BECP developed a value-based code compliance methodology. This methodology was initially developed and tested in a pilot study with nine office buildings in climate zone 4C [65], and was later expanded to include two additional climate zones (2A and 5A) and one additional building type (retail) with 230 buildings in the 2016 commercial FOA field study [96,97]. To facilitate the discussion, the developed methodology will be called the commercial FOA methodology in the context of this paper.

In the commercial FOA methodology, all the requirements in the non-residential provisions of the 2012 IECC that directly affect energy use were inventoried and grouped into related measures. For the pilot study [65], there were 63 measures, which increased to 67 measures in the expanded field study [96,97]. A list of possible conditions to each code measure likely to be encountered in the newly constructed buildings was defined and building energy simulations were performed using commercial prototype building models [98]. Regression models were then established based on the estimated energy costs for various conditions of each measure and used to determine the lost energy savings for different measure conditions based on the collected data from the field study.

The commercial FOA methodology [99] differs from other studies in several aspects. Firstly, it considers a range of possible measure conditions that could be encountered in newly constructed buildings, and prototype building energy simulations are used to estimate the energy impacts. Regression analysis takes into account variability in energy impact due to building type, size, climate zone, and observed measure compliance. A spreadsheet-based calculation tool was developed for estimating the present value (PV) of lost energy savings for each measure, and a subsequent total for each building when measures were not in compliance with the energy code was developed [100].

Secondly, the annual lost savings from each measure are converted to a present value over the lifetime of the building to account for the time value of money and lifetime differences of the measures. The use of the present value of lost savings provides a more accurate metric and is used to rank the measures based on their potential for lost savings over the lifetime of the building [65,99,100].

Third, the methodology includes a step for the field team to track their time spent verifying the code measures in the field. The potential savings of the measures are then normalized by the number of hours used for verification. This information allows future compliance evaluations to focus on measures that have a significant impact on energy use over the building’s lifetime and those with the highest potential for savings recovery per verification hour.

Lastly, the standard errors and coefficient of variation from the regression analysis can be used to update the coefficient of variation used in the stratified sampling formulae for sample size determination. Leveraging all data points across strata in the regression analysis reduces standard errors and the coefficient of variation, resulting in a smaller sample size compared to the survey-sampling approach.

Overall, the commercial FOA methodology provides a comprehensive framework for evaluating code compliance and estimating energy cost savings lost due to non-compliance in commercial buildings.

3.3.3. Northwest Commercial Code Compliance Evaluation

To go beyond the compliance rate code compliance evaluation of the BECP 2010 methodology, NEEA funded a pilot study to explore the energy focus of code compliance evaluation and developed an integrated, empirically based methodology for evaluating building characteristics, code compliance, and energy performance in fully constructed and occupied commercial buildings. This methodology, referred to as the Northwest methodology in this paper, aims to serve as the foundation for NEEA’s future state-level commercial code evaluation in the Northwest region of the United States [63,64]. The term “Northwest methodology” is also used in a broader way to refer to other methodologies used for conducting code compliance in the Northwest region.

The Northwest methodology has several distinct features. First, it focuses on building systems rather than individual building component. Commercial energy codes in the Northwest jurisdictions encompass three prescriptive codes that target different parts of the buildings: the envelope, mechanical systems, and lighting systems. Within each code section, designers are allowed to trade off requirements within the design context. Therefore, the code compliance evaluation in the Northwest is conducted for each subcomponent of the major building systems, and binary compliance logic is used to assess the overall compliance of each system. This strategy has been employed throughout the whole series of commercial code compliance studies in the Northwest [22,23,28,29,33,34,63].

Secondly, the methodology re-focuses the evaluation of energy codes around their outcomes in fully constructed and occupied buildings. This approach differs significantly from other code compliance studies and previous studies in the Northwest, where buildings under construction were selected for a site visit. Many code provisions require controls to be in place with optimal settings, such as temperature setbacks, resets, and deadbands. The effective implementation of controls plays a crucial role in achieving energy savings from mechanical and lighting systems. By including fully constructed and occupied buildings, the methodology can assess the presence of controls and evaluate all building systems in their completed form during a single site visit. The shift from assessing buildings under construction to buildings as-built and as-operated has started to be attempted in earlier code compliance studies conducted in the Northwest [34,35].

Third, the Northwest methodology collects electric and gas utility billing data for a minimum of 12 months for each building. This empirical data is used to explore the relationship between energy use and code compliance. The EUI is calculated for each building based on the conditioned floor area data collected during the compliance assessment. EZ Sim is then used to disaggregate the energy use by end-use categories. The total and end-use EUIs are benchmarked against results from previous studies to identify progress in building energy use over time by building type.

The Northwest methodology developed from the pilot study [63,64] was applied to the commercial energy code evaluation of the Oregon Energy Efficiency Specialty Code (OEESC) 2010 and 2014 in Oregon by Ecotope [56], and to the evaluation of the 2015 Washington State Energy Code (WSEC) and Seattle Energy Code (SEC) in Washington state by Cadmus [101].

In the Oregon [56] and Washington studies [101], the Dodge database of new construction starts used to create the sample frame and the sample design included four major building types (schools, retail, multifamily, and office) in Oregon [56], or commercial and multifamily buildings in Washington [101] with a few size categories. The Oregon study used a Dodge dataset of new construction starts from the second quarter of 2013 through the second quarter of 2016 to create the sample frame, which consists of 222 buildings and the sample size was 46 buildings. The sample frame covers 28% of the overall floor area [56]. The intent was to reach a 90/10 confidence/precision on building floor area. The original sample plan called for a target of 64 buildings using a three-strata design. The original sample plan was revised in response to recruiting challenges by varying the number of strata across building types.

The Washington sample frame was developed by combining Dodge data and Construction Monitor data, supplemented with permit data from select building departments, and developer data from team members’ networks and other contacts. The sample frame consisted of 2211 buildings, and the sample size was reduced from 108 to 76 for site visits due to the COVID pandemic [101]. Sample weights were derived to account for the sample stratification to ensure an unbiased estimate was produced from the data gathered at the individual buildings.

3.3.4. Other Methodologies and Studies

In addition to the research studies focusing on code compliance evaluation on residential and commercial buildings, BECP also conducted work on low-rise multifamily buildings in various regions of the U.S. The methodology developed for low-rise multifamily buildings [102] was built upon and extended from the established residential FOA methodology [72]. The methodology aimed to provide a consistent and replicable approach to code compliance in low-rise multifamily buildings.

The study included a sample of nearly 100 buildings across four states: Illinois, Minnesota, Oregon, and Washington. Low-rise multifamily buildings are a specific type of mixed-use building, where the conditioned floor area is mainly subjected to the residential energy code requirement. However, certain areas of the building such as corridors and common spaces fell under the provisions of the commercial energy code. To account for this, a catalog of building characteristics and key items was compiled from both residential and commercial building codes.

Data collection for the study involved plan reviews and field inspections to assess compliance with relevant code provisions. Building energy simulations were then conducted to estimate the distribution of EUI for the sampled buildings and to quantify the energy savings potential with each individual key item. This analysis was performed for each of the four states involved in the study [102].

In Minnesota, the low utility rates and the restrictions on claiming on energy savings against the Conservation Improvement Program (CIP) goal posed challenges for utilities in implementing code compliance programs [103]. To address this, a study was conducted to test the potential for cost-effectively achieving CIP program savings by providing guidance and technical assistance to designers and plan reviewers to improve compliance with the Minnesota Energy code.

The primary goals of the study were twofold. Firstly, it aimed to establish a local precedent for utility-funded energy code compliance enhancement programs in Minnesota, which could serve as a model for the development of full-scale programs. Secondly, it aimed to evaluate the pilot program itself. The focus of the study was on maximizing cost-effectiveness rather than energy savings, considering the relatively low utility rates in Minnesota. The objective was to minimize costs while capturing readily available energy savings associated with increased energy code compliance.

The pilot test included two approaches for enhancing commercial energy code compliance programs, which differed in the method of engagement with builders and the critical times for intervention in the building design and development process. To estimate the average building’s annual penalties for non-compliance, a sample of 24 commercial buildings was analyzed [103]. Overall, the study aimed to demonstrate the feasibility of cost-effectively enhancing energy code compliance in Minnesota by targeting specific code items and implementing strategic intervention approaches. The findings from this study could provide valuable insights for utilities and policymakers in developing comprehensive code compliance programs that align with state energy goals.

Most of the code compliance assessment studies conducted at the state and regional levels provide valuable insights into energy code compliance for residential and commercial buildings. However, these studies may not fully meet the needs of cities for several reasons. Firstly, statewide compliance studies tend to focus on broader trends at the state level, which may limit the level of interaction between data collection teams and individual jurisdictions within cities. Secondly, the sample sizes in statewide studies typically include only a small number of buildings from each jurisdiction, which may not provide a comprehensive understanding of compliance issues at the city level.

Furthermore, cities that participate in these evaluations often receive minimal or no feedback on the findings or specific actions they can take to address compliance issues within their jurisdiction. Additionally, city and state governments may have different goals and priorities in utilizing the data that comes out of a compliance assessment.

To address these limitations and cater to the unique needs of cities, the Institute for Market Transformation (IMT) and the Natural Resources Defense Council (NRDC) launched a joint initiative called the City Energy project (CEP) [104,105]. As part of this initiative, a specific protocol known as the CEP Assessment Methodology was developed. This methodology is designed as a plug-and-play approach for cities to evaluate their building construction practices and assess code compliance processes.

The CEP Assessment Methodology primarily focuses on qualitative evaluation, involving interviews, process evaluation, and limited data collection on building systems. The aim is to uncover potential areas of low compliance and provide actionable feedback to cities. Depending on the interests and resources of the cities, the methodology can be implemented with or without energy modeling [105].

By utilizing the CEP Assessment Methodology, cities can gain insights into their local compliance challenges and identify opportunities for improvement. The qualitative evaluation approach allows for a more tailored assessment that aligns with the unique goals and priorities of the city, enabling them to take targeted actions to enhance energy code compliance within their jurisdiction.

4. Discussion

Table 1. Summary of sampling characteristics and methodologies of the code compliance studies.

Reference	State	Building Type	Methodology	Sample Source	Code Permit Year Home Built	Sample Frame	Site Visit	Miscellaneous
[21,23,26]	Oregon	Commercial	Northwest	Dodge database	April 1990–April 1991	213	71	Ten building categories (office); two stratum stratification by size; random selection; building with documents and permission for site visit and interview; no target sample size; DOE2 for energy simulation.
[21,22,26]	Washington	Commercial	Northwest	Dodge database	Jan 1990–Jan 1991	468	70	Eleven building categories (office); two stratum stratification by size; random selection; building with documents and permission for site visit and interview; no target sample size; DOE2 for energy simulation.
[28,29,30,32]	Washington	Commercial	Northwest	Dodge database permitted in 1995.	1994 Washington NREC	792	88	Eleven building categories (mainly retail and warehouse); three stratum stratification by size; random selection except all largest buildings included; building with documents and permission for site visit and interview; no target sample size; DOE2 for simulation.
[33]	Idaho Montana Oregon Washington	Commercial	Northwest	Dodge database	ID and MT: ASHRAE 90.1-1989 OR: Oregon 1989 WA: Washington 1994	356 168 1020 655	71 48 64 32	ID and MT: permitting from June 1997 to June 1998. OR: permitting from June 1997 to June 1998. WA: Washington 1994
[34,35]	Idaho Montana Oregon Washington	Commercial	Northwest	Dodge database	ID: 2000-IECC MT: ASHRAE90.1-1989 OR: Oregon 1998 WA: Washington 2001 Permit from 2002 to 2004	1196 257 1780 2846	64 29 107 146	Fourteen building categories; three-stratum by size; target sample size determined on a 90% confidence interval with a 10% significance; 190 of 346 homes with billing data for EZ Sim analysis; shift from building under construction to fully constructed and occupied.
[40]	California	Commercial/Residential		Nine building departments	Title 24		418	Sample sizes determined with 90% confidence and 10% precision; permitted Nov 2005–June 2006.
[41]	Indiana	Commercial		New and alteration submitted to IDFBS	2000-IECC/2003-IECC		55	Retail, office, school, medical, and grocery; no target sample size; COMcheck to determine compliance; original sample size was set to 50 with 5 added at the end of the study.
[42,43]	Oregon	Residential		SF permitted in 1993		14,000	283	No centralized permit existed in Oregon and sample frame assembled from individual jurisdictions; random sample size was determined with a 95% confidence interval and a 5% margin of error, based on a binomial distribution with assumed 75% compliance rate; 50% field visit conducted at the time of final inspection.
[45]	Vermont	Residential		Multiple sources	Between 1999 and 2001		159	Census bureau and other data sources.
[47]	Illinois	Commercial Residential	BECP 2010		2009-IECC		44 10	BECP 2010 has a statistically determined sample size of +/−44 with a State Sample Generator Tool (same for all rows marked with BECP 2010 in the Methodology rows).
[48]	Florida	Commercial Residential	BECP 2010 (Variant)		Florida 2007		50 43	Specific recruitment approach established by FSEC applied on the commercial sample selection and a mailing list recruitment used for residential sample selection; commercial sample were occupied buildings; all residential sample were permitted using Florida performance methodology; Energy Gauge USA.
[50,51,52,53]	Montana Idaho Washington Oregon	Residential	BECP 2010	MT: Multiple sources ID, WA, OR: Census permits	MT and ID: Amended 2009-IECC WA: 2009-WSEC OR: 2011-ORSC	1130 + 2207	61 + 64 66 66 88	BECP 2010 has a statistically determined sample size of +/−44 with a State Sample Generator Tool. The two numbers for MT are for both areas with local official and self-certifying areas; SEEM, significant item.
[58]	Illinois	Commercial Residential	BECP 2010	96 jurisdictions	Amended 2012-IECC		44 (30) 42 (13)	The sample size from the sample generator tool were 44 and 42, but there were only 30 and 13 site visits for commercial and residential buildings; E+ and eQuest for energy simulation.
[59]	Massachusetts	Commercial	BECP 2010 variant	Dodge database	2012-IECC	655	39	Partial credit for partial compliance
[60]	New York	Commercial Residential	BECP 210 variant	Dodge database	ASHRAE 90.1-2007 2009_IECC		44 26	eQuest simulation for commercial composite building; REM/Rate modeling for residential individual building.
[74,75]	Alabama	Residential	BECP Res FOA		2015 Alabama Code	9506	134 | 126	BECP Res FOA has a statistically determined sample size of 63 for all key items (same for all rows marked with BECP 2010 in the Methodology rows). Average of the 19 most recent months of Census Bureau permit.
[76]	Arkansas	Residential	BECP Res FOA		2014 Arkansas Energy Code	5257	226	Average of the three most recent years of Census Bureau permit.
[77,78]	Georgia	Residential	BECP Res FOA		2011 Georgia Energy Code	27,503	216 | 139	Average of the three most recent years of Census Bureau permit.
[79,80]	Kentucky	Residential	BECP Res FOA		Amended 2009 IECC	7345	140 | 121	HVAC and plumbing permits; two phases of field studies for the pilot state.
[81,82]	Maryland	Residential	BECP Res FOA		2015 IECC	10,541	207 | 185	Average of the three most recent years of Census Bureau permit.
[83,84]	North Carolina	Residential	BECP Res FOA		2012 North Carolina Code	30,029	249 | 134	Average of the three most recent years of Census Bureau permit.
[85,85]	Pennsylvania	Residential	BECP Res FOA		2009 IECC	16,371	171 | 160	Average of the three most recent years of Census Bureau permit.
[87,88]	Texas	Residential	BECP Res FOA		2015 IECC	100,608	133 | 136	30 counties in south central and southeast.
[89]	Idaho	Residential	BECP Res FOA		Idaho Energy Conservation Code	11,019	127	Average of the three most recent years of Census Bureau permit.
[90]	Michigan	Residential	BECP Res FOA		Michigan Residential Energy Code	12,381	124	Target sample sizes vary by code item being either 63 or 40.
[91]	Montana	Residential	BECP Res FOA		Amended 2012 IECC	3161	125	Average of the three most recent years of Census Bureau permit.
[92]	Nebraska	Residential	BECP Res FOA		2009 IECC	5436	147	Average of the three most recent years of Census Bureau permit.
[93]	Oregon	Residential	BECP Res FOA		2017 ORSC	11,041	162	Average of the three most recent years of Census Bureau permit.
[94]	Tennessee	Residential	BECP Res FOA		Amended 2009 IECC	28,021	138	Average of the three most recent years of Census Bureau permit.
[95]	Virginia	Residential	BECP Res FOA		2015 Virginia Code	22,497	138	Average of the three most recent years of Census Bureau permit.
[65,98]	4C	Commercial	BECP Com FOA		2012-IECC		9	Pilot study; one building type (office); a single climate zone (4C); 63 measures.
[97]	2A, 5A	Commercial	BECP Com FOA		2012-IECC/ASHRAE 90.1-2010	579	230	Two building types (office and retail); two climate zones (2A and 5A); 67 key code items; regression.
[64]	Washington	Commercial	Northwest		WA: 2009-WAEC		12	Pilot; fully constructed and occupied with a minimum of 12 months electric and gas billing data; EZ Sim; building categories: education, multifamily, office, healthcare, retail, and warehouse; binary compliance logic.
[63]	NEEA Idaho Montana Oregon Washington	Commercial	Northwest	Dodge database Q2 2011–Q1 2013	ID, MT, OR: amended 2009-IECC WA: 2009-WSEC Construction occupied in 2013/2014	3207 (Region) 353 (ID) 251 (MT) 901 (OR) 1702 (WA)	112 (Region) 45 (ID) 44 (MT) 68 (OR) 99 (WA)	Pilot; fully constructed and occupied with a minimum of 12 months electric and gas billing data; target sample size determined based on a 90% confidence interval with a 10% significance; fifteen building categories (education, office, health, retail, and multifamily).
[56]	Oregon	Commercial	Northwest	Dodge database Q2 2013–Q2 2016	Permitted on 2010-OEESC and 2014-OEESC	222	46	Fully constructed and occupied with a minimum of 12 months electric and gas billing data; not all commercial buildings but focused on key building type: office, retail, school, and multifamily; sample size determined based on a 90% confidence interval with a 10% precision. Original sample size was 64 and a revised design in response to recruiting challenges led to a final sample of 46; billing analysis.
[101]	Washington	Commercial	Northwest	Multi-data sources	2015 WSECC	2211	76	Dodge + Construction Monitor + selected building department + developer data from team member’s network; target sample size: 108 with 76 completed visits due to COVID-19; billing analysis; not fully constructed.
[102]	Illinois Minnesota Oregon Washington	Multifamily		Dodge database/building permit survey			24	16 projects that received city plan review support, 8 “control” projects that were used for baseline comparison purposes, and 12 projects that received design team support.

summarizes the sampling characteristics and compliance methodologies of the reviewed code compliance evaluation studies. It includes information such as the state where the study was conducted, the type of buildings examined, the data sources used to establish the sample frame, the criteria (e.g., year built or code permitted) for building inclusion, the number of buildings in the sample frame, the number of buildings visited, and other relevant details about the studies.

4.1. Code Compliance Evaluation Methodologies

Prior to the ARRA, there were no common methodologies used in the variety of code compliance studies, and the difference and inconsistency in methodologies meant that results from different code compliance studies were not directly comparable. The BECP 2010 methodology, developed to support the ARRA 2009, was the first commonly used approach of its kind, and many U.S. states and jurisdictions adopted the BECP 2010 methodology or a variant of it. The widespread application of the BECP 2010 methodology or its variants symbolized the first national effort in U.S. history to assess building energy code compliance nationwide.

The testing and adoption of the BECP 2010 methodology by different states and jurisdictions led project teams to identify its deficiencies and propose enhancements. The findings and recommendations that emerged from the adoption of the BECP 2010 methodology by various entities prompted the U.S. DOE to issue a Funding Opportunity Announcement for field studies related to the improvement of compliance evaluation methodology for new residential construction in 2014 [18] and for new commercial construction in 2016 [19]. An enhanced code compliance methodology, with an emphasis that shifted from a binary pass/fail compliance rate to an energy consumption-based metric, was developed and tested through pilot studies in a few states. The residential FOA methodology has since been widely adopted by states and jurisdictions for the code compliance evaluation in states other than the pilot states, while the commercial FOA methodology has expanded from one single climate zone of one building type to two building types across three climate zones.

4.2. Steps Involved in Code Compliance Evaluation Studies

Although the steps involved in typical code compliance evaluation studies have evolved, common steps remain.

4.2.1. Sample Frame

The code compliance evaluation study starts with a sample fame. The sample frame is a subset of the building population. The preparation of a sample frame depends on the purpose of the study, the methodology used, the geographic area, climate zones, type of buildings (new construction or renovation), and the date and code under which the buildings are permitted. For example, if the code compliance study aims to evaluate the compliance of new residential construction with the most recently adopted energy code, the sample frame will consist of new residential homes permitted under the most recent energy code.

While the sample frame for the residential FOA [72] and commercial FOA methodologies [97] includes buildings at any stage of the construction process, the buildings included in the sample frame for the Northwest methodology [63] have shifted from buildings under construction to fully constructed and occupied buildings.

For commercial code compliance field studies, the Dodge database remains the main data source used over time [21,22,23,33,34,59,60,63,64,102]. For residential code compliance field studies, permit data compiled from local jurisdictions or stored in the state’s centralized database were used in earlier studies [45,50,58]. More recently, Census Bureau permitting data have been used widely [74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95].

However, the data in these main data sources may not always accurately represent the actual new construction activities, especially in home-rule states [50,79,80]. For example, the large portion of new home construction in the self-certifying area made the use of Census Bureau’s permitting data inappropriate in Montana [50]. In such scenarios, alternative and/or supplemental data sources are then needed for establishing the sample frame.

Due to the diversity of commercial buildings, which vary in terms of building type and size, stratified random sampling based on building size is commonly employed. This sampling technique ensures the representation of larger-sized buildings, which often have a significant energy impact despite being less numerous [21,22,23,26,28,29,30,34,35,64,99]. To accurately account for the impact of the stratified sample, weights of the sample from different strata are usually derived in order to obtain results aggregated across sample buildings. In some cases, all large buildings are selected for inclusion in the sample [28].

4.2.2. Sample Selection

Once a sample frame is established, the sample selection follows. Earlier studies often lacked a predefined target sample size [21,22,23,27] in the random sample selection process. In more recent studies, a target sample size is usually determined based on specific statistical criteria [10,64,72,97], such as a 90% confidence interval with 10% precision [8,34,35,44].

The BECP 2010 methodology proposed a statistically valid sample of approximately 44 buildings [8]. It should be noted that, in the residential FOA methodology, the target sample size of 63 is established for individual key code items [72] rather than the number of buildings. Since not all code items can be observed during a single site visit, setting up a target sample size based on the number of key code items makes more sense.

The determination of the sample size depends on the variance of the variables of interest. While the CV of key code items in the residential FOA methodology [72] and of code measures in the commercial FOA methodology [97] vary among items, a constant sample size of 63 was adopted in the residential FOA methodology based on a conservative estimation of the CV across key code items and a consideration of simplicity in practice [72]. In contrast, the commercial FOA methodology continues to use different sample sizes for different code measures due to their varying CVs [97].

The determination of the sample size also takes into consideration factors other than the variance of the target variables. In the Northwest methodology, the sample size determination considers the significance of the derived samples at both the state and region levels [64].

The CV used in the sample size determination can be assumed or derived from other means. In the residential FOA methodology, large-scale energy simulations were conducted to obtain the variances of the energy impact of the key code items [72]. In the commercial FOA methodology, the standard errors from the regression analysis of the lost savings based on a range of possible code compliance conditions can be used to update the CV used in the sample size derivation formula [99].

4.2.3. Site Recruitment

With a sample frame in place and a sample selection mechanism, the recruitment of buildings to participate in the study becomes a critical step. However, it is often the most challenging part of the process, as it involves convincing building owner, builders, and code officials to participate. In some cases, jurisdictions may be unwilling to participate, or there may be lack of response from contacted code officials. Additionally, the availability of new construction projects at the time of the study can impact recruitment, resulting in fewer buildings than initially targeted for site visits [57].

To address these challenges, one potential solution is to make participation in code compliance evaluation mandatory [57]. This concept of mandatory requirements in code compliance and enforcement has been discussed in various context. For example, in China, the government implemented measures to enforce mandatory building energy efficiency codes, raising energy code compliance from 10% in 2000 to 100% in 2012 [106], In the European Union, experts have advocated for a greater number of common objectives with mandatory compliance to achieve common zero energy consumption goals [107]. Similarly, researchers have highlighted the enhanced enforcement system for building energy codes in China, emphasizing the importance of detailed requirements for ensuring enforcement and introducing penalties for non-compliance. They suggest that countries like the U.S. could learn from China’s enforcement mechanisms while China could benefit from adopting user-friendly enforcement approaches developed in the U.S. [108].

4.2.4. Data Collection

Data collection includes the plan review documents, field inspections, and interviews.

Plan Review

Plan reviews were a common method used in code compliance evaluation studies to gather building specifications and other compliance information from construction drawings. However, several studies noted the need for more standardization in how compliance was demonstrated in construction and compliance documentation. The drawings were obtained either from the design team or from co-operative jurisdictions where they were available. In some studies, the plan documents were trusted and used as a substitution when field observations were not available [34,50,51,59,63,64]. However, in other studies, plan reviews were not recommended for evaluating code compliance because compliance documents often differed significantly from the original plans [16].

Field Inspection

On-site field inspections were conducted to verify the as-built conditions of the buildings. However, the extent and focus of field inspections varied across the studies. Some studies focused on main systems within the buildings, such as the building envelope, lighting, and mechanical systems [21,22,23,26,28,29,30,33,34,35,56,101]. Others focused on specific building types [56] or on buildings near or at the completion stage [34,63,64].

The residential FOA methodology [72] and the commercial FOA methodology [96] randomized the site selection, allowing field visits to occur at any stage of construction [72,96]. This eliminated the potential bias associated with knowing a building would be visited.

Regarding the recording of field data, there were large variations among the studies. Some studies considered construction design documents as the most reliable data source [34]. In cases where field data were unavailable, some studies assumed that building components not visible after occupancy did not differ significantly from the construction documents [64]. They substituted unavailable field data with plan verified data, and, in some cases, even substituted unavailable plan verified data with code requirement data [50,51,59,63,64]. In the Northwest methodology, the buildings subjected to a site visit are fully constructed and occupied buildings. Some building components are not observable and substitution with values from as-built documentation are expected. In contrast, the residential and commercial FOA methodologies restricted the appliable data to only field-verified data without making assumptions to minimize possible bias [72,96].

Interview

Many studies incorporated interviews as part of their compliance evaluation methodology [21,22,23,26,28,29,30,33,34,35]. These interviews involved a range of questions, from general practices and attitudes related to energy code compliance to specific inquiries about building projects. The interviews were conducted with various parties, included code officials, design teams, and others, depending upon the study goals.

The purpose of these interviews was to gain insights into the experiences and perspectives of architects and engineers regarding compliance with the energy code in their respective states. However, it was found that architects and engineers often provided general information based on their experiences rather than project-specific details. While the information obtained from these interviews can be valuable, it can be challenging to obtain specific projects from architects and engineers after the completion of their projects.

Interviews with code officials were also conducted to gain a deeper understanding of building design, the commercial building marketplace, enforcement activities, and compliance challenges within a jurisdiction. These interviews could also help identify members of affinity groups. However, conducting interviews requires careful attention, and having a structured set of questions is crucial for ensuring the production of useful information.

It is worth noting that, while interviews were a part of many earlier code compliance studies, they are not included in the BECP methodology [18] or the residential and commercial FOA methodologies [72,96].

4.2.5. Code Compliance Evaluation

The analysis of the collected data for code compliance evaluation has undergone significant changes over the years. At a broad level, two types of metrics are used for code compliance evaluation, namely, compliance rate and energy impact due to non-compliance.

Compliance rates, based on the binary pass and fail decision, have been included in most code compliance evaluation studies. These rates can be calculated on a building level, considering all applicable code components in each individual building, or on a code measure level, considering a specific code component across all buildings. Since the Northwest methodology focuses on building systems rather than code components, the so-called binary compliance logic is used to combine code components and assess the compliance of each building system [21,22,23,26,28,29,30,33,34,35,56,101]. Some studies have assigned different points to code items based on their importance in terms of energy impact, such as the one-to-three-point tiers in the BECP 2010 methodology [8]. Partial compliance with fractional points has also been used [59].

While a comprehensive list of code items is included in the checklist of the BECP 2010 methodology [8], a subset of code items with the largest energy impact has been preferred [50,51,52,53,67,68,69,72].

It has been recognized that compliance rates alone do not provide a direct link to the energy impact of code compliance. As a result, compliance rates nowadays are primarily intended to indicate the effectiveness of compliance enforcement rather than serving as a code compliance metric.

The evaluation of the energy impact of non-compliance has been a crucial component of code compliance studies. Building energy models have been developed based on data collected during field visits. Various energy simulation engines, including DOE2 [21,22,23,26,28,29,30], eQuest [58,60], REM/Rate [60], SEEM [50,51,52,53], EnergyGauge [48], EnergyPlus [72,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,99], and others, have been employed for this purpose.

In most studies, data collected from field studies are utilized on a by-building basis, and building models are constructed for each individual building involved in the field studies. Therefore, data substitution inevitably becomes an issue because not all code items can be observed during the site visit of each building. To address this, the concept of pseudo homes is used in the residential FOA methodology [72] to avoid data substitution and associated biases. In this methodology, all data collected on each key code item is treated as an empirical distribution and utilized collectively. A large number of pseudo homes are created using statistical sampling to randomly draw values from the pool of collected data. These pseudo homes collectively represent all newly constructed homes in a state in terms of the distributions of all key code items. The EUI of the suite of pseudo homes provides the energy consumption distribution of new homes, offering a comprehensive picture of code compliance in a state.

Another approach to collectively utilizing the field data from individual field-visited buildings was the construction of composite models [60]. Three composite models, one for each size of stratum of the commercial buildings in the samples, were constructed for the code compliance evaluation based on the energy cost budget. A composite model represents the typical features of buildings within the sample.

There have been changes in the methodologies used in the code compliance evaluation studies conducted in the Northwest region, specifically in the sample design, data collection, and subsequent code compliance evaluation based on building energy performance [56,63,64,101]. There are two key aspects of the changes. Firstly, the buildings included in the field studies have shifted from those under construction to fully constructed and occupied buildings. Secondly, billing data have become an integral part of the data collection and analysis process. The total and end-use EUI of the buildings are obtained by analyzing the billing data, and these EUI values are used to establish a correlation between energy use and code compliance in the buildings.

One potential limitation of this approach is the inability to directly observe certain building components during a field visit to a fully constructed and occupied building. It is assumed that the building plans and documentation accurately represent the as-built condition of the building. Additionally, the disaggregation of the total EUI into end-use EUI relies on the assumption of stable occupancy. These assumptions may or may not be the case.

It is also noted that the energy metrics used in the commercial FOA methodology [96] differ from the ones used in the residential FOA methodology [72]. Instead of using the first year of annual energy and energy cost [72], the life cycle cost savings potential is used [96].

4.2.6. Cost of Code Compliance Evaluation Activities

The cost of code compliance evaluation has been considered in certain studies. In their research, Vine et al. [109] examined the cost of enforcing building energy codes and discovered that the cost varied from typically $50 or less per residential home to nearly $200, and from typically less than $150 per commercial building to over $1000, excluding overhead and travel expenses. This study provided valuable insights into the approximate cost range required to develop and enhance code compliance and enforcement infrastructure.

APEC [47] determined that the average duration of a visit, which included interviews, plan review, and onsite inspections, was 4 h. The average cost per visit was $496.62, excluding travel expenses.

In the commercial FOA methodology, the time spent verifying individual measures was utilized to calculate the lost savings, measured in dollars per verification hour [96].

5. Conclusions and Future Research

5.1. Conclusions

In conclusion, this paper presents a review of code compliance evaluation studies conducted over the past three decades, with a focus on more recent activities. As most of the studies were documented in internal technical reports by the entities conducting them, they have been less accessible compared to peer-reviewed journal articles. This review examined and compared these studies in several key aspects, summarizing their commonalities and differences. The goal is to bridge the gap between the academic research community and the code compliance evaluations conducted by various entities. The following section outlines the major findings of this study:

1.

The widespread acknowledgment of the importance of conducting code compliance evaluation. Builders, architects, and code officials increasingly recognize the benefits of building energy codes. Furthermore, there is now a widespread acknowledgment of the importance of conducting code compliance evaluation studies. The ACEEE publishes an annual State Scorecard to assess the stringency of the residential building energy code (CSit) and evaluate additional code activities (ACit). For instance, in both 2015 and 2016, states had the opportunity to earn up to 3 extra points beyond code stringency by implementing compliance studies based on standardized protocols and a statistically significant sample (up to 1 point), as well as engaging in additional code compliance activities such as training and compliance surveys (up to 2 points) [110].

2.

The ARRA 2009 played a significant role in promoting code compliance evaluation activities in the U.S. While there were code compliance evaluation activities prior to the advent of the ARRA, the code compliance requirement introduced by the ARRA sparked nationwide attention on the energy code and created a nationwide need for a standardized and consistent methodology to conduct code compliance evaluation studies.

3.

The emergence of standardized methodology and protocols. Code compliance evaluation studies have traditionally been focused on meeting short-term and specific needs, with little emphasis on designing standardized and consistent methodologies for the long term and the comparability of results among studies. The code compliance goal for all states within an eight-year timeframe mandated by the ARRA presented an opportunity for developing consistent and standardized methodologies, reducing barriers for states to conduct a code compliance evaluation by eliminating the need to develop state-specific methods from scratch. The BECP 2010 methodology [8] was the first of its kind, and its widespread adoption and testing led to several suggestions for its improvement, resulting in the development of enhanced standardized protocols and tools for conducting code compliance evaluations, such as the BECP FOA methodologies [72,97] and the Northwest [63,64] methodology.

4.

Common steps remain with little change but with an improvement and enhancement in each step. The common steps involved in a code compliance evaluation study have remained largely unchanged, but improvements and enhancements have been made to each of these steps:

a.

In more recent methodologies, rigorous statistical significance has been incorporated into sample designs. Target sample sizes are derived based on predefined statistical criteria, which guide data collection and ensure the credibility of the study results.

b.

Most recent methodologies place more emphasis on random sample selection and strive to avoid self-selection bias.

c.

The principle of objectivity has been enforced in the field data collection and recording in more recent methodologies, such as the BECP’s FOA methodologies, to ensure accuracy and impartiality of the collected data. However, in many studies, it is not uncommon to encounter data substitution when unobservable code item are present in the field studies. Including data that is not directly observed but assumed can compromise the credibility of the studies.

d.

Incomplete data collection is inherent in building field surveys, and data analysis needs to account for it. Code compliance assessments using building simulations are primarily conducted at the individual building level, and data substitution is inevitable when preparing input for building models, intoducing potential biases. The use of large-scale pseudo homes, as demonstrated in the residential FOA methodology [72], provides an effective approach for analyzing limited and incomplete field data. The results from large-scale pseudo homes offer an overview of the state’s average EUI and EUI distribution of new residential homes. However, it does not provide energy-wise code compliance information of each individual home visited during the field survey.

5.

Energy-based metrics used to supplement or replace the compliance rate. It is now widely recognized that the binary compliance rate alone cannot adequately assess the energy impact of non-compliance, although it can still serve as an indicator of code enforcement performance. The focus of code compliance evaluation has shifted from mere compliance verification to methodologies that evaluate the overall effectiveness of energy codes that were created in buildings. In most studies, various types of building energy models were created using field-collected data, and EUI of building components, building systems, and the entire building were used as metrics to evaluate code compliance. More recently, the Northwest methodology has incorporated the collection and analysis of billing data to assess the code’s impact through an energy performance measurement. Unlike EUI derived from building simulations, the total and end-use EUI obtained from billing data analysis represent the actual energy consumed by the buildings. The disaggregated EUIs for major building end-uses are compared with compliance results as specified in the Northwest methodology [63,64]. The use of billing data offers the potential for code compliance based on absolute energy use, similar to the approach used in Europe.

6.

Recruitment is one of the most challenging steps in the code compliance evaluation process, and various issues have been reported in previous studies, including the following:

a.

Sample representativeness: Ensuing that the selected sample of buildings is a representative of the building population is crucial for drawing accurate conclusions. However, difficulties arose in obtaining a truly representative sample due to factors such as self-selection bias, limited access to certain types of buildings, or challenges in obtaining consent from building owners.

b.

Data availability: Code compliance studies often require access to detailed data on building characteristics, energy consumption, and other relevant information. Obtaining comprehensive and reliable data could be a challenge, particularly when building owners are reluctant to share information.

c.

Building owner corporation: The willingness of the building owner to participate in the study and grant access to their properties for a site inspection is essential but can be difficult to achieve. Some studies have found success by implementing certain forms of mandatory actions [106,107].

d.

Time and resource constraints: Conducting code compliance evaluation studies requires significant time, effort, and resources. Recruiting a sufficient number of buildings and completing all necessary data collection and analysis can be time-consuming and costly. To address this, the commercial FOA methodology [99] has been enhanced by using the standard error from the regression analysis of lost savings against possible code compliance condition as the estimate of the coefficient of variation in the sample size formula. This approach allows for lower sample sizes compared to traditional survey-sampling methods, as the regression analysis leverages data across strata rather than using data from a single stratum in the survey-sampling approach.

5.2. Future Research

Code compliance evaluation studies are time-consuming and costly, but they are indispensable because stand-alone or independent code evaluation is the only way to fully understand the status quo of code compliance. Research has been conducted to investigate alternative approaches to the well-recognized labor-, travel-, and cost-intensive in-person inspections for code compliance. Mott et al. [111] explored how virtual inspections, particularly in light of the recent COVID-19 pandemic, have impacted processes for building code compliance checks in jurisdictions and communities worldwide. The authors collected data on four key parameters (time and financial savings, scope of inspections, changing practices and technological innovation, and benefits to consumers) from six jurisdictions and communities in five countries (Australia, Canada, Singapore, United Arab Emirates, and the United States) to analyze the impacts of virtual inspections on code compliance checks. It is important to note that virtual inspections, like most technological developments, are not inherently good or bad, and their effectiveness is highly dependent on how they are implemented. Quantifying the accuracy of an inspection is challenging, and this aspect needs to be addressed in future work to determine if virtual inspections can be equally effective and accurate as in-person inspections.

The prescriptive compliance path is the most widely used compliance path adopted in the U.S. In Europe and other parts of the world (such as China), the performance path was adopted as a common way to evaluate energy performance with the absolute energy use target methodology [112,113]. While this provides a possible approach for evaluating code compliance, it is vital to have accurate buildings’ performance predictions, because a building’s energy use is influenced by various factors, such as time, climate, and buildings’ types and configurations (e.g., height and density), operation and maintenance, and space utilizations [114]. The collection and development of comprehensive databases of buildings’ performance, such as China’s Quota Standard [115] and Energy Star Portfolio Manager [116], which consider various contextual information, might be a critical prerequisite for implementing absolute targets. Additionally, the collection and analysis of billing data in the Northwest methodology [63,64] could provide an opportunity to evaluate code compliance using the absolute energy use target approach.