Empirical Estimation of the Energy Impacts of Projects Installed through Residential Property Assessed Clean Energy Financing Programs in California

Abstract

We examine the energy use impacts of energy efficiency and solar PV projects financed by residential property assessed clean energy (R-PACE) programs in California. We leverage household-level interval meter data to apply normalized metered energy consumption (NMEC) methods at significant scale—more than 25,000 electric meters and more than 15,000 gas meters. We develop a comparison group to account for non-project-related changes in usage. The projects include homes that replaced existing HVAC equipment with higher-efficiency units and homes that installed central heating or air conditioning equipment for the first time. We have limited information on pre-project household equipment stock so we develop a method to infer new installations. We find that projects that installed energy efficiency technologies reduce electricity consumption by approximately 3% and gas consumption by approximately 3.5% on average. When we remove homes that installed new cooling and heating equipment for the first time, savings rise to approximately 5% for electricity and approximately 6% for gas. Given the California climate and the results of an existing study of similar California projects, these results are in line with expectations. Solar PV projects produce electricity that offsets approximately 69% of household electricity consumption on average. We estimate that California R-PACE projects installed through the end of 2019 produce annual reductions in grid-tied electricity consumption of 506 GWh (equivalent to the electricity consumption of approximately 74,000 California households) and gas consumption reductions of 2 million therms (equivalent to the gas consumption of approximately 4700 California households) in a typical weather year.

1. Introduction

Achieving significant improvements in the energy efficiency of existing U.S. residential buildings is a major challenge as service providers (e.g., contractors, architects, and engineers) must often overcome a variety of barriers. Some energy efficiency investments have “higher first costs” compared to conventional measures [1,2]. Examples in the residential market include high-efficiency windows, wall or floor insulation, or a more efficient heating, ventilation, or air conditioning (HVAC) system. These investments deliver energy cost savings and other benefits (e.g., improved comfort and reduced maintenance expenses), although these benefits are realized over long time periods (10–25 years) given the effective useful life of high-efficiency equipment and envelope measures. Solar PV systems are also expensive up front and yield energy cost savings over similarly long time frames. Financing offers an opportunity to better align the timeline of customer costs with energy cost savings for energy efficiency and solar PV projects, enhancing the affordability and attractiveness of these projects.

Many state policy makers and utility regulators have established aggressive energy efficiency savings goals that will necessitate investing billions of dollars in existing buildings over the next decade or two [3]. There is increasing interest among policy makers in promoting financing programs as a way to supplement or develop alternatives to incentives (rebates) that have traditionally been offered in utility energy efficiency programs [4]. These funds are limited, necessitating significant levels of private investment as savings goals increase over time. Policy makers are also interested in approaches that leverage public–private partnerships to support energy efficiency.

Property assessed clean energy (PACE) programs are operating in 22 states for commercial customers and in three states (California, Florida and Missouri) for residential customers [5]. Residential PACE (R-PACE) programs allow owners of single-family and small multifamily (4 units or fewer) residential buildings to finance certain energy- and water-related home improvement measures and repay project costs through a special assessment charge on the property tax bill. (In California, the vast majority of R-PACE customers are single-family building owners). While commercial PACE programs operate in many more states, R-PACE programs have deployed more capital in total, as Florida and especially California R-PACE programs have achieved significant scale relative to other residential energy efficiency financing programs. Per Deason et al. (2016), in 2014, R-PACE programs accounted for nearly half the capital that flowed through energy efficiency financing programs nationwide, despite operating in only two states at the time [6]. R-PACE volumes grew significantly after that time through 2017, though they have declined somewhat in recent years.

Most R-PACE programs are administered as public–private partnerships, with the involvement of state and local governments (who have the authority to place and collect the assessments) and private companies (who source capital for the programs and help administer them). Cumulative investments in residential PACE projects nationwide through 2019 exceeded $6 billion [5].

Figure 1 presents a high-level overview of the R-PACE process and actors. Typically, public partners with the authority to place special assessments issue short-term bonds to fund projects. Private partners then purchases these bonds, and often later sell securities backed by R-PACE payments to the capital markets to replenish their capital.

Several studies estimate the impact of R-PACE program availability in encouraging the deployment of residential solar PV and generally find that the presence of R-PACE programs is associated with increased PV deployment [7,8,9] and increased PV self-generation [10] in jurisdictions that adopt the program while controlling for other relevant factors. Goodman and Zhu (2016) finds that R-PACE participation has a net positive impact on home sale value: the sale value of an R-PACE home, on average, increases by more than the amount of the outstanding assessment [11]. Rose and Wei (2019) and Oliphant et al. (2020) evaluate economy-wide impacts of residential and commercial PACE programs in California and Florida co-administered by Ygrene (a private program administrator and capital provider), finding positive macroeconomic impacts in the hundreds of million dollars in each state over the lifetime of the currently funded projects [12,13].

Our study is the first to conduct an ex post analysis of the energy impacts of R-PACE projects. In contrast to utility-administered energy efficiency programs, financing programs such as R-PACE do not have measurement and verification requirements. This paper also contributes to the limited literature on the impact of financing programs. Financing programs funded by utility customers have generally been treated as non-resource programs, meaning that savings attributable to these programs are rarely calculated [4]. Horkitz et al. (2016) and Stewart et al. (2016) evaluate the HERO R-PACE program, but these studies are akin to process evaluations of utility programs—they explain how well the programs are functioning from a process standpoint, but do not attempt to quantify energy impacts [14,15].

In this paper, we estimate typical California R-PACE household electricity and gas savings as well as aggregate savings for all R-PACE projects in the state through 2018. Our study leverages normalized metered energy consumption (NMEC) methods (see Section 2) to generate building-level energy impacts at a significant scale—more than 25,000 electric meters and more than 15,000 gas meters. We show how the energy impacts vary by investor-owned utility, measure category, and climate zone. Finally, we compare R-PACE energy impacts to those of similar multi-measure energy efficiency programs.

2. Materials and Methods

In this section, we describe the data on energy usage, measures installed, and project costs that we compiled on participating R-PACE households in California.

2.1. R-PACE Projects and Measures

The California Alternative Energy and Advanced Transportation Financing Authority (CAEATFA) administers a loss reserve supporting most R-PACE assessments in the state. CAEATFA collects data on these individual assessments, including financing dates and assessment amounts. We gathered all available data from R-PACE projects submitted to CAEATFA through June 2017. These projects cover all of the major R-PACE programs (

Table 1. California R-PACE program coverage.

Program	Data Coverage
Sonoma County Energy Independence Program (SCEIP)	June 2009–June 2017
mPower	Jan 2010–June 2017
HERO	July 2014–June 2017
CaliforniaFIRST	July 2014–June 2017
Ygrene Energy Fund	September 2015–June 2016 (at the time we gathered data from CAEATFA, data on Ygrene assessments were not available from July 2016 through June 2017)

) and include over 80% of all California R-PACE projects through that same date.

We obtained data on measures financed from each of these programs. Our data on project measures varied by provider; some providers provided a detailed list of installed measures (e.g., floor insulation and duct sealing) while other providers reported installed measures at a much higher level (e.g., “HVAC”). Most of the studied R-PACE programs set minimum efficiency standards for allowable measures that exceeded the minimums required by codes and standards, but in almost all cases, we do not have data on the specific efficiency level of installed measures (e.g., an energy efficiency rating for an air conditioner or an R-value for insulation). In order to harmonize across projects and providers, we developed a standard set of measure categories to which we mapped each project (see

Table 2. Sample size by measure category.

	Measure Category	Share of Electric Meters Analyzed	Share of Gas Meters Analyzed
Energy efficiency—single measure category	HVAC only	23.8% (6075)	22.8% (3629)
	Windows, doors, and skylights only	17.5% (4474)	14.5% (2317)
	Other envelope measures only	16.7% (4260)	14.6% (2325)
	Water heating only	0.9% (231)	0.7% (112)
Energy efficiency—multiple measure categories	HVAC, windows, doors, and skylights	1.5% (393)	1.2% (184)
	HVAC, other envelope	3.3% (848)	2.9% (464)
	Other envelope, windows, doors, and skylights	4.0% (1029)	3.2% (513)
	HVAC, other envelope, Windows, doors, and skylights	0.8% (210)	0.7% (112)
Solar	Solar	20% (5098)	28.5% (4543)
	Solar and EE	3.9% (987)	4.1% (653)
Other measures	Other efficiency	2.8% (724)	2.2% (352)
	Water only	4.6% (1181)	4.6% (737)
Total		25,510	15,941

).

In the case of solar PV projects, we cross-checked the providers’ classification with Berkeley Lab’s Tracking the Sun dataset, which includes all residential grid-connected solar PV systems in the service territories of the utilities in this study [16]. In general, the data on PV installations from PACE providers matched well with the Tracking the Sun data.

Collectively, these measure categories account for all R-PACE projects in our data. Some projects included multiple measure categories (e.g., HVAC, new windows, doors and skylights) with varying combinations of measures. We also observe the distribution of projects by high-level measure category (e.g., energy efficiency, solar PV, energy efficiency measures and solar PV). Approximately 25% of households installed only solar PV, while approximately 65% installed only energy efficiency measures, and less than 5% installed both energy efficiency measures and solar PV.

We summarize the number of projects assigned to each category for our electricity and gas analysis in Table 2. The distribution of projects by measure category is similar between the full set of projects provided to us by PACE providers and the sub-samples of households for which we were able to match utility accounts and estimate reductions in grid electricity and gas use. When estimating total impacts across all R-PACE projects in the Results section, we weight the observed reductions in grid electricity and gas use in our samples to account for differences between the full dataset and the sub-samples of households included in the grid electricity and gas use analysis.

2.2. Utility Electricity and Gas Usage Data for R-PACE Households

We received data on electricity and gas usage for R-PACE households from the four large California investor-owned utilities (IOU): Pacific Gas & Electric Company (PG&E), San Diego Gas & Electric (SDG&E), Southern California Edison (SCE), and Southern California Gas Company (SCG). PG&E and SDG&E are dual-fuel utilities from which we received both electric and gas data; SCE is electric only; and SCG is gas only.

In each case, we gathered all data that the utility in question was able to provide on the list of R-PACE households that were submitted. We were not able to obtain usage data for one or more fuels for some R-PACE households. See Appendix A for more detail on data availability and data cleaning procedures.

2.3. Temperature Data

To adjust for the potential impact of weather, we associated each of our households with a weather station that had sufficient temperature data for both actual and typical weather years. For each R-PACE household, we identified the closest station within 200 km that was in the same climate zone as the household and met our data sufficiency screen. Our screen excluded any station for which 10% or more of the hourly readings in a month were missing for either the historical or typical weather year datasets. If the chosen station’s temperature data fails those sufficiency screens, we move on to the next closest station. If there was no station within 200 km of the R-PACE household that has sufficient data, we discarded the household.

Hourly temperature data for each weather station are from the National Oceanic and Atmospheric Administration (NOAA) [17]. As noted above, we use TMY3 for typical weather year data [18].

2.4. Modeling of Energy Usage Impacts

We estimate the energy impacts of R-PACE projects using normalized metered energy consumption (NMEC) methods. NMEC methods leverage household-level energy usage data, comparing energy usage before and after projects were implemented. They adjust usage to remove the effects of actual weather on energy consumption, and then project post-project usage into a “typical” weather year. Both models used for our calculations—the Princeton Scorekeeping Method (PRISM) and the Time-of-Week and Temperature (TOWT) model—are NMEC models [19,20]. Both models follow the same general procedure for calculating energy savings; they differ in the approach used for weather adjustment, as described in more detail below.

The CalTRACK working group (see https://www.caltrack.org/ (accessed on 1 September 2021)) has been developing standardized methods and software for using NMEC methods in California, driven largely by the usage of these models to estimate savings for pay-for-performance energy efficiency programs. CalTRACK software (see https://github.com/openeemeter/eemeter for Python code implementing the models (accessed on 1 September 2021)) implements both PRISM (using monthly or daily usage data) and TOWT (using hourly data) models in a manner consistent with CalTRACK recommendations. Our approach facilitates comparison with NMEC-based project impacts calculated for other California efficiency programs (see Discussion section for details).

To estimate gas usage impacts, we employ the PRISM model [19]. The model leverages temperature and gas usage data in the year prior to the beginning of the project (the “baseline period”) and the year following completion of the project (the “reporting period”).

Estimation of the PRISM model proceeds at the household level in four steps:

• First, we fit a regression of gas usage on historical heating degree days in the baseline period to estimate the pre-project temperature dependency of the household’s gas usage.
• Second, we fit a regression of observed gas usage on historical heating degree days in the reporting period to estimate the post-project temperature dependency of the household’s gas usage.
• Third, we use the reporting period normal-year heating degree days as inputs into the models produced in the first two steps. The model then estimates post-project usage in a normal weather year and estimates what usage would have been had the project not occurred, using heating degree days in a normal year.
• Fourth, we subtract weather-normalized gas usage during the reporting period from the baseline period to estimate changes in gas usage because of the R-PACE project.

We use a floating balance point method at the household level for choosing the base temperature for defining heating degree days, in keeping with PRISM model best practices [19]. Consistent with CalTRACK methodology, we also fit intercept-only weather models for each period and use whichever model has the higher adjusted r² for each household. (Since the adjusted r² of an intercept-only model is zero, this model is only chosen if the adjusted r² of the HDD model is negative. This might be the case for non-gas-heated households.)

Our method for estimating electricity usage changes employs a different statistical model, but follows a similar process of generating and differencing weather-adjusted baseline and reporting period usage. To estimate electricity impacts, we employ the time-of-week-and-temperature (TOWT) model [20]. TOWT leverages hourly (or more disaggregated) usage data to enrich the PRISM approach to weather-normalization in two principal ways. First, TOWT uses baseline usage patterns to infer whether each hour of the week is generally “occupied” or “unoccupied” and employs this occupancy variable as an additional explanatory variable in its weather regressions. Second, TOWT removes weather effects via a piecewise linear regression model, allowing the relationship between weather and temperature to be somewhat non-linear. Following the CalTRACK implementation of TOWT, we estimate weather dependency separately for each calendar month, rather than for the entire period. (Specifically, we model weather dependence using data from the month in question as well as the month immediately preceding and following that month, again per CalTRACK methods. Data from both shoulder months receive a weight of 0.5 in the regressions, while data from the month in question receive a full weight of 1). The electricity weather fits consider four models: intercept only, HDD only, CDD only and both HDD and CDD. Again, we choose the model with the highest adjusted r² in each month for each household.

We use the TOWT model to analyze electricity usage over time because we have hourly electricity usage data from the three California electric IOUs (see Data and Methodology section for details). We have hourly gas data for SCG, daily for PG&E, and monthly for SDG&E. Due to the lack of hourly data for two of the three gas IOUs, we use the PRISM model to analyze changes in gas usage over time. In order to test whether results between the two models are consistent and comparable, we also ran the PRISM model on a sub-sample of our electricity meters. Results were similar and are available upon request from the authors.

2.5. Defining Project Installation Dates

To estimate pre- and post-retrofit energy usage in our models, we must define project start and end dates. Some energy efficiency projects might take a number of days or weeks to complete, and their impacts might begin to take effect before project completion. We assume that the impact of solar projects on household electricity usage takes effect on the date of system interconnection.

Through discussion of the project implementation, completion, and funding process with each provider, we defined program-specific time windows that we believe encompass and reflect the vast majority of project installations. For energy efficiency projects, these project installation time windows are generally two to four weeks in duration, meaning that we defined the project start date to be two to four weeks prior to the best project completion end date that we received. We do not consider the dates within these project installation time windows when estimating energy usage impacts: the baseline period is the year prior to the start date of the time window and the reporting period is the year following the end date of the time window.

We assume that the impact of solar projects on household electricity usage takes effect on the date of system interconnection. Some solar projects in California were subject to interconnection delays during the time period studied. To address this issue, we used project interconnection dates from Berkeley Lab’s Tracking the Sun dataset [16]. If these interconnection dates were too distant from our R-PACE project dates (more than 60 days before or 120 days after), we discarded the household due to this date inconsistency.

2.6. Selection of Comparison Households

To account for non-weather factors that might influence energy usage over time, we select a comparison group of households. As our usage dataset contains only R-PACE households, we use a prior and future participants method to select our comparison households [21]. Our selection algorithm for comparison households proceeds at the meter level as follows:

• For each R-PACE household, we identify other R-PACE households that meet all of the following criteria:
  • Their project start and end dates are outside the baseline and reporting periods of the household being matched (such that their usage was not affected by projects installed during the time period of comparison);
  • They are in the same zip code and utility jurisdiction at the household being matched (and therefore may be subject to similar factors affecting energy usage, including prices and economic factors);
  • Their usage (for the fuel in question) and weather data pass our data quality and sufficiency checks (see Appendix A for details); and
  • They did not have solar PV installed before or during the baseline and reporting periods of the household being matched (because the presence of PV significantly alters the level and pattern of electricity usage in households).
• Of these potential candidates, we selected the household that had the most similar usage (electricity or gas, as appropriate) during the matched project’s baseline period as measured by the minimum sum of absolute differences in monthly consumption of that fuel over the R-PACE household’s baseline period. This prioritizes comparison households with similar monthly consumption to the matched household.

We ran the selected comparison meters through the appropriate model (i.e., TOWT for electricity, PRISM for gas) using the project dates of the matched household. For some meters, no candidate comparison meters meet all the criteria in step 1 above. We exclude these unmatched meters from our results. We are able to identify comparison meters for approximately two-thirds of our electric accounts and more than 80% of our gas accounts. Appendix A details how this and other exclusions affect our sample size. The change in grid electricity or gas use calculated for these comparison meters represents how consumption in the comparison household changed over the same time period due to non-project factors. Subtracting the comparison household’s change in usage from the R-PACE household’s change in usage yields an estimate of the reduction in grid electricity or gas use attributable to the R-PACE project.

3. Results

The energy use impacts presented here reflect the difference in usage changes between the matched R-PACE and comparison group. In aggregate, the average change in electricity usage for comparison group households was less than 1.5% for each measure category. Separate estimates of energy impacts for R-PACE and comparison households are available upon request.

3.1. Usage Impacts by Utility

Table 3. Average R-PACE project impacts by California IOU.

		Electric			Gas
Measure Category	Utility	Reduction in Grid Electricity Use (%)	Confidence Interval	Sample Size	Reduction in Gas Use (%)	Confidence Interval	Sample Size
EE	PGE	4.8%	0.1%	2491	7.2%	0.2%	3447
	SCE	2.7%	0.0%	12,668	N/A	N/A	N/A
	SCG	N/A	N/A	N/A	2.1%	0.2%	4471
	SDGE	−0.4%	0.1%	3085	3.0%	0.7%	2090
Solar	PGE	68.7%	0.1%	1313	2.0%	0.2%	2321
	SCE	66.0%	0.1%	1574	N/A	N/A	N/A
	SCG	N/A	N/A	N/A	−2.9%	0.7%	831
	SDGE	83.1%	0.1%	2211	−3.1%	1.0%	1391
Solar and EE	PGE	67.5%	0.3%	125	2.6%	0.7%	176
	SCE	57.1%	0.2%	368	N/A	N/A	N/A
	SCG	N/A	N/A	N/A	5.3%	0.9%	154
	SDGE	83.3%	0.2%	494	3.0%	1.8%	323

shows our estimates of the average reduction in grid electricity and gas use for R-PACE households in each measure category for each California IOU. (We use the term “reduction in grid electricity use” here because it includes both electricity savings from energy efficiency measures and avoided grid electricity consumption due to on-site PV generation. We study the change in household load net of PV generation, and (for PV homes) cannot separate avoided grid consumption from efficiency savings, since both reduce net load.) We also present confidence intervals in Table 3. To generate confidence intervals by utility and measure category, we first calculate the confidence interval for each R-PACE household using the method specified in ASHRAE Guideline 14 [22]. Next, we calculate a confidence interval for the difference in savings between the R-PACE and comparison household, assuming independence in the errors from each household [23]. Finally, we aggregate the confidence intervals for these differences to the desired level (e.g., energy efficiency projects in PG&E households) again assuming independence of errors.

For R-PACE households that only installed efficiency measures, electricity savings ranged between 3 and 5% for households located in SCE and PG&E service territory while average electricity usage increased slightly for SDG&E R-PACE customers (0.4%). In analyzing savings for households that installed energy efficiency measures, it is important to note that the vast majority of R-PACE energy efficiency projects affect only space conditioning energy usage, and that the majority of California households are heated by natural gas. (86% of California single-family homes used natural gas as the main source of space heating and only 2% used electricity, 8% used other fuels, and 4% were not heated in 2009, the most recent data available [24].) Moreover, many homes in coastal areas of California enjoy very mild summers and do not have air conditioning. (In 2009 43% of California households were not air-conditioned [25]. Air conditioning saturation varied dramatically by climate zone, from less than 20% in the San Francisco Bay Area, to around 50% in coastal urban areas in southern California, to over 90% in many inland areas [24]. In particular, SDG&E’s service territory is largely coastal, making it difficult to save electricity via upgrades to space conditioning equipment. The negative electricity savings in SDG&E territory may also reflect installation of air conditioning equipment in some households that previously did not have air conditioning.

Average gas usage declined by 2%, 3% and 7% in R-PACE households that are located in SoCal Gas, SDG&E and PG&E territories, respectively. PG&E’s service territory is in northern and central California, where winters—while still mild by national standards—are cooler. Consistent with this, R-PACE energy efficiency projects save more gas usage on average in PG&E territory.

Our metered data analysis shows much larger reductions in grid electricity use from solar PV than from energy efficiency projects. For R-PACE households that installed solar PV, reductions in grid electricity use ranged between 66 and 83% of electricity usage across the three IOUs. Not surprisingly, average reductions in grid electricity use were quite high for these households (7300–8800 kWh per year). SDG&E solar projects generate more electricity on a household percentage basis than those in the other utilities, likely due to a combination of high insolation and lower average pre-project household usage due to low cooling loads. For these homes that installed solar PV, gas usage increased modestly (3%) for homes located in SoCal Gas and SDG&E service territory after installation and declined modestly (2%) for homes located in PG&E service territory. In general, solar projects have little impact on gas consumption, though the impacts vary across utilities in a manner for which we have no immediate explanation.

In PG&E and SDG&E service territories, R-PACE households that installed both solar PV and energy efficiency measures saw average reductions in grid electricity use of similar magnitude to homes that just installed solar PV. However, in SCE territory, reductions in grid electricity use were approximately 10% lower in homes that installed solar PV and efficiency measures compared to PV-only households (57% vs. 66% savings). This result is counter-intuitive and surprising. Given that we find this result only in SCE territory, it may be explained by particular characteristics of these projects. All the joint EE/PV projects in SCE territory were installed by a single R-PACE program. While we cannot test this given our data, it is possible that many of these 368 projects were done by a single R-PACE contractor that may have installed smaller-than-typical PV systems. For this group of homes that installed both solar PV and efficiency measures, we observe reductions in gas use at households in all three service territories, ranging from 2.6% at PG&E to 3% at SDG&E and 5.3% at SoCal Gas.

Figure 2 presents California’s climate zones, and Figure 3 shows our estimates of the average R-PACE project impacts by California climate zone for households that installed various types of energy efficiency measures. Percentage electric savings from energy efficiency projects are highest (4 to 13%) in non-coastal climate zones (4, 12, 13, 14, and 15–Central Valley, Sierra Nevada region, and inland desert areas) which experience warmer summers and therefore higher cooling loads.

Note that the savings for projects in various climate zones include some projects that installed new central AC in households that had previously lacked such units, or installed a gas furnace in a home that had previously used an electric space heater or wood stove. We might expect that a greater share of R-PACE projects would include installation of new HVAC technologies, rather than replacement of existing units, in climates where heating or cooling demand is low and therefore many homes would not have previously had these technologies. Such effects likely explain the negative savings for electricity in climate zone 7 (a temperate coastal Southern California climate zone where air conditioning penetration is low) and for gas in climate zone 15 (a hot desert climate where minimal heating is necessary).

Gas savings are highest in more northern or more inland zones (3, 4, 12, 13, and 14—the greater Bay Area, northern Central Valley, and Sierras). Zone 3 is coastal but far enough north to require more significant space heating in the winter (4% savings); zones 4, 12, 13, and 14 are far enough inland that they also have significant heating demand (6 to 11% average gas savings). Zone 15 is essentially a desert environment, warm enough to not require much space heating despite being inland.

3.2. Usage Impacts by Energy Efficiency Measure Category

Figure 4 shows average energy usage impacts by energy efficiency measure. The results align well with the expected fuel impacts of each technology. For example, building shell measures (other envelope, windows and doors) save on both gas (3–6%) and electricity (3–5%). Average gas usage declined by approximately 12% for customers that installed more efficient water heaters or other related measures. (88% of California single-family households had gas water heaters in 2009 and only 5% had electric water heaters [23]. The increase in electricity usage from water heater projects that we estimate might reflect some switching from gas to electric water heaters, which we cannot directly observe in the data.) In general, multi-category projects have higher savings, especially for gas. For example, households that installed HVAC, envelope measures, and new windows/doors reduced electricity usage by 6% and average gas usage by 10%.

3.3. HVAC Project Savings

The measure data we received from PACE providers generally do not establish whether the HVAC projects in our data installed new heating or air conditioning equipment, both, or neither (e.g., household installed new duct sealing or new whole house fans). Moreover, an unknown fraction of projects that installed HVAC equipment represent households that did not have space cooling equipment (A/C) or central heating systems prior to installation. This may be a larger issue in California’s mild climate than in most parts of the U.S. (As of 2009, approximately 40% of California single-family households did not have air conditioning and approximately 14% did not have heating systems, leaving room for penetration of new systems. Nationwide, only 4% of households are not heated and only 17% lack air conditioning [24,25]. In the case of heating, some R-PACE households may be installing new gas- or electric- heating equipment that is replacing or supplementing wood or propane heating.)

As such, the average result for each fuel is based on an agglomeration of projects expected to reduce usage of that fuel (e.g., a furnace replacement’s impact on gas usage); projects expected to have little impact on that fuel (e.g., an air conditioning project’s impact on gas usage); and projects expected to increase usage of that fuel (e.g., a new gas-fired furnace installed in a house that did not have gas heating equipment). Thus, our HVAC results should not be interpreted to mean that the average air conditioning replacement reduced electricity use by only 4%, or that the average gas furnace replacement reduced gas usage by only 2% (see Figure 4).

Figure 5 and Figure 6 show distributions of impact estimates for electricity and gas usage for R-PACE project and comparison households that installed HVAC only measures. Note that households with negative savings in Figure 5 or Figure 6 increased electricity or gas usage, respectively, after the R-PACE project was installed. Households with positive savings decreased their electricity or gas usage. The distribution of “impacts” for comparison households (shown in gray shading)—which did not implement projects in the time window analyzed—gives a sense of the natural variability of year-on-year household energy usage due to non-project factors, even after we have normalized for the effects of weather.

Figure 5 and Figure 6 show that a greater share of comparison HVAC households than R-PACE HVAC households had energy impacts within 10% of baseline usage than project households. A total of 67% of comparison households had changes in usage between 𢄬10% and 10% of baseline usage, whereas only 50% of R-PACE households had usage changes in that range. A greater share of R-PACE HVAC households than comparison households decreased their consumption. A total of 27% of R-PACE households saved more than 10% of baseline electricity usage as compared to 18% of comparison households. However, a higher percentage of R-PACE households also increased usage after the R-PACE project. A total of 22% of R-PACE households increased electricity usage by 15% or more relative to their baseline usage (“negative savings”) compared to only 15% of comparison households. We find similar patterns among HVAC-only households in our gas analysis.

The greater share of R-PACE households on the far left side of the distributions suggests that these projects installed a new furnace or air conditioning equipment and did not have this type of equipment prior to the project. We do not find higher shares of project households on the left-hand side of the distribution for other measure categories. This finding is consistent with R-PACE program rules in California, which allow for installations of new HVAC equipment irrespective of prior HVAC equipment practices.

To gain greater clarity on HVAC impacts, we use an algorithmic method to infer which projects may be installing new HVAC equipment and did not have such equipment prior to the project. We focus on seasons and times of day that are most likely to reveal the actual context and baseline conditions underlying projects that installed HVAC equipment. Specifically, we examine each HVAC-only project’s electricity usage impact from noon to midnight in the summer months of July through September (where we would expect air conditioning to be used). If weather-normalized electricity use increases by more than 15% during these hours, we infer that the project includes a new air conditioning installation. If weather-normalized electricity use decreases by more than 10% during these hours, we infer that the project includes a replacement of existing air conditioning equipment. Projects with usage changes that fall between these values are ambiguous or may not have included a cooling-related efficiency measure at all (e.g., only measures affecting gas usage were installed). For natural gas, we apply similar logic to all usage during the winter months of December to February. An increase in gas consumption of 25% after the R-PACE project was completed is our threshold for inferring installation of a new gas furnace in a household that previously did not have gas central heating equipment. If gas usage for HVAC-only projects decreased by at least 10%, then we assume that a new gas furnace replaced existing gas-fired equipment. We define these cut-off points from the data underlying Figure 3 and Figure 4. We choose cut-off points near the parts of the savings distributions where the share of project households begins to be greater than the share of comparison households, suggesting that values at and beyond these cut-off points are more likely to reflect project effects than other factors. Figure 3 and Figure 4 show full-year results. By focusing on seasonal time periods where HVAC effects ought to be more pronounced, we are more confident that the observed effects are related to installed measures in projects. For example, we guard against mis-identifying electric heat impacts as air conditioning impacts. Note that we restrict our inquiry to air conditioning and gas heat, so are not considering electric heat (such as via an air-source heat pump).

Figure 7 shows annual savings estimates for HVAC-only projects categorized in this manner, along with the sample sizes of each inferred category. The data suggest that many R-PACE HVAC-only projects may be installations of new equipment rather than one-for-one replacement of existing types of HVAC equipment. Among projects we classify as A/C equipment replacement, average annual electric savings are 15%. Among projects we classify as heating equipment replacement (e.g., furnaces), average annual gas savings are 23%. For the projects that we classify as having installed new equipment in households that did not previously have that equipment, usage increased significantly—by 16% on average for inferred cooling equipment installs and by 40% on average for inferred heating equipment installs (e.g., furnaces). In general, the impact of inferred new cooling equipment projects on gas use and of inferred gas heating projects on electricity use are small (1–2%), as would be expected.

We view this effort to characterize the underlying baseline condition of HVAC projects as exploratory, yielding approximate distributions, because our algorithm likely mischaracterizes some projects.

3.4. Usage Impacts across the R-PACE Portfolio

In order to estimate average usage impacts from all R-PACE projects, we calculate a weighted average of the estimated impacts by utility. We weight by the sample size of R-PACE projects installed in each utility service territory to account for variation in sample size attrition from data sufficiency checks by utility (see Appendix A for details).

Table 4. Average reductions in grid electricity and gas use by type of project in California R-PACE programs.

Measure Category	% Reduction in Grid Electricity Use	% Reduction in Gas Use
Efficiency	2.9%	3.5%
Efficiency without inferred new HVAC installations	4.9%	6.0%
Solar	68.9%	−1.7%
Solar and EE	63.6%	4.3%

summarizes the average energy impacts by type of project.

Our results suggest that, on average, an R-PACE-financed energy efficiency project reduces a household’s electricity usage by approximately 3% and gas usage by approximately 4%. If we exclude those energy efficiency projects that we infer to be installations of new HVAC equipment (rather than replacement upgrades of existing equipment), then average household savings increase slightly to 5% for electricity and 6% for gas. For those projects that installed solar PV only, the reduction in grid electricity usage averaged 69% across households while average gas usage increased slightly (1.7%).

4. Discussion

4.1. The California Context

Much of California, especially the heavily-populated coastal part of the state where most R-PACE projects are located, has a much milder climate than most other parts of the U.S. As a result, space conditioning accounts for a relatively small share of California’s energy use. According to the U.S. Energy Information Administration (EIA), in 2009, gas space heating accounted for 39% of California’s household gas use, while space cooling accounted for 10% of household electricity use [26]. For comparison, these values are 63% and 15%, respectively, for the U.S. as a whole [27]. Per Table 2, the vast majority of R-PACE energy efficiency projects affect space conditioning energy use only. Therefore, potential savings from space conditioning-related energy efficiency measures are lower in California, both in absolute and percentage terms, than they would be in the country as a whole.

Per EIA, 15% of California households were not heated and over 40% did not use air conditioning in 2009 [25]. Air conditioning penetration in particular has been increasing in California [24]. Unlike some other energy efficiency programs, R-PACE programs can finance air conditioner and heating unit installations. Our data suggest substantial installation of both new air conditioning and heating units in California households that may not have had this type of equipment previously. Installation of these measures tends to result in increased energy use, but would not likely be as common for R-PACE programs implemented in parts of the country where heating and cooling units are more ubiquitous.

Due to the combination of these two factors, similar R-PACE programs in other parts of the country might be expected to achieve greater energy efficiency savings, both in percentage and absolute terms.

4.2. Comparison to Home Upgrade Program and Advanced Home Upgrade Program Impacts

To put our results in context, we compare them to a recent evaluation of California’s Home Upgrade Program (HUP) and Advanced Home Upgrade Program (AHUP). DNV GL (2019) evaluated electricity and gas savings in households participating in HUP and AHUP from mid-2016 through mid-2017 and used similar energy analysis methods [28]. HUP and AHUP are whole home retrofit programs that provide incentives for the installation of multiple measures. The programs focus on similar efficiency end uses to R-PACE and are available statewide in IOU service territories. HUP projects consist of at least three measures, including at least one of the following measures: air sealing (other envelope in our rubric), attic insulation (other envelope in our rubric), or duct sealing (an HVAC measure). Other potential measures are envelope, HVAC, and water heating related. AHUP projects consist of at least four (in some places at least five) measures, and also includes window replacements.

These programs address similar end uses to R-PACE, although HUP and especially AHUP projects are likely to be more comprehensive than R-PACE projects on average. Unlike AHUP projects, R-PACE projects do not have a minimum number of measures. As shown in Table 2, most R-PACE energy efficiency projects’ measures fall into a single measure category (though they may include multiple measures within that measure category—our data are not granular enough to reveal this). Moreover, HUP and AHUP do not allow installation of new HVAC equipment in homes that do not have air conditioning; they only target households that replace an existing HVAC unit with higher-efficiency equipment.

Table 5. Average electricity and gas savings of HUP, AHUP, and R-PACE projects.

		HUP Average Savings	AHUP Average Savings	R-PACE Average Savings, Energy Efficiency Projects	R-PACE Average Savings, Inferred New Installations Excluded
Electric utility	PG&E	8%	5%	4.8%	6.1%
	SCE	2%	6%	2.7%	4.7%
	SDG&E	0%	*	−0.4%	2.7%
Gas utility	PG&E	10%	11%	7.2%	9.3%
	SCG	5%	15%	2.1%	4.8%
	SDG&E	8%	*	3.0%	4.9%

* DNV GL did not report savings for SDG&E AHUP projects due to small sample size.

shows HUP, AHUP, and our R-PACE energy efficiency results by utility.

On average, per-household electric savings were slightly higher for HUP and AHUP participants compared to R-PACE households that implemented efficiency measures. Average gas savings were significantly higher for HUP and AHUP participants. When considering our R-PACE estimates with inferred installations removed, both gaps close somewhat, with average R-PACE electric savings in particular appearing to be comparable to average savings in the HUP/AHUP program. Given the likely differences in the mix and frequency of measures installed in typical projects for each program, our energy impact results for R-PACE appear to be reasonable.

During this time period, R-PACE participation rates were considerably higher than HUP and AHUP. A total of 5700 households participated in HUP in these 12 months and 3500 in AHUP. In contrast, R-PACE programs had approximately 60,000 participants, of which we estimate that approximately 43,000 installed energy efficiency measures. Therefore, in aggregate, total savings from energy efficiency measures in R-PACE households in California during this time are higher than from HUP and AHUP programs.

4.3. Program-Level Impacts

Table 6. Estimated R-PACE program first-year and lifetime savings.

Project Category	Average Relative Household First-Year Electricity Savings	Average Absolute Household First-Year Electricity Savings (kWh)	Total First-Year Electricity Savings (GWh)	Total Lifetime Electricity Savings (TWh)	Average Relative Household First-Year Gas Savings	Average Absolute Household First-Year Gas Savings (Therms) 1	Total First-year Gas Savings (Million Therms)	Total Lifetime Gas Savings (Million Therms)
Energy efficiency	2.9%	245	35	0.7	3.5%	16	2.3	44
Solar PV	68.9%	7,937	414	7.9	−1.7%	−9	−0.5	−9
EE + PV	63.6%	6,542	59	1.1	4.3%	21	0.2	4
Water only	−2.4%	−209	−2	0.0	−2.8%	−13	−0.1	−3
Total	25%	2,352	506	9.6	1.9%	9	2.0	37

¹ One therm is approximately 105.48 megajoules (MJ). In SI units, the average household adopting energy efficiency measures saved approximately 1700 MJ in gas use in the first year. Total R-PACE program first-year gas savings were 211 terajoules (TJ) and total R-PACE program lifetime savings for projects implemented through 2019 were 3900 TJ.

summarizes our estimates of total California R-PACE program impacts through 2019. We estimate normal year savings of 35 GWh and 2.3 million therms for projects that installed energy efficiency measures, and total savings of 0.7 TWh and 44 million therms over the full lifetimes of these efficiency projects. Savings are greater for projects with measures in multiple measure categories and in areas of the state with high space conditioning demand. We estimate the projects installing solar PV (including those with EE measures as well) reduce grid electricity consumption by 473 GWh annually and 9 TWh over the full project lifetimes, with very small impacts on gas consumption. Across all R-PACE projects implemented through the end of 2019, we estimate first year reductions of 506 GWh of grid electricity consumption and 2 million therms of gas consumption, and lifetime reductions of 9.6 TWh and 37 million therms, respectively. The reduction in grid-tied electricity consumption is electricity equivalent to the electricity consumption of approximately 74,000 California households, and the gas savings are equivalent to the gas consumption of approximately 4700 California households.

4.4. Adoption of Energy Efficiency Equipment and Solar PV in Financing Programs

Our study also provides some insight on customer choices and contractor preferences regarding combinations of higher-efficiency equipment, envelope measures and solar PV in California residences. R-PACE provides opportunities for service providers to bundle various demand-side strategies as part of a comprehensive service offering to customers. However, only approximately 5% of the households in our study chose to combine energy efficiency measures and solar PV. It appears that most contractors focus on providing various types of energy efficiency services distinctly from installing solar PV systems rather than bundling higher-efficiency technologies and solar PV. Homeowner decisions to replace existing HVAC and water heating equipment are often driven by equipment failure or the end of useful service lifetime while decisions to install new windows, doors, and envelope measures may be more linked to home remodeling projects; this may account for the challenges of scaling up a home performance contractor market in the U.S. that offers comprehensive efficiency and solar PV packages.

4.5. NMEC at Scale

Our analysis demonstrates the power of NMEC methods at scale. These methods have the immediate virtue that they are ex post—based on actual consumption rather than ex ante engineering estimates. The building-level estimates we generate allow thorough parsing of the impacts—by measure category, climate zone, and utility jurisdiction. Results generated by this parsing in our dataset are quite reasonable. Moreover, while household-level NMEC results can be uncertain, aggregating results across large numbers of households yields precise estimates of collective energy impacts. NMEC methods are being increasingly used for program impact evaluation, and we believe this analysis demonstrates their power for these applications.

NMEC methods do have limitations. They generally do not account for other changes in the household that occur in the baseline or reporting periods. For example, we cannot observe changes in occupancy; if one or more people move into or out of the household during the period of observation, this may influence estimates of electricity and/or gas savings. Control groups can account for both of these issues to some extent by differencing out any factors that would be expected to equally affect R-PACE and similar non-program households. We have no reason to believe that occupancy changes would vary systematically between R-PACE and comparison households, thus our comparison groups likely resolves any issues with occupancy changes. In another case, if a homeowner adds floor area as part of an addition at the same time as a small R-PACE energy efficiency project, we will likely estimate an increase in usage even if the project reduced usage on its own. Such impacts—those that specifically tend to be associated with undertaking the energy project—are harder to handle with control groups, and we do not attempt to do so here.

The lack of detailed project data can limit the interpretation of energy impacts. We do not have data on the existing building and equipment stock or the condition of existing equipment in R-PACE households prior to the R-PACE projects. The likely presence of many installations of new equipment (rather than replacement of existing equipment) is one clear case where the baseline HVAC equipment prior to the project makes a critical difference in our interpretation of the savings results. Ultimately, these issues could be addressed in more data-rich environment.

Finally, NMEC methods cannot address the attribution of energy impacts to programmatic interventions. We estimate the energy impacts from R-PACE projects, but cannot determine whether the presence of R-PACE financing caused the projects to happen. Presumably, some households would have pursued similar projects absent R-PACE. These projects could have been more or less energy efficient than the ones we analyze. However, several analyses of R-PACE and solar PV have shown that the presence of R-PACE has been associated with greater PV deployment. Deason and Murphy (2018) [9] discuss this issue further, including considerations for extending this result to energy efficiency.

5. Conclusions

We implement NMEC methods at significant scale to provide the first energy impact estimates for an R-PACE program. Despite the considerable variation in non-project drivers of residential energy use, NMEC in this application is capable of delivering detailed information about project performance. We find moderate electricity and gas savings from R-PACE energy efficiency projects in California. Our results are sensible when broken down by type of measure, utility jurisdiction, and climate zone (see, e.g., Figure 1 and Figure 2). In the context of California’s mild climate and R-PACE’s focus on space-conditioning projects, the savings are reasonable. California’s mild climate limits the savings attainable by R-PACE energy efficiency projects—which mostly affect space conditioning and not other end uses—in both absolute and percentage terms. In other parts of the country, savings from projects that installed similar types of efficiency measures would likely be larger in both absolute and percentage terms. R-PACE programs in Florida have achieved significant scale, and a study of these programs using similar methods would help understand impacts in a different climate.

Given our sensible results, the increased prevalence of NMEC methods in program evaluation and performance appraisal seems warranted. Further work on assessing energy efficiency and other demand-side programs using similar methods could enrich understanding of program impacts. For example, Murphy and Deason (2021) [29] use NMEC methods on the same dataset studied here to estimate time-of-day impacts of R-PACE energy efficiency projects. These projects provide extra value to the grid due to the timing of their impacts, largely because the savings they provide are concentrated in peak hours.

R-PACE programs finance installation of new HVAC units as well as replacement of existing units with higher-efficiency equipment. We develop an approximate algorithmic method for identifying these projects, and our analysis suggests that savings are higher if these projects are set aside. These types of data gaps on measures—particularly on pre-project measures—are common in programmatic data. Our simple algorithm is a first step towards addressing these issues, but we could imagine considerable additional work on this front.

While we neither consider the cost-effectiveness of R-PACE projects from a societal or participant perspective nor assess the consumer protection issues associated with R-PACE, our analysis demonstrates the significant scale of R-PACE program impacts on energy usage in California.