Probabilistic HVAC Load Forecasting Method Based on Transformer Network Considering Multiscale and Multivariable Correlation

Abstract

Accurate load forecasting for community-level heating, ventilation, and air conditioning (HVAC) plays an important role in determining an efficient strategy for demand response (DR) and the operation of the power grid. However, community-level HVAC includes various building-level HVACs, whose usage patterns and standard parameters vary, causing the challenge of load forecasting. To this end, a novel deep learning model, multiscale and cross-variable transformer (MSCVFormer), is proposed to achieve accurate community-level HVAC probabilistic load forecasting by capturing the various influences of multivariables on the load pattern, providing effective information for the grid operators to develop DR and operation strategies. This approach is combined with the multiscale attention (MSA) and cross-variable attention (CVA) mechanism, capturing the complex temporal patterns of the aggregated load. Specifically, by embedding the time series decomposition into the self-attention mechanism, MSA enables the model to capture the critical features of time series while considering the correlation between multiscale time series. Then, CVA calculates the correlations between the exogenous variable and aggregated load, explicitly utilizing the exogenous variables to enhance the model’s understanding of the temporal pattern. This differs from the usual methods, which do not fully consider the relationship between the exogenous variable and aggregated load. To test the effectiveness of the proposed method, two datasets from Germany and China are used to conduct the experiment. Compared to the benchmarks, the proposed method achieves outperforming probabilistic load forecasting results, where the prediction interval coverage probability (PICP) deviation with the nominal coverage and prediction interval normalized averaged width (PINAW) are reduced by 46.7% and 5.25%, respectively.

1. Introduction

The climate change problem in the 21st century, which is attributed to large amounts of CO₂ emissions, has attracted much attention around the world which is attributed to large amounts of CO₂ emissions [1,2]. Based on the statistical data of World Development Indicators from 1990 to 2020, the top five fossil fuel-consuming countries account for 55.43% of global CO₂ emissions [3,4]. To solve the environmental problem and reduce the reliance on fossil fuels, the fuel consumption pattern conversion from fossil fuels to renewable energy is considered to be effective [5,6]. Meanwhile, recent research has found that improving the utilization proportion of renewable energy can mitigate the adverse influence of climate change [7]. In this context, more and more nations are commencing the reconfiguration of energy usage patterns, constructing low-carbon and sustainable cities. However, the generation characteristics of renewable energy, such as intermittency and low inertia, pose a challenge for the energy usage pattern conversion. Specifically, the intermittency implies a sharp fluctuation in the power supply in the power system [8]. The low inertia implies that the power system with high-penetration renewable energy will suffer a sharp reduction in voltage and frequency when the power shortage appears, damaging the stability of the power system. To this end, demand response (DR) provides a scalable solution, which adjusts the load demand of controllable loads to make the power demand adapt to the power supply fluctuation caused by the renewable energy, thereby ensuring the safety of the power system [9,10,11,12].

For these controllable loads participating in the DR, heating, ventilation, and air conditioning (HVAC) systems are the crucial appliances participating in the DR due to their unique characteristics. Their energy consumption occupies a large proportion of the building’s total consumption but could be effectively reduced by guiding HVAC to work in varying modes [13,14]. On the other hand, HVAC is utilized to improve the comfort level of the environment. According to the thermal inertia of a building, it can maintain the comfort level of the environment when HVAC is transferred into low energy consumption mode or turned off. For this characteristic, a large number of published studies have investigated effective control strategies and proved their enormous potential in reducing energy consumption [15,16,17]. Therefore, accurate energy consumption forecasting for HVAC in buildings is essential for constructing the power system with high-penetration renewable energy and mitigating the energy shortage and climate change problem [18,19], and it is the main research purpose in this paper.

Nowadays, the load forecasting of HVAC can be divided into two categories: the physical model-based method and the data-driven method. The physical model-based method is also called the white-box model, which constructs the mathematical model considering various variables, such as the temperature, the physical characteristics, and the occupancy rate, to calculate or forecast the HVAC load. This method provides great interpretation for the forecasting results [20]. However, the physical model is built based on some assumptions, which makes this model fail to reflect the real operation state of HVAC. Meanwhile, the physical model cannot capture the temporal pattern of HVAC, which leads to a low accuracy when an emergency appears [21,22]. With the development of smart meters and artificial intelligence, many researchers have investigated the employment of data-driven methods in HVAC load forecasting, which has low requirements for the building but analyzes the usage pattern of historical HVAC load. Ref. [23] utilized a temporal convolutional network (TCN) to construct the long-term temporality of HVAC load, thereby achieving determined load forecasting. Ref. [24] added the constraint of physical variables in the training of neural networks (NNs), enhancing their accuracy. Ref. [25] extended the HVAC load forecast from single-step forecasting to multi-horizon forecasting, conforming to the actual requirements for load forecasting. However, both the physical-based and data-driven methods focus on the building-level instead of the community-level load forecasting. For the community-level HVAC, because it includes HVACs from many households, where the usage pattern and standard parameters have some differences, analyzing its characteristics and forecasting its load accurately are challenging. Meanwhile, limited by the high cost of smart meters and the risk of privacy intrusion, it is difficult to acquire the measurements of HVAC.

To this end, some researchers have employed nonintrusive load monitoring (NILM), which decomposes the device load from the aggregated load, including varying the device load [26,27,28,29,30,31,32,33]. Ref. [29] combined multiple machine learning approaches, forecasting the state and load of HVAC. Ref. [30] employed a single processing technique in deep learning, decomposing the charge curve of an electric vehicle from the total electricity consumption. However, some research gaps exist, including the following:

(1)

Multiscale temporal patterns are commonly extracted, while the potential correlation between them remains unmodeled, limiting the capture of the accurate HVAC load fluctuation.

(2)

Exogenous variables are uniformly encoded by the NN, which neglects the intrinsic dependencies, preventing the modeling of complex HVAC load temporal patterns.

(3)

Probabilistic HVAC load forecasting is usually ignored in NILM, which is crucial for determining an efficient scheme of DR.

To this end, aiming at constructing a model that considers multiple factors, including the historical load data and exogenous variables, to forecast the HVAC load distribution in the future, this paper proposes the multiscale and cross-variable transformer (MSCVFormer). It employs the cross-variable attention (CVA) mechanism, capturing similar patterns between the exogenous variable and endogenous variable, thereby filtering potential patterns in the time series of the aggregated load series. To fully explore the temporality of time series, the multiscale attention (MSA) mechanism is used to handle load series with varying resolutions. Finally, the pinball loss is expanded to a multi-horizon pinball loss to achieve the prediction interval (PI) of HVAC, thereby evaluating the uncertainty of the predicted HVAC load. The main contributions of this research are as follows.

(1)

The MSA mechanism is proposed, which allows the model to capture the correlation between multiscale historical data. Specifically, it decomposes the historical data into multiple components with varying time scales, followed by single-scale and cross-scale attention mechanisms to capture their correlation and concretize them as attention scores. These attention scores enable the model to excavate the complex usage pattern of HVAC. This is in contrast to those models, which forecast the future data of each decomposition and obtain the prediction through addition.

(2)

The CVA mechanism is proposed, which enables the model to analyze the relevance between the exogenous variable patterns and HVAC load patterns. Specifically, it calculates the similarity of temporal patterns between the exogenous variables and aggregated load, which is further concretized as the weight matrix to extract the task-specific features from the aggregated load. Unlike conventional models that treat all input variables uniformly, the CVA mechanism explicitly considers the distinct temporal patterns, thereby preserving their unique predictive signatures.

(3)

The proposed model combines deep learning and quantile regression to achieve day-ahead probabilistic forecasting of the HVAC load. The parameters of the proposed model are optimized through the pinball loss, and the full load distribution of HVAC is learned by forecasting the multiple quantiles of HVAC simultaneously.

2. Problem Formulation

Multi-horizon HVAC probabilistic load forecasting aims to forecast the future HVAC load distribution through the historical load and exogenous variables, which can be formulated as Equation (1) [34,35].

$${\left[{\widehat{\mathit{y}}}^{q_1},\dots ,{\widehat{\mathit{y}}}^{q_k},\dots ,{\widehat{\mathit{y}}}^{q_n}\right]}^T=f\left(\mathit{x}|\mathit{\theta }\right)$$

(1)

where $f(\cdot |\mathit{\theta })$ represents the mapping function of the MSCVFormer model. $\mathit{\theta }$ represents the MSCVFormer model’s parameters. Assume the time sequence $\mathit{x}=\{{\mathit{x}}_1,\dots ,{\mathit{x}}_l,\dots ,{\mathit{x}}_L|{\mathit{x}}_l\in {ℝ}^d\}$ as the input of the model, where the time length is $L$. For the features ${\mathit{x}}_l$ at each time stamp, it includes the historical data of the aggregated load and exogenous variables, formulated as $x_l$ and ${\mathit{w}}_l$, respectively. ${\mathit{w}}_l=[w_l^1,\dots ,w_l^{d-1}]$ represents $d-1$ exogenous variables. $\left[{\widehat{\mathit{y}}}^{q_1},\dots ,{\widehat{\mathit{y}}}^{q_k},\dots ,{\widehat{\mathit{y}}}^{q_n}\right]$ represent the predicted quantile sequence of the future HVAC load, where the number of quantiles is $n$. For the $q_k$-th predicted quantile sequence, ${\widehat{\mathit{y}}}^{q_k}=[{\widehat{y}}_{L+1}^{q_k},{\widehat{y}}_{L+2}^{q_k},\dots ,{\widehat{y}}_{L+H}^{q_k}]$, its time sequence length is $H$. Through the sliding window approach with a fixed-size window, the training and test sets are obtained, where the training set is used to optimize $\mathit{\theta }$ and the test set is used to evaluate the performance of the constructed model.

3. Methodology

As shown in Figure 1, the proposed method mainly includes two parts: variable selection, where the exogenous variables are used as the input of the MSCVFormer model, and the MSCVFormer. Their concrete content will be described in detail below.

3.1. Variable Selection

Historical load data are typically used to predict future load. However, in contrast to the conventional electricity load, the load patterns of HVAC also exhibit strong relevance to external environmental factors. For instance, high ambient temperatures often lead to lower temporary setpoints of HVAC, while high humidity levels may trigger increased use of ventilation modes. In this case, incorporating meteorological and environmental variables into the model can improve the forecasting accuracy. Nevertheless, not all potential input variables are strongly correlated with the HVAC load pattern. The variables with a weak correlation may introduce noise, which is harmful to the model’s performance. To address this issue, variable selection was conducted by calculating the Pearson Correlation Coefficient (PCC) [23,25], which detects the linear relationship between external variables and HVAC load. Meanwhile, the HVAC load at the same time stamp may exhibit a similar load distribution. Therefore, the Autocorrelation Coefficient (ACC) was also calculated to capture temporal dependencies in the historical load itself [36].

Specifically, the inputs of PCC and ACC are two time sequences. Their concrete calculation processes are as follows.

$$PCC={\displaystyle \frac{{\displaystyle \sum _{i=1}^m\left(X_i-\overline{X}\right)}(Y_i-\overline{Y})}{\sqrt{{\displaystyle \sum _{i=1}^m{\left(X_i-\overline{X}\right)}^2}{\displaystyle \sum _{i=1}^m{\left(Y_i-\overline{Y}\right)}^2}}}}$$

(2)

$$ACC={\displaystyle \frac{{\displaystyle \sum _{i=1}^{m-lag}\left(X_i-\overline{X}\right)}\left(X_{i+lag}-\overline{X}\right)}{{\displaystyle \sum _{i=1}^m{\left(X_i-\overline{X}\right)}^2}}}$$

(3)

where $X_i$ and $Y_i$ represent the i-th values of time sequences belonging to the HVAC load and external variable, respectively. $\overline{X}$ and $\overline{Y}$ represent the average value of the time sequence. $m$ is the length of the time sequence. $lag$ is the time lag of the ACC. The range of PCC is from −1 to 1. When the absolute value of PCC is close to 1, it indicates a strong linear correlation between two variables. Otherwise, it indicates a weak linear correlation when the absolute value of PCC is close to 0. The range of ACC is from −1 to 1. When the absolute value of ACC is close to 1, it indicates an obvious periodicity pattern, such as the day-scale or week-scale periodicity, where the periodicity scale is equal to $lag$. Through the calculation of PCC and ACC, the variables selected as the input of the model can be determined.

3.2. Multiscale and Cross-Variable Transformer

Figure 2 displays the architecture of the proposed model, MSCVFormer. It can be divided into three parts, including the MSA blocks capturing the complex temporal patterns of time series from the weather variables and the aggregated load, as well as the CVA blocks exploring the temporal pattern representation of the aggregated load, which is related to the weather variables. Finally, multiple multilayer perceptions are utilized to achieve the multi-horizon quantile regression.

3.2.1. Multiscale Attention Mechanism

To learn the complex temporal pattern of time series, multiple time series decompositions are employed, such as Empirical Mode Decomposition [37] and Seasonal and Trend Decomposition using Loess (STL) [38], which decompose the raw series into multiple sub-series and forecasts them, followed by the additive model or the multiplicative model to integrate the predictions. However, these separate predictions ignore the potential correlation between these sub-series. Even if these sub-series are handled by the NN simultaneously, they are treated equally and ignored in their relationship, which could affect the performance of the NN [38]. To explicitly capture the potential correlation between these sub-series and enable the model to focus on the critical information for the current task, the MSA Block is proposed, whose details are shown in Figure 3.

In the proposed network, multiple MSA blocks are cascaded to capture the temporal pattern of the time series. Assume the input of MSA blocks, $X^{i,MSA}$, where $X^{i,MSA}\in {ℝ}^{\mathrm{B}\times \mathrm{L}\times d_{\mathrm{model}}}$. $\mathrm{B}$ represents the number of mini-batches. $d_{\mathrm{model}}$ represents the length of the feature at each timestamp. Referring to the decomposition of STL, the average pool layer is utilized to produce the seasonal and trend components of the raw time series, regarded as $X_{season}^{i,MSA}$ and $X_{trend}^{i,MSA}$ and formulated as Equations (4) and (5), respectively. $k_s$ and $k_l$ represent the size of the averaging window, where $k_l$ is larger than $k_s$. To keep the length of the features consistent, $\mathrm{P}\mathrm{a}\mathrm{d}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}\left(\cdot \right)$ is utilized to supply 0 at both ends of the input. The residual components, $X_{residual}^{i,MSA}$, are obtained through Equation (7). For the multiscale sub-series, $\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}\left(\cdot \right)$, formulated as Equation (7), is utilized to concatenate them, thereby obtaining a comprehensive representation of the time series, $X_{complex}^{i,MSA}$, where $X_{complex}^{i,MSA}\in {ℝ}^{\mathrm{B}\times \mathrm{L}\times 3{\mathrm{d}}_{\mathrm{model}}}$.

$$X_{season}^{i,MSA}=\mathrm{AvgPool}(\mathrm{Padding}(X^{i,MSA}),kernel=k_s)$$

(4)

$$X_{trend}^{i,MSA}=\mathrm{AvgPool}(\mathrm{Padding}(X^{i,MSA}),kernel=k_l)$$

(5)

$$X_{residual}^{i,MSA}=X^{i,MSA}-X_{season}^{i,MSA}-X_{trend}^{i,MSA}$$

(6)

$$X_{complex}^{i,MSA}=\mathrm{Concat}(X_{residual}^{i,MSA},X_{season}^{i,MSA},X_{trend}^{i,MSA})$$

(7)

Then, the MSA mechanism is employed to explicitly calculate the pattern correlation between the sub-series, followed by picking up these critical features for the current task, which could be formulated as Equations (8) and (10). First, as shown in Equation (8), varying linear layers, including ${\mathrm{Dense}}_Q(\cdot )$, ${\mathrm{Dense}}_K(\cdot )$, and ${\mathrm{Dense}}_V(\cdot )$, are employed on $X_{complex}^{i,MSA}$ to produce different perspectives, including $Q_{\mathrm{MSA}}$, $K_{\mathrm{MSA}}$, and $V_{\mathrm{MSA}}$, respectively. These linear layers are learnable, thereby enhancing the learning capacity of the model. After that, the dot product between $Q_{\mathrm{MSA}}$ and $K_{\mathrm{MSA}}$ is calculated to obtain the attention score, assigning the large weights for these features, which are closely related to the current task. Different from the normal self-attention mechanism, the proposed MSA mechanism calculates the attention scores considering multiscale time series, explicitly exploring the correlation between the residual, seasonal, and trend components. By employing this mechanism on each component, the critical temporal pattern can be comprehensively captured, thereby enhancing the model’s learning capacity. In the end, as shown in Equation (10), the residual connection is utilized to improve the performance of the current task and avoid the performance degradation that usually appears in NNs with many layers.

$$\left\{\begin{array}{c}{Q}_{\mathrm{MSA}}={\mathrm{Dense}}_Q(X_{complex}^{i,MSA})\\ K_{\mathrm{MSA}}={\mathrm{Dense}}_K(X_{complex}^{i,MSA})\\ V_{\mathrm{MSA}}={\mathrm{Dense}}_V(X_{complex}^{i,MSA})\end{array}\right.$$

(8)

$$\mathrm{Score}=\mathrm{Softmax}({\displaystyle \frac{Q_{\mathrm{MSA}}{K}_{\mathrm{MSA}}^T}{d_{\mathrm{model}}}})$$

(9)

$$X^{o,MSA}={\mathrm{Dense}}_{\mathrm{o}}(\mathrm{Score}\times V_{\mathrm{MSA}})+X^{i,MSA}$$

(10)

3.2.2. Cross-Variable Attention Mechanism

In load forecasting, due to the potential interactions with the load, exogenous variables like the weather are usually considered by the model to enhance its performance. For instance, in the estimation of photovoltaic generation, the radiation intensity and angle are used because their nonlinear relationship can be formulated as equations. In HVAC load forecasting, many exogenous variables have been investigated, such as the outdoor temperature, relative humidity, atmospheric pressure, and so on [37,39]. However, the data of these exogenous variables are usually handled equally with the HVAC load by the model, which may result in the degradation of the model since it ignores these exogenous variables and focuses on the historical load. To allow the model to mine the temporal patterns of these exogenous variables and construct their correlation with the HVAC load, the CVA mechanism is proposed, which explicitly extracts the sub-series from the aggregated load, while these sub-series have strong correlations with the exogenous variables.

Specifically, the function of the CVA mechanism can be formulated as Equations (11) and (13). In Equation (11), similar to the MSA mechanism, $Q_{\mathrm{CVA}}$, $K_{\mathrm{CVA}}$, and $V_{\mathrm{CVA}}$ need to be produced according to the different linear layers. In this process, $Q_{\mathrm{CVA}}$ is the linear projection of the time series $W^{i,CVA}$ from the exogenous variables, while the $K_{\mathrm{CVA}}$ and $V_{\mathrm{CVA}}$ are the linear projections of the time series $X^{i,CVA}$ from the aggregated load. Using Equation (12), the dot product between the $Q_{\mathrm{CVA}}$ and $K_{\mathrm{CVA}}$ is calculated, which displays the similarity between the time series from different variables, thereby allowing the model to evaluate the correlation between the HVAC load and the exogenous variables. This differs from the usual processing of exogenous variables, which just concatenates the time series from the exogenous variables and HVAC load, followed by the NN to analyze it. After that, this attention score is multiplied by $V_{\mathrm{CVA}}$, which captures the critical information $V_{\mathrm{CVA}}$ related to exogenous variables, thereby facilitating the model learning of the complex pattern of the aggregated load.

$$\left\{\begin{array}{c}{Q}_{\mathrm{CVA}}={\mathrm{Dense}}_Q(W^{i,CVA})\\ K_{\mathrm{CVA}}={\mathrm{Dense}}_K(X^{i,CVA})\\ V_{\mathrm{CVA}}={\mathrm{Dense}}_V(X^{i,CVA})\end{array}\right.$$

(11)

$$\mathrm{Score}=\mathrm{Softmax}({\displaystyle \frac{Q_{\mathrm{CVA}}{K}_{\mathrm{CVA}}^T}{d_{\mathrm{model}}}})$$

(12)

$$X^{o,CVA}={\mathrm{Dense}}_{\mathrm{o}}(\mathrm{Score}\ast V_{\mathrm{CVA}})+X^{i,CVA}$$

(13)

3.2.3. Multi-Horizon Quantile Regression

In a real-world scenario, multi-horizon forecasting is more practical than single-step forecasting. For instance, the strategies of DR or unit commitment for the next day will be determined by referring to day-ahead load forecasting results. Meanwhile, the biases between the prediction and ground truth are unavoidable. Accurate uncertainty estimation, which offers the PIs and ensures the ground truth is located within this interval, is crucial for certain fields when the safety of strategies is seriously considered. Therefore, multi-horizon probabilistic forecasting, providing the predictions and their uncertainty, is necessary. In this research, the pinball loss is used to achieve probabilistic forecasting, whose formulation is shown in Equation (14).

$$L_{i,q_k,h}(y_h^i,{\widehat{y}}_h^{i,q_k})=\left\{\begin{array}{cc}(1-q_k)\ast ({\widehat{y}}_h^{i,q_k}-y_h^i)& if\hspace{1em}{\widehat{y}}_h^{i,q_k}\ge y_h^i\\ q_k\ast (y_h^i-{\widehat{y}}_h^{i,q_k})& if\hspace{1em}{\widehat{y}}_h^{i,q_k}\ge y_h^i\end{array}\right.$$

(14)

where $y_h^i$ represents the label of ${\mathit{x}}^i$ at the $h$-th forecasting horizon. This loss function lets the model predict multiple quantiles of the sample ${\mathit{x}}^i$. $q_k$ is set to varying values, thereby allowing the model to capture the entire PI. To expand it to multi-horizon probabilistic forecasting, the output layer of the model is adjusted according to Figure 2. Multiple linear layers are employed, where the number of linear layers is equal to the number of target quantiles. Combined with the backpropagation, the parameters of the NN are optimized, as shown in Equation (15).

$$\theta =\arg \underset{\theta }{\min }{\displaystyle {\sum }_{i\in I}{\displaystyle {\sum }_{h\in [L,L+H]}{\displaystyle {\sum }_{q_k\in [q_1,\dots ,q_n]}{L}_{i,q_k,h}(y_h^i,{\widehat{y}}_h^{i,q_k})}}}$$

(15)

where $\theta$ represents the parameters of the NN. $I$ represents the sample index in the training set. When $q_k$ is equal to 50, it could be viewed as the determined prediction.

4. Case Study

4.1. Dataset Setup

Two datasets were used to conduct the experiments, including the WPuQ dataset [40] and the EnergyDetective2020 dataset [41]. Specifically, the WPuQ dataset collected the electricity consumption of whole households and thermal appliances from some households in the same community. The collection period was from 2018 to 2020 with a 1 h time resolution. Due to the missed measurements, 21 households, whose measurements exceeded 90%, were selected for this experiment. The sum of their whole household electricity consumption was used as the aggregated load, while the sum of their thermal appliance electricity consumption was used as the HVAC load. Figure 4 shows the curve of the aggregated load. It can be observed that the energy demand in the winter season is higher than in the summer season. Apart from the aggregated load and HVAC load, the WPuQ dataset also provides the measurements of exogenous variables, including apparent temperature (AT), atmospheric pressure (AP), relative humidity (RH), solar irradiance (SI), and wind speed (WS). Through the calculation of PCC and ACC, the exogenous variables strongly related to HVAC load and the periodicity pattern of HVAC can be obtained, whose concrete values are shown in Figure 5. According to the content of Figure 5a, the AT has a strong relationship with HVAC, while other exogenous variables have a weak one. Therefore, the AT was utilized in the HVAC load forecasting. Furthermore, according to the content of Figure 5b, the HVAC load displays strong periodicity in the time scales of one day and one week. Therefore, the timestamp was utilized in the HVAC load forecasting.

The EnergyDetective2020 dataset collected the electricity consumption of 20 office buildings in Shanghai, which includes the HVAC load and the light and socket load, respectively. The collection period was from 2015 to 2017 with a 1 h time resolution. Four buildings were categorized into the same community, whose electricity consumption sum is regarded as the aggregated load. Their HVAC load sum is regarded as the HVAC load. Figure 6 shows the curve of aggregated load. It can be observed that the energy demand in the winter season is higher than in the summer season. Apart from the aggregated load and HVAC load, the EnergyDetective2020 dataset also provides the measurements of exogenous variables, including AT, Dew Point Temperature (DT), RH, AP, and WS. Their PCC and ACC results are shown in Figure 7. Similar to the WPuQ dataset, the HVAC load exhibits strong correlation with AT, as well as the periodicity patterns in the time scales of one day and one week. Therefore, the historical data of AT and timestamps were used in the HVAC load forecasting.

Four cases of each dataset are considered to simulate various seasons. The detailed ranges of the training set, test set, and validation set of the WPuQ and EnergyDetective2020 datasets are shown in

Table 1. The time range of the four cases.

		Training Set	Validation Set	Test Set
WPuQ	Case 1	1 January 2019–30 November 2019	1 December 2019–31 December 2019	1 January 2020–31 March 2020
	Case 2	1 April 2019–29 February 2020	1 March 2020–31 March 2020	1 April 2020–30 June 2020
	Case 3	1 July 2019–31 May 2020	1 June 2020–30 June 2020	1 July 2020–30 September 2020
	Case 4	1 October 2019–31 August 2020	1 September 2020–30 September 2020	1 October 2020–31 December 2020
EnergyDetective2020	Case 1	1 January 2015–30 November 2015	1 December 2015–31 December 2015	1 January 2016–31 March 2016
	Case 2	1 April 2015–29 February 2016	1 March 2016–31 March 2016	1 April 2016–30 June 2016
	Case 3	1 July 2015–31 May 2016	1 June 2016–30 June 2016	1 July 2016–30 September 2016
	Case 4	1 October 2015–31 August 2016	1 September 2016–30 September 2016	1 October 2016–31 December 2016

. The selected variables include the following: (1) the historical aggregated load; (2) the historical AT data; and (3) the timestamps, including the day and week. In the training phase, the training set is utilized to optimize the model parameters, while the validation set is used to adjust the hyperparameters. In the test phase, the test set is employed to evaluate the performance of the trained model. The values of $d$, $L$, and $H$ are set as 3, 48, 24.

4.2. Evaluation Metrics

To evaluate the performance of determined forecasting results and probabilistic forecasting results, four widely used metrics are utilized in this study [42,43,44,45].

(1)

Metrics for deterministic forecasting

The Root Mean Squared Error (RMSE) and Mean Absolute Percentage Error (MAPE) are used to evaluate deterministic forecasting.

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{mH}}{\displaystyle {\sum }_{i=1}^m{\displaystyle {\sum }_{h=1}^H{\left({\widehat{y}}_h^i-y_h^i\right)}^2}}}$$

(16)

$$\mathrm{MAPE}={\displaystyle \frac{1}{mH}}{\displaystyle {\sum }_{i=1}^m{\displaystyle {\sum }_{h=1}^H\frac{|{\widehat{y}}_h^i-y_h^i|}{\left|y_h^i\right|}}}$$

(17)

where $m$ represents the number of samples. A lower RMSE and MAPE indicate a better performance of deterministic forecasting.

(2)

Metrics for probabilistic forecasting

Prediction interval coverage probability (PICP) and prediction interval normalized averaged width (PINAW) are utilized to evaluate probabilistic forecasting.

$$\mathrm{PICP}={\displaystyle \frac{1}{mH}}{\displaystyle {\sum }_{h=1}^H{\displaystyle {\sum }_{i=1}^m\mathbb{I}\left(L_h^i\le {\widehat{y}}_h^i\le U_h^i\right)}}$$

(18)

$$\mathbb{I}\left(L_h^i\le {\widehat{y}}_h^i\le U_h^i\right)=\left\{\begin{array}{c}1,if{L}_h^i\le {\widehat{y}}_h^i\le U_h^i\\ 0, else\end{array}\right.$$

(19)

$$\mathrm{PINAW}={\displaystyle \frac{1}{mR}}{\displaystyle {\sum }_{h=1}^H{\displaystyle {\sum }_{i=1}^m\left(U_h^i-L_h^i\right)}}$$

(20)

where $U_h^i$ and $L_h^{i}h$ represents the upper and lower bounds of the PIs constructed for the $i$-th samples at the $h$-th forecasting horizon. $R$ represents the range of ${\mathit{y}}^i$. When PICP is close to the confidence level $\alpha$, it indicates that the model can accurately capture the forecasting uncertainty. When PINAW is smaller, it indicates the low forecasting uncertainty of the model.

4.3. Baseline and Benchmark

(1)

Baseline

The persistent model (PM) is regarded as the baseline, which utilizes the load of the previous day as the predicted future load. However, the model is fed with the total load instead of the HVAC load, where these two kinds of load have some differences. Therefore, the percentage between the HVAC load and the total load in each season is calculated, which is regarded as the coefficient that is multiplied by the predicted total load to obtain the HVAC load.

(2)

Benchmark

One machine learning-based and three deep learning-based models, including gradient boosting regressor (GBR), bidirectional long and short-term memory (BiLSTM), Convolutional Neural Network (CNN)-LSTM, and Transformer, which are usually utilized in load forecasting, are regarded as the benchmarks.

GBR [46]: This is a machine learning algorithm based on ensemble learning. It builds multiple weak regressors, followed by analyzing their predicted results to obtain excellent performance.

BiLSTM [24]: This is a deep learning algorithm that focuses on time series forecasting. It explicitly analyzes the temporal patterns through sequential processing, thereby accurately predicting the load.

CNN-LSTM [28]: This is a deep learning algorithm. Before the features are fed into the LSTM, a CNN is employed on the samples to first extract the features at each timestamp.

Transformer [18]: Its structure is similar to CNN-LSTM, while the section of LSTM is replaced with a Transformer Encoder.

Autoformer [47]: This is an advanced Transformer-based deep learning algorithm. It combines the autocorrelation mechanism and series decomposition to extract the complex temporal features of the input. The hyperparameters of these benchmarks are shown in

Table 2. The hyperparameter sets of benchmarks.

Model	Hyperparameter Settings
PM	-
GBR	Weak regressor = ‘Decision tree’, Estimators number = 20, Learning rate = 0.1, Max depth = 3
BiLSTM	LSTM structure = [128, 128], Bidirectional = True
CNN-LSTM	Conv1d structure = [48,48], LSTM structure = [128,128], activation function= ReLU, learning rate = 2 × 10−4
Transformer	Conv1d structure = [48,48], Transformer structure = [128,128], activation function= ReLU, learning rate = 2 × 10−4
Autoformer	Conv1d structure = [48,48], Autocorrelation structure = [128,128], activation function= ReLU, learning rate = 2 × 10−4

. The structure of the proposed method is shown in

Table 3. The structure of the proposed method.

Layer	Hyperparameters
Input	Historical load series	Historical temperature series
1	Linear (1, 128)	Linear (1, 128)
2	Conv1d (48, 48)	Conv1d (48, 48)
3	Position encoding	Position encoding
4	MSA Block	MSA Block
5	CVA Block	-
6	[Fully connected layer (48 × 128, 24)] × 19	-
Output	24 h multi-horizon quantile prediction result	-

.

To achieve probabilistic forecasting, each deep learning-based method is optimized to forecast 19 quantiles, where the values of the quantile are $[5,10,15,\dots ,95]$. When GBR is employed in each case, 456 GBRs are built, i.e., 19 GBRs for each step corresponding to 19 quantile regressor problems. For determined forecasting, their forecasting results when the quantile is 50 are utilized to evaluate their performance.

4.4. Determined Forecasting

4.4.1. Results

Table 4. The deterministic forecasting results of the proposed method on varying datasets.

	Method	Case1		Case2		Case3		Case4
		RMSE	MAPE	RMSE	MAPE	RMSE	MAPE	RMSE	MAPE
WPuQ	PM	5.37	19.66	3.32	45.16	1.89	49.16	4.31	21.38
	GBR	4.16	19.14	2.08	67.91	1.49	76.92	3.28	18.46
	BiLSTM	4.12	18.22	1.67	40.80	1.43	40.15	3.40	19.75
	CNN-LSTM	3.79	17.17	1.49	36.67	1.19	37.05	3.20	18.15
	Transformer	3.60	15.60	1.45	35.28	1.23	32.04	3.19	18.37
	Autoformer	3.63	15.74	1.44	32.68	1.24	35.08	3.21	19.23
	MSCVFormer	3.54	15.03	1.46	31.96	1.23	33.41	3.09	17.09
Energy Detective 2020	PM	463.58	59.80	325.81	66.65	785.32	55.91	287.96	52.36
	GBR	455.43	55.36	301.25	60.83	733.62	48.41	256.94	46.23
	BiLSTM	423.76	47.13	287.65	40.76	611.89	39.43	213.46	38.77
	CNN-LSTM	412.61	34.17	240.33	32.46	546.06	34.57	204.76	27.82
	Transformer	216.68	18.29	186.73	22.95	294.28	23.56	145.63	19.43
	Autoformer	172.63	17.22	188.54	23.55	301.46	22.97	137.85	18.55
	MSCVFormer	156.28	15.66	165.28	20.22	248.56	21.14	120.28	17.79

lists the deterministic forecasting results of MSCVFormer and other benchmark methods on two datasets, WPuQ and EnergyDetective2020. For their performance on the WPuQ dataset, it can be observed that MSCVFormer can obtain competitive RMSE and MAPE, outperforming these basic and advanced deep learning-based methods, such as BiLSTM, Transformer, and Autoformer. Specifically, compared to other comparative methods, MSCVFormer reduces the RMSE by 1.67% and 3.13% in Case 1 and Case 4, respectively. Similarly, for the EnergyDetective2020 dataset, MSCVFormer achieves the best performance, reducing the RMSE by 9.47%, 12.34%, 15.54%, and 12.75% and reducing the MAPE by 9.06%, 14.14%, 7.97%, and 4.10%. In summary, MSCVFormer exhibits an excellent deterministic load forecasting ability for the HVAC load.

Furthermore, its deterministic load forecasting ability is explored at different forecasting horizons. The MAPEs of MSCVFormer and other benchmark models at varying forecasting horizons on the WPuQ dataset are plotted in Figure 8. It can be observed that MSCVFormer obtains worse performance when the forecasting horizon increases. Meanwhile, compared to these benchmark models, MSCVFormer achieves the optimal forecasting results at each forecasting horizon. To this end, MSCVFormer possesses a superior ability for accurate HVAC load forecasting for a longer forecasting horizon.

4.4.2. Discussion of the Results

As the baseline method, PM is used to test the task difficulty. Its unsatisfactory performance is derived from the complex load pattern of HVAC, where the ratio of the HVAC load to the aggregated load is random, which conflicts with the load forecasting method of PM, using the fixed ratio and the aggregated load to calculate the HVAC load. For GBR, relying on its decision tree to capture the nonlinear correlation between the aggregated load and HVAC load, it achieves an improved performance in most cases. However, due to the limited learnable parameters and feature extraction, its performance fails to match these deep learning-based models. Through the employment of LSTM, which analyzes the time series sequentially, BiLSTM and CNN-LSTM can capture the short-term and long-term temporal patterns but fail to capture the correlation between two features with long time intervals. In contrast to the LSTM, due to the self-attention mechanism, the Transformer can directly calculate the correlation between any two features. Its flexibility provides the paths for Transformer-based models to capture the task-specific features and achieve better HVAC load forecasting. Meanwhile, its multi-head attention mechanism further enables the model to obtain various features and construct a comprehensive understanding of the HVAC load. Nevertheless, MSCVFormer still achieves an improved performance compared to these Transformer-based methods, where the improvements in performance are significant on the EnergyDetective2020 datasets. This is because MSCVFormer can utilize the CVA block to individually analyze the fluctuation pattern of different variables, calculating the specific influence of external variables on the aggregated load, thereby ensuring the effective utilization of these external variables in understanding the HVAC load. Meanwhile, MVA blocks ensure that the proposed model captures the load pattern from the multiscale time series, where the tendency and fluctuation of the HVAC load assist the model in achieving better HVAC load forecasting at the long forecasting horizon.

4.5. Probabilistic Forecasting

4.5.1. Results

Table 5. The probabilistic forecasting results of the proposed method on varying datasets.

	Method	Case1		Case2		Case3		Case4
		PICP	PINAW	PICP	PINAW	PICP	PINAW	PICP	PINAW
WPuQ	GBR	0.959	0.505	0.773	0.699	0.852	1.089	0.975	0.596
	BiLSTM	0.905	0.368	0.800	0.343	0.872	0.363	0.923	0.395
	CNN-LSTM	0.893	0.319	0.804	0.288	0.836	0.302	0.915	0.352
	Transformer	0.907	0.308	0.825	0.264	0.914	0.319	0.894	0.330
	Autoformer	0.880	0.354	0.847	0.225	0.904	0.330	0.880	0.274
	MSCVFormer	0.877	0.279	0.890	0.292	0.906	0.290	0.897	0.296
Energy Detective 2020	GBR	0.937	0.408	0.944	0.548	0.853	0.645	0.922	0.421
	BiLSTM	0.898	0.289	0.880	0.220	0.869	0.242	0.872	0.210
	CNN-LSTM	0.931	0.242	0.878	0.157	0.873	0.232	0.871	0.180
	Transformer	0.859	0.067	0.807	0.061	0.839	0.078	0.817	0.073
	Autoformer	0.889	0.168	0.850	0.119	0.880	0.203	0.918	0.214
	MSCVFormer	0.894	0.065	0.881	0.065	0.875	0.098	0.873	0.081

displays the probabilistic load forecasting results when the confidence level is 90%. According to the experimental results, MSCVFormer achieves superior performance across all cases on both datasets, particularly in terms of PINAW, while maintaining high PICP values. On the WPuQ dataset, MSCVFormer consistently achieves low PINAW values in most cases, indicating narrower PIs compared to other methods, where PINAW values are reduced by 9.42% and 3.97% in Case 1 and Case 3, respectively. In Case 2 and Case 4, while the PINAWs of MSCVFormer are slightly higher than those of Autoformer, a competitive PICP is obtained, where the deviations between the PICP and the nominal confidence level are 0.01 and 0.003, which are lower than those of Autoformer. Notably, MSCVFormer overall outperforms Transformer and Autoformer. Furthermore, on the EnergyDetective2020 dataset, only MSCVFormer and Transformer achieve lower PINAW values in all four cases, whose PINAW is lower than 0.01, significantly outperforming other models. However, compared to Transformer, MSCVFormer obtains a higher PICP, where the deviations in the PICP with the nominal confidence level are 0.006, 0.019, 0.025, and 0.027. In summary, MSCVFormer can deliver accurate PIs with a narrow interval width and reliable coverage.

Considering that both Transformer and MSCVFormer can provide high-quality PIs with a high PICP and low PINAW when the nominal confidence is set to 90%, another experiment was conducted to compare their performance in implementing probabilistic HVAC load forecasting when the nominal confidence is set to 60%, 70%, 80%, and 90%. Figure 9 shows the PICP of these two methods in different cases. It can be observed that the proposed method can provide a PICP that matches the nominal confidence level in Case 2, Case 3, and Case 4, which is significantly better than the performance of the Transformer, which obtains a PICP with a large bias from the nominal confidence level [48]. This indicates that the proposed method can capture the uncertainty of model predictions to a limited extent in most cases and generate PIs of an appropriate width to cover these true values, providing effective information for formulating feasible DR strategies.

To directly demonstrate the performance of the proposed method in probabilistic HVAC load forecasting, Figure 10 shows the PIs of one week of each case on the WPuQ dataset. These weeks are randomly selected. It can be observed that significant differences in the range of HVAC loads in different cases exist, with the load range being [10, 40] in Case 1 and [0, 10] in Case 3. At the same time, there are differences in their load patterns. For example, in Case 1 and Case 2, the HVAC load shows a decreasing trend. For different HVAC load modes, the proposed method can provide a PI with an adaptive interval width, which is narrower in Case 3 and wider in Case 1. At the same time, these PIs can cover the vast majority of HVAC loads. To this end, the proposed method can ensure that the model outperforms probabilistic forecasting.

4.5.2. Discussion of the Results

Among these benchmark models in Table 5, GRB exhibits an unstable probabilistic load forecasting performance on both datasets, as evident from a significant gap between its PICP and nominal confidence level. This is because the construction of multiple GBRs is required to achieve probabilistic HVAC load forecasting, with each GBR predicting a percentile separately. This approach prevents GBR from learning the complete HVAC load distribution using the same set of parameters, resulting in lower robustness, i.e., an unstable PICP. Unlike GBR, these deep learning-based models are combined with quantile regression, where their output is no longer a single deterministic prediction, but rather multiple quantiles, which prompts the model to perceive prediction uncertainty and provide effective PIs. However, the sharpness and coverage of the constructed PIs still depend on the ability to capture complex HVAC load patterns. For BiLSTM and CNN-LSTM, these two basic deep learning-based models, as analyzed in Section 4.4.2, fail to provide optimal deterministic HVAC load forecasting, which means they have higher prediction uncertainty and require higher interval widths to ensure sufficient coverage of the intervals. The advanced Transformer-based model, Autoformer, achieves better probability prediction performance compared to Transformer on the EnergyDetective2020 dataset, providing PIs with a higher PICP. This is due to its strong ability to extract periodic features, as well as its autocorrelation attention mechanism, which automatically analyzes potential periodicity in the input and achieves tailored modeling, thereby resulting in a better performance on the aggregated load with a strong periodicity, such as EnergyDetective2020. Compared to Autoformer, MSCVFormer can construct the PI with enough PICP and a narrow interval width on both datasets. This benefits from the utilization of CVA, which integrates the fluctuation degrees of various external variables and aggregated loads, enabling the accurate prediction of the load distribution of HVAC. Meanwhile, MSA decomposes the time series into components of different time scales, which assists the model in accurately perceiving the degree of fluctuation of these variables, thus laying a solid foundation for achieving accurate probabilistic HVAC load forecasting.

5. Discussion

5.1. Application

The proposed method allows for accurate HVAC load probabilistic forecasting, which provides the critical support for DR and the energy management of the power grid. Specifically, it can be used to evaluate the potential ability of HVAC systems to participate in DR and to make energy scheduling more economic and resilient due to the accurate load distribution forecasting.

5.2. Limitations

The proposed method assumes that large amounts of HVAC load at the initial training phase are available, while they are impractical for the newly constructed building and community. In this case, it is a challenge to utilize even fewer zero samples to achieve accurate HVAC load forecasting.

6. Conclusions

This article proposes a novel data-driven method, MSCVFormer, to predict the HVAC load of the next day based on the historical data of aggregated load and exogenous variables. In this process, through the PCC and the ACC, the temperature and timestamp are utilized in the HVAC load forecasting. MSCVFormer includes two novel blocks to extract the complex temporal pattern of time series. Specifically, MSA is proposed, embedding the STL into the self-attention mechanism. This design enables the NN to focus on the pattern correlation between multiscale time series, thereby capturing the critical information for HVAC load forecasting. This is in contrast to the conventional methods, which separately conduct the feature extraction and prediction for each component of the input, ignoring their correlation. Meanwhile, CVA is proposed, calculating the similarity between the time series from different variables, thereby achieving the essential feature extraction. Unlike conventional models that treat all input variables uniformly, this block innovatively considers the differences between varying variables, thereby efficiently utilizing these exogenous variables to assist the model in capturing the complex temporal pattern of the load. Compared to the advanced Transformer-based methods, such as Autoformer, which employs the series decomposition and delayed aggregation, the proposed method has better performance in deterministic forecasting, where the RMSE and MAPE are reduced by 13.04% and 8.81% on average on the WPuQ and EnergyDetective2020 datasets, respectively. Meanwhile, the proposed method can deliver the PIs with a PICP close to the nominal confidence intervals and reduce the interval width by 43.89% compared to Autoformer on the EnergyDetective2020 dataset.