Multi-Step Ahead Forecasting of the Energy Consumed by the Residential and Commercial Sectors in the United States Based on a Hybrid CNN-BiLSTM Model

Abstract

COVID-19 has continuously influenced energy security and caused an enormous impact on human life and social activities due to the stay-at-home orders. After the Omicron wave, the economy and the energy system are gradually recovering, but uncertainty remains due to the virus mutations that could arise. Accurate forecasting of the energy consumed by the residential and commercial sectors is challenging for efficient emergency management and policy-making. Affected by geographical location and long-term evolution, the time series of the energy consumed by the residential and commercial sectors has prominent temporal and spatial characteristics. A hybrid model (CNN-BiLSTM) based on a convolution neural network (CNN) and bidirectional long short-term memory (BiLSTM) is proposed to extract the time series features, where the spatial features of the time series are captured by the CNN layer, and the temporal features are extracted by the BiLSTM layer. Then, the recursive multi-step ahead forecasting strategy is designed for multi-step ahead forecasting, and the grid search is employed to tune the model hyperparameters. Four cases of 24-step ahead forecasting of the energy consumed by the residential and commercial sectors in the United States are given to evaluate the performance of the proposed model, in comparison with 4 deep learning models and 6 popular machine learning models based on 12 evaluation metrics. Results show that CNN-BiLSTM outperforms all other models in four cases, with MAPEs ranging from 4.0034% to 5.4774%, improved from 0.1252% to 49.1410%, compared with other models, which is also about 5 times lower than that of the CNN and 5.9559% lower than the BiLSTM on average. It is evident that the proposed CNN-BiLSTM has improved the prediction accuracy of the CNN and BiLSTM and has great potential in forecasting the energy consumed by the residential and commercial sectors.

1. Introduction

With the declaration of the COVID-19 outbreak as a pandemic on 11 March 2020 by the Word Health Organization (WHO), COVID-19 has resulted in a massive global toll and unprecedented impact on public health, economic growth, and energy security. As of 23 December 2022, there have been 99,027,628 confirmed cases of COVID-19, with 1,080,010 deaths, reported to WHO [1], and the economy also took a devastating hit due to the COVID-19 pandemic, with increasing unemployment rates and dramatically slowed GDP growth rate of the country, as shown in Figure 1. As a country, with the remaining energy security as a priority issue, the United States has been influenced by the pandemic, with the total energy consumption falling to 93 quads, a decrease of 7%, compared to 2019, according to EIA’s Energy Review [2]. In an effort to reduce the spread of COVID-19 associated with community activities, the government and the Centers for Disease Control and Prevention (CDC) have issued mandatory stay-at-home orders in 42 states and territories from 1 March through 31 May 2020 [3]. Affected by the lockdowns (such as closed offices, businesses, schools, industrial facilities, and curtailment recreations), energy consumption by the commercial and residential sectors fell by 7% to less than 17 quads in 2020 and 1% to less than 21 quads (less energy consumption for home heating, due to the warm weather in 2020), respectively. Specifically, compared to April 2019, U.S. residential electricity sales were up 8%, while the commercial sector was down 11% [4]. Several studies have further implemented the significant changes related to the energy consumption by the residential and commercial sectors and analyzed the reasons from the perspective of the lockdowns during the COVID-19 pandemic [5,6,7], revealing that it will take some time for the energy sector to return its “normal” status. More importantly, with more relaxed measures by governments and societies worldwide after the omicron wave and economic recovery activities, but also the risk of a new phase of new variants, COVID-19 will continue to be a major source of uncertainty for energy consumption, especially in the residential and commercial sectors that are tightly connected to human life. Therefore, accurate forecasting of residential and commercial energy consumption attaches great significance to energy security, social operation, and living safe ground in facing these challenges.

For energy consumption forecasting, the forecasting horizon is conventionally divided into short-term, medium-term, and long-term forecasting: short-term forecasting usually predicts the data from one hour to one week (hourly, daily, and weekly horizons), medium-term forecasting is usually applied to intervals ranging from one week to one year (monthly and quarterly horizons), and long-term forecasting is for one year or more than one year (annual horizons) [8,9,10]. For the application fields, Soldo et al. [11] divided the application forecasting fields of energy consumption mainly from the world level, national or state level, and urban areas, based on the existing research. Currently, the forecasting methods for energy consumption can be roughly divided into statistical forecasting models (mainly time series models), grey system models, deep learning models, and machine learning models.

The time series models are the most traditional forecasting methods. In recent years, many scholars have expanded the traditional time series models from the modeling and application level and made a series of achievements in the energy consumption forecasting field. For instance, the seasonal autoregressive integrated moving average with exogenous factors (SARIMAX) is proposed to forecast residential natural gas consumption in Turkey [12], recursive autoregressive with Extra Input is utilized to forecast the short-term gas consumption [13], autoregression distributed lag mixed data sampling model for the long-term energy demand forecasting of China [14], wavelet transform auto regression integrated moving average is proposed for the load forecasting [15], and Bayesian vector autoregressive model for the energy supply and demand forecasting [16]. The energy consumption forecasting based on the time series models is carried out on both the variations and historical patterns in the data with a consideration of the trend or seasonal effects, and the forecasting horizons are diverse, ranging from short-term to long-term forecasting. However, the time series models always hold a better forecasting performance in medium-term and long-term forecasting than the short-term one, as there will be limitations in dealing with large-scale data with complex nonlinearity.

Grey system models can be used in nonlinear time series forecasting and have been widely applied in energy forecasting in recent years. Zhang et al. [17] proposed the grey Lotka–Volterra model to analyze the internal relationship between the energy consumption structure of China, the United States (US), and Germany. A fractional nonlinear grey Bernoulli model with rolling was proposed to forecast the share of renewables in primary energy consumption [18]. Khan and Osińska [19] proposed an optimized nonlinear grey Bernoulli model for energy consumption forecasting in Brazil and India. Chen et al. [20] proposed the optimal nonlinear metabolic grey system model to forecast the energy consumption in the Yangtze River Delta region. Ding et al. [21] proposed the seasonal adaptive grey model to forecast the medium-term renewable energy consumption of the commercial sector in the US. The nonlinear interactive grey system model based on dynamic compensation was proposed for total energy consumption forecasting in China [22]. With its outstanding ability to cope with uncertain problems characterized by small samples and inadequate information, the grey system models are efficient in long-term yearly forecasting and always yield higher precision than the models that heavily rely on historical data [23]. Besides, the grey system models also perform well in nonlinear long-term forecasting tasks with the development of the “grey-box” models combined with other nonlinear approaches. However, a few studies have applied the grey system models for medium-term and short-term forecasting and always resulted in poor precision [9].

Compared to the conventional time series model and grey system models, the data-driven machine learning models are preferred to short-term and medium-term forecasting, as these models need to learn useful information from the data, as well as the hyperparameter tuning to obtain a capability for efficient forecasting. At present, many scholars directly apply single machine learning models to energy forecasting and obtain good prediction results. For example, Gaussian process regression was used in short/medium-term forecasting of solar and wind power [24]. Using multiple machine learning models for energy forecasting and comparing their prediction performance is also a popular research method [25,26]. However, many studies have shown that hybrid models outperform single models, such as the hybrid models based on support vector regression (SVR) [27,28,29,30], the hybrid model based on extreme gradient boosting (XGB) [31], and the hybrid models based on multiple machine learning models [32]. The machine learning models have the advantage of dealing with complex nonlinear problems in short-term and medium-term forecasting. However, there are limitations of the machine learning models in the monthly and quarterly forecasting and always present a “time delay” problem, leading to relatively low prediction accuracy.

With the popularity of deep learning, many neural network models are widely utilized in energy consumption forecasting. Similar to machine learning models, deep learning models are also data-driven models that need a large amount of data for training the model and optimizing the model hyperparameters. Thereby, deep learning models are widely used in short-term and medium-term forecasting with a strong capability of handling complete nonlinear problems, as well as outstanding precision and robustness. At present, the long short-term memory (LSTM), the gate recurrent unit (GRU), the recurrent neural network (RNN), and the convolutional neural network (CNN) are the most popular neural network models. Additionally, many studies have been carried out based on these models. Mustaqeem et al. [33] proposed the CNN-assisted deep echo state network for short-term solar energy forecasting. Amalou et al. [34] compared the RNN, LSTM, and GRU in different energy consumption forecasting. A hybrid model based on the variational auto-encoder and LSTM was proposed to forecast the renewable electricity supply in South Korea [35]. Yang et al. [36] used the nonlinear mapping network to forecast wind power. Ren et al. [37] used the LSTM, Stack LSTM, and Bidirectional LSTM to forecast the multiple energy loads for a university. Ding et al. [38] made renewable energy generation forecasting by the seasonal and trend decomposition using Loess (STL) and LSTM. Khan et al. [39] proposed the integration model based on artificial neural network (ANN) and the LSTM for solar PV generation forecasting. Many studies show that connecting neural networks with different layers is an effective method to improve the prediction accuracy of a single neural network, such as LSTM-RNN [40], A-CNN-LSTM (A is attention layer) [41], CNN-GRU [42], CNN-LSTM-AE (AE is autoencoders) [43], RNN-LSTM [44], ConvLSTM-BDGRU-MLP (ConvLSTM is convolutional LSTM; BDGRU is bidirectional GRU; MLP is multilayer perceptron) [45], CNN-BiGRU (BiGRU is bi-directional GRU) [46], EWT-attention-LSTM (EWT is an empirical wavelet transform) [47], LSTM-GAM (GAM is global attention mechanism) [48], and CNN-LSTM [49]. It can be seen from the above research that CNN and LSTM are the most popular layers, as they have the strong ability to capture the data features. Actually, the bidirectional LSTM (BiLSTM) has been proven that it has better performance than the LSTM in forecasting energy consumption [50,51]. Therefore, a hybrid model based on the CNN and the BiLSTM is proposed to forecast the energy consumed by the residential and commercial sectors in this paper.

The rest of this paper is arranged as follows: Section 2 introduces the modeling process; Section 3 presents the parameter-solving process of the proposed model; Section 4 gives the four forecasting cases of the energy consumed by the residential and commercial sectors; Section 5 presents the discussion; Section 6 gives the conclusions.

2. Methodology

2.1. Spatial Features Extraction of Time Series Based on Convolutional Neural Network

The time series of energy consumption of residential and commercial sectors presents nonlinear and nonstationary features affected by the geographical location of residential and commercial areas. It is difficult to capture these spatial features. However, the convolutional neural network (CNN) can easily solve this problem, as it can learn the structure of the time series very well. CNN is a neural network with convolution, which can reduce the complexity of the neural network. The local features extraction and weights sharing are the key operations of CNN. Since the spatial connection of the time series is local, each neuron only needs to catch the local features. Then, the global information can be obtained by combining these local neurons, which can reduce the number of connections. Weights sharing can reduce the hyperparameters of the CNN, making the structure of the network more adaptable and simpler. The convolution layer and pooling layer are the main modules to achieve the feature extraction of the CNN.

Convolution layer: The main effect of the convolution layer is to capture the spatial features of the time series in this paper. Different from the traditional weight connection layer, this layer extracts features through the convolution operation. Additionally, the one-dimensional convolution layer is used in this paper. When performing a convolution operation, the convolution kernel slides over the input and calculates the inner product with the input to obtain a new feature matrix. Assuming a one-dimensional time series input $\mathbf{x}={\left(x^{\left(\xi \right)}\right)}_{\xi =1}^N$, the output from the convolution layer with one convolution kernel $\mathbf{w}={\left(w^{\left(\xi \right)}\right)}_{\xi =1}^{\kappa }$ is

$$\begin{array}{c}\hfill \mathbf{H}={\left(h^{\left(j\right)}\right)}_{j=1}^{N-\kappa +1},h^{\left(j\right)}=\mathbf{w}·{\mathbf{x}}_{\mathbf{j}}=\sum _{\xi =1}^{\kappa }{w}^{\left(\xi \right)}{x}_j^{\left(\xi \right)},\end{array}$$

(1)

where ${\mathbf{x}}_{\mathbf{j}}$ is the input subsequence covered by the jth slide of $\mathbf{w}$ and $\kappa$ is the kernel size. The convolution kernel only has one channel, as the input is one-dimensional. In the above way, the output size is absolutely controlled by the kernel size and input size.

Pooling layer: The main function of this layer is to compress the feature matrix by reducing the number of features to avoid overfitting. The two commonly used methods of the pooling layer are max pooling and average pooling. The max pooling divides the feature matrix output from the convolution layer into different subregions and then calculates the maximum value of each subregion to obtain a new feature matrix that has a smaller size than the raw feature matrix. Similarly, the average pooling is to calculate the average value of each subregion to obtain a new feature matrix. The average pooling preserves the overall features of the data but reduces the feature differences, while the max pooling can extract the most responsive part of the features. Therefore, the one-dimensional max pooling layer is utilized in this paper. Assuming the input size of the max pooling is $(M,C,N_{\mathrm{in}})$, the output can be described as:

$$\begin{array}{c}\hfill out\left(M_j,C_i,\xi \right)=\underset{t=0,\dots ,\kappa -1}{max}input\left(M_j,C_i,\alpha \times \xi +t\right),\end{array}$$

(2)

where $\alpha$ is the stride of the sliding window. The output size is $(M,C,N_{\mathrm{out}})$, where

$$\begin{array}{c}\hfill N_{\mathrm{out}}=⌊\frac{N_{\mathrm{in}}-\kappa }{\alpha }+1⌋.\end{array}$$

(3)

2.2. Temporal Features Extraction of Time Series Based on Bidirectional Long Short-Term Memory (BiLSTM)

The long short-term memory (LSTM) introduced in 1997 is an extension of RNN that solves the problem of exploding and vanishing gradients more effectively than the conventional RNN models [52]. It has been popularized in a variety of tasks in different research fields, such as machine translation [53], handwriting recognition [54], time series prediction[55], and so on. Out of its various applications, the LSTMs are well-suited in time series forecasting, with the advantage of precisely learning long-term dependencies. With the popularity of LSTM and outstanding behaviors, several variants and different versions of the LSTM appeared in these years, of which the BiLSTM developed by Jurgen Schmidhuber [56] is one of the widely applied LSTMs and always provides better forecasting performance, compared to regular LSTM models in time series forecasting [57]. Therefore, the BiLSTM model is utilized in this work.

For a common LSTM unit, let $C_t$ denotes the cell state, which indicates the memory of the LSTM containing both the previous and current information. Besides, there are three gates: the input gate $i_t$, the output gate $o_t$, and the forget gate $f_t$ in each LSTM unit to control the flow of information optionally. Within the weight matrix $\left\{W_j\right|j=f,C,i,o\}$, bias term $\left\{b_j\right|j=f,C,i,o\}$, current input data $x_t$, and previously hidden state $h_{t-1}$, the whole process of a single LSTM network unit can be explained in three parts: the forget gate to select the information, the input gate and new memory to store and update new information, and the output gate to determine what output from the cell state. A more direct illustration of the LSTM model is shown in Figure 2.

The forget gate: The output value of forget gate is obtained by a Sigmoid function $\sigma$, which is based on the combination of the output value of the $h_{t-1}$ and $x_t$ to decide which information should be deleted from the memory and which information should be retained. It can be given as

$$f_t=\sigma \left(W_f·\left[h_{t-1},x_t\right]+b_f\right).$$

(4)

The input gate and new memory network: Within the same $x_t$ and $h_{t-1}$, this section consists of the following two parts: one is the input gate, deciding which new information to add according to the Sigmoid function $\sigma$, similar to the forget gate, the other one is a new vector ${\widehat{C}}_t$ created by the “tanh” function that shifts the output in the interval $\left[-1,1\right]$. In short, implements are described in the following formulations:

$$i_t=\sigma \left(W_i·\left[h_{t-1},x_t\right]+b_i\right),$$

(5)

$${\widehat{C}}_t=tanh\left(W_C·\left[h_{t-1},x_t\right]+b_C\right).$$

(6)

Then, the new cell state $C_t$ is updated by multipling $f_t$ and $C_{t-1}$, and adding $i_t\odot {\widehat{C}}_t$, which is illustrated as below:

$$C_t=f_t\odot C_{t-1}+i_t\odot {\widehat{C}}_t.$$

(7)

The output gate: The output gate determines what output information to generate, and the formula for calculating the $o_t$ and the new hidden layer $h_t$ is given as

$$\begin{array}{c}{o}_t=\sigma \left(W_o·\left[h_{t-1},p_t\right]+b_o\right),\hfill \\ h_t=o_t\odot tanh\left(C_t\right).\hfill \end{array}$$

(8)

Compared with the conventional one-directional LSTM, the BiLSTM contains two independent LSTM structures in which feature learning is performed in forward and reverse order for the input sequence. In this way, the model is not only trained from input to output but also can be trained from output to input, thereby effectively improving the model dependence and improving the model’s forecasting accuracy. For BiLSTM, the hidden layer $h_t$ consists of both the forward ${\overrightarrow{h}}_t$ and backward $\overleftarrow{h_t}$, which is expressed as follows:

$$h_t={\overrightarrow{h}}_t\oplus \overleftarrow{h_t},$$

(9)

where ⊕ represents the element-wise summation of the forward and backward output components, and the network structure of the BiLSTM is presented in Figure 3.

2.3. The CNN-BiLSTM

The hybrid model that combines the CNN, the BiLSTM and the connection layer is proposed to forecast the energy consumed by the residential and commercial sectors, abbreviated as CNN-BiLSTM. The input of the CNN-BiLSTM first enters the CNN layer, and a new feature matrix is generated after convolution computation and max pooling. The feature matrix obtained from the CNN is utilized as the input of the BiLSTM, and then the hidden output of the BiLSTM is obtained. The hidden output is pushed into the connection layer, which consists of a linear layer. Then, the final predicted results are obtained from the connection layer.

2.4. Forecasting Strategy for Univariate Time Series

The hybrid model CNN-BiLSTM describes the relationship between input and output. A common method is using the recursive multi-step forecasting strategy to construct the univariate time series forecasting scheme. Since the CNN-BiLSTM is a supervised learning method, the univariate time series needs to be reconstructed to satisfy the input and output of the CNN-BiLSTM. Assuming a univariate time series sample $y^{\left(1\right)},y^{\left(2\right)},\dots ,y^{\left(n\right)}$ with lag $\tau$, the predicted value of the $y^{(\tau +1)}$ can be obtained from the previous $\tau$ steps. Then, the 1-dimensional vector is reconstructed into $(\tau +1)$-dimensional matrix. The reconstructed sample matrix $\Theta$ can be generated as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \Theta & =[{\mathbf{Y}}^{\left(1\right)},{\mathbf{Y}}^{\left(2\right)},\dots ,{\mathbf{Y}}^{\left(\tau \right)},{\mathbf{Y}}^{(\tau +1)}]\hfill \\ \hfill & ={\left[\begin{array}{ccccc}{y}^{\left(1\right)}& y^{\left(2\right)}& \cdots & y^{\left(\tau \right)}& y^{(\tau +1)}\\ y^{\left(2\right)}& y^{\left(3\right)}& \cdots & y^{(\tau +1)}& y^{(\tau +2)}\\ ⋮& ⋮& \ddots & ⋮& ⋮\\ y^{(n-\tau -1)}& y^{(n-\tau )}& \cdots & y^{(n-2)}& y^{(n-1)}\\ y^{(n-\tau )}& y^{(n-\tau +1)}& \cdots & y^{(n-1)}& y^{\left(n\right)}\end{array}\right]}_{(n-\tau )\times (\tau +1)},\hfill \end{array}\end{array}$$

(10)

where ${\mathbf{Y}}^{\left(1\right)}={[y^{\left(1\right)},y^{\left(2\right)},\dots ,y^{\left(\tau \right)},y^{(\tau +1)}]}^T$ is a column vector with lag $\tau$. Then, the input of the CNN-BiLSTM is a matrix $\mathbf{X}$ consisting of the previous $\tau$ column vectors $[{\mathbf{Y}}^{\left(1\right)},{\mathbf{Y}}^{\left(2\right)},\dots ,{\mathbf{Y}}^{\left(\tau \right)}]$, and the output is the $(\tau +1)$-step column vector ${\mathbf{Y}}^{(\tau +1)}$ in Equation (10). The detailed input and output can be presented by

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill \mathbf{X}& ={\left[\begin{array}{cccc}{y}^{\left(1\right)}& y^{\left(2\right)}& \cdots & y^{\left(\tau \right)}\\ y^{\left(2\right)}& y^{\left(3\right)}& \cdots & y^{(\tau +1)}\\ ⋮& ⋮& \ddots & ⋮\\ y^{(n-\tau -1)}& y^{(n-\tau )}& \cdots & y^{(n-2)}\\ y^{(n-\tau )}& y^{(n-\tau +1)}& \cdots & y^{(n-1)}\end{array}\right]}_{(n-\tau )\times \tau },\mathbf{Y}\hfill & \hfill ={\left[\begin{array}{c}{y}^{(\tau +1)}\\ y^{(\tau +2)}\\ ⋮\\ y^{(n-1)}\\ y^{\left(n\right)}\end{array}\right]}_{(n-\tau )\times 1}.\end{array}\end{array}$$

(11)

When forecasting the time series, the new predicted value of $y^{\left(\xi \right)}$ is added to the input for the next time step, then the future points can be forecasted by the CNN-BiLSTM based on the recursive strategy. The detailed recursive process is expressed as follows:

$$\begin{array}{cc}\hfill {\widehat{y}}^{(n+1)}& =g\left(\left[y^{(n-\tau +1)},\dots ,y^{(n-1)},y^{\left(n\right)}\right]\right),\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill {\widehat{y}}^{(n+2)}& =g\left(\left[y^{(n-\tau +2)},\dots ,y^{\left(n\right)},{\widehat{y}}^{(n+1)}\right]\right),\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill ⋮& \hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill {\widehat{y}}^{(n+\tau +1)}& =g\left(\left[{\widehat{y}}^{(n+1)},{\widehat{y}}^{(n+2)},\dots ,{\widehat{y}}^{(n+\tau )}\right]\right),\hfill \end{array}$$

(15)

$$\begin{array}{c}\hfill ⋮\end{array}$$

(16)

It is interesting to see that when the forecasting reaches step $\tau +1$, the input vector already completely consists of the forecasted values, which implies that a complete extrapolation has been achieved. The detailed calculation process is given by Algorithm 1.

Algorithm 1: Algorithm of recursive multi-step forecasting strategy.

The overall workflow of the CNN-BiLSTM is plotted in Figure 4.

3. Hyperparameters Tuning Based on the Grid Search Approach

It is generally acknowledged that the performance of a deep learning model heavily depends on the selection of hyperparameters since neural networks are notoriously difficult to configure and require multiple hyperparameters to be set. In recent years, swarm intelligence optimization algorithms have been widely used to tune the model hyperparameters. However, some essential factors that may affect the optimization precision exist, such as the high computational cost, algorithm-based parameters tuning and control, rate of convergence and control, etc. [58]. For instance, the popular swarm-based particle swarm optimization (PSO) is weakened in the local explanation and easily falls into the local optimum with a low convergence rate, resulting in a low precision or even failure [59,60,61]. The grid search method is one of the most common and also a direct way of efficiently tuning the model hyperparameters and has been successfully developed for tuning the hyperparameters of the deep learning models [62,63,64]. Therefore, the grid search approach is developed to tune the optimal hyperparameters of the models in this work.

The framework for the hyperparameters optimization based on the grid search is developed in mainly two aspects: data division and grid search scheme. Firstly, the nested cross-validation [65] is utilized in this work for data division, a popular method for hyperparameters optimization and model selection to get over the overfitting problem in time series forecasting. Assumed the training size $\varsigma$ (80% of the data), validation size $\upsilon$ (10% of the data), and testing size p (10% of the data), the raw input $\mathbf{X}$ and output $\mathbf{Y}$ data are split into the three parts: the training data ${\mathbf{X}}_{\varsigma }$, ${\mathbf{Y}}_{\varsigma }$ for model fitting, the validation data ${\mathbf{X}}_{\upsilon }$, ${\mathbf{Y}}_{\upsilon }$ for model validation, and the testing data ${\mathbf{X}}_p$, ${\mathbf{Y}}_p$ for out-of-sample forecasting performance model evaluation. Then, the grid search approach is employed to search the optimal model hyperparameters based on the training and validation data. For the CNN-BiLSTM model, there are three main hyperparameters that need to be tuned: the kernel size $\kappa$ for CNN, the hidden size ℏ for the BiLSTM, and the learning rate $lr$, an important hyperparameter to train a good model in deep learning. Within the grid of the hyperparameters, the key to the hyperparameters optimization based on the grid search approach is to search the $\kappa$, ℏ, and $lr$ to minimize the mean squared error (MSE) on the validation data ${\mathbf{X}}_{\upsilon }$, ${\mathbf{Y}}_{\upsilon }$, which can be expressed as

$$min\frac{1}{\upsilon }\sum _{k=1}^{\upsilon }{\left(y_{\upsilon }^{\left(k\right)}-{\widehat{y_{\upsilon }}}^{\left(k\right)}\right)}^2.$$

(17)

Based on the optimally selected hyperparameters ${\kappa }^{*}$, ${\hslash }^{*}$, and $l{r}^{*}$, the CNN-BiLSTM model will be refitted based on the in-sample data (containing both the training ${\mathbf{X}}_{\varsigma }$ and validation data ${\mathbf{X}}_{\upsilon }$). Then, the fitted model with optimal hyperparameters is prepared for evaluating the model’s predictive ability on the out-of-sample testing data ${\mathbf{X}}_p$. In general, the integral computational algorithm procedure of the hyperparameters tune based on the grid search approach is presented in the following Algorithm 2.

Algorithm 2: Grid search for tuning the optimal hyperparameters of the CNN-BiLSTM.

4. Case Studies

4.1. Data Collection and Pre-Processing

In this paper, the data were collected from the EIA Monthly Energy Review (https://www.eia.gov/totalenergy/data/monthly/, accessed on 13 June 2020) from January 1973 to May 2020, including the total energy consumed by the residential sector, end-use energy consumed by the residential sector, primary energy consumed by the commercial sector, and end-use energy consumed by the commercial sector. Before data division, the data were first normalized into the interval $\left[0,1\right]$, based on the MinMax scaling. Then, data from January 1973 to May 2018 (535 samples) were used as the in-sample data to fit and tune the hyperparameters of the models, and data from June 2018 to May 2020 (24 multi-step ahead forecasting) were used as the testing data to verify the out-of-sample predictive ability of the models in this work.

4.2. Benchmarked Models for Comparisons and Evaluation Metrics

In this study, five benchmarked deep learning models, including the single BiLSTM [56], CNN [66], LSTM [52], and combined CNN-LSTM [49], as well as several popular machine learning models, including the SVR [67] based on the RBF kernel, least squares support vector regression (LSSVR) [68] based on the RBF kernel, random forecast (RF) [69], gradient boosting with categorical features support (CatBoost) [70], light gradient boosting machine (LGBM) [71], and XGBoost [72] were developed for model comparison. In addition, twelve assessment metrics were utilized to verify the model forecasting performance. Detailed illustrations of the metrics are presented in

Table 1. Metrics to evaluate the predictive quality of the CNN-BiLSTM model.

Abbreviation	Definition	Expression
MAPE	Mean Absolute Percentage Error	$\frac{1}{p}{\displaystyle \sum _{\xi =1}^p}\left|\frac{y_p^{\left(\xi \right)}-{{\widehat{y}}_p}^{\left(\xi \right)}}{y_p^{\left(\xi \right)}}\right|\times 100\%$
MAE	Mean Absolute Error	$\frac{1}{p}{\displaystyle \sum _{\xi =1}^p}\left|y_p^{\left(\xi \right)}-{{\widehat{y}}_p}^{\left(\xi \right)}\right|$
RMSE	Root Mean Square Error	$\sqrt{\frac{1}{p}{\displaystyle \sum _{\xi =1}^p}{\left(y_p^{\left(\xi \right)}-{{\widehat{y}}_p}^{\left(\xi \right)}\right)}^2}$
MSE	Mean Squared Error	$\frac{1}{p}{\displaystyle \sum _{\xi =1}^p}{\left(y_p^{\left(\xi \right)}-{{\widehat{y}}_p}^{\left(\xi \right)}\right)}^2$
MAAPE	Mean Arctangent Absolute Percentage Error	$\frac{1}{p}{\displaystyle \sum _{\xi =1}^p}arctan\left(\left|\frac{y_p^{\left(\xi \right)}-{{\widehat{y}}_p}^{\left(\xi \right)}}{y_p^{\left(\xi \right)}}\right|\right)$
NRMSE	Normalized Root Mean Square Error	$\frac{\sqrt{\frac{1}{p}{\displaystyle \sum _{\xi =1}^p}{\left(y_p^{\left(\xi \right)}-{{\widehat{y}}_p}^{\left(\xi \right)}\right)}^2}}{y_{p}max-y_{p}min}$
RMSPE	Root Mean Square Percentage Error	$\sqrt{\frac{1}{p}{\displaystyle \sum _{\xi =1}^p}{\left|\frac{y_p^{\left(\xi \right)}-{{\widehat{y}}_p}^{\left(\xi \right)}}{y_p^{\left(\xi \right)}}\right|}^2}$
SMAPE	Symmetric Mean Absolute Percentage Error	$\frac{1}{p}{\displaystyle \sum _{\xi =1}^p}\left|\frac{y_p^{\left(\xi \right)}-{{\widehat{y}}_p}^{\left(\xi \right)}}{0.5y_p^{\left(\xi \right)}+0.5{{\widehat{y}}_p}^{\left(\xi \right)}}\right|\times 100\%$
U1	Theil U Statistic 1	$\frac{\sqrt{\frac{1}{p}{\displaystyle \sum _{\xi =1}^p}{\left(y_p^{\left(\xi \right)}-{{\widehat{y}}_p}^{\left(\xi \right)}\right)}^2}}{\sqrt{\frac{1}{p}{\displaystyle \sum _{\xi =1}^p}{\left(y_p^{\left(\xi \right)}\right)}^2}+\sqrt{\frac{1}{p}{\displaystyle \sum _{\xi =1}^p}{\left({{\widehat{y}}_p}^{\left(\xi \right)}\right)}^2}}$
U2	Theil U Statistic 2	$\frac{\sqrt{\frac{1}{p}{\displaystyle \sum _{\xi =1}^p}{\left(y_p^{\left(\xi \right)}-{{\widehat{y}}_p}^{\left(\xi \right)}\right)}^2}}{\sqrt{{\displaystyle \sum _{\xi =1}^p}{\left(y_p^{\left(\xi \right)}\right)}^2}}$
IA	Index of Agreement	$1-\frac{{\displaystyle \sum _{\xi =1}^p}{\left(y_p^{\left(\xi \right)}-{\widehat{y_p}}^{\left(\xi \right)}\right)}^2}{{\displaystyle \sum _{\xi =1}^p}{\left(\left|y_p^{\left(\xi \right)}-{\overline{y_p}}^{\left(\xi \right)}\right|+\left|{\widehat{y_p}}^{\left(\xi \right)}-{\overline{\widehat{y_p}}}^{\left(\xi \right)}\right|\right)}^2}$
R${}^2$	Coefficient of Determination	$1-\frac{{\displaystyle \sum _{\xi =1}^p}{\left(y_p^{\left(\xi \right)}-{\widehat{y_p}}^{\left(\xi \right)}\right)}^2}{{\displaystyle \sum _{\xi =1}^p}{\left(y_p^{\left(\xi \right)}-{{\overline{y}}_p}^{\left(\xi \right)}\right)}^2}$

.

4.3. Forecasting Results Analysis

In order to comprehensively compare the CNN-BiLSTM model with other models, four actual cases (including the total energy consumed by the residential sector, end-use energy consumed by the residential sector, primary energy consumed by the commercial sector, and end-use energy consumed by the commercial sector) with the same lag are carried out.

4.3.1. Case I: Total Energy Consumed by the Residential Sector

In the 24 multi-step ahead forecasting of the total energy consumed by the residential sector, the out-of-sample predicted values are displayed in Figure 5, and the corresponding evaluation metrics are presented in

Table 2. 24 multi step out-of-sample forecasting metrics of all models in Case I.

	CNN-BiLSTM	BiLSTM	CNN-LSTM	CNN	LSTM	SVR	LSSVR	RF	CatBoost	LGBM	XGBoost
	lr = 0.01 ℏ = 60 $\kappa$ = 2	lr = 0.1 ℏ = 20	lr = 0.001 ℏ = 25 $\kappa$ = 2	lr = 0.1 $\kappa$ = 2	lr = 0.001 ℏ = 60	C = 55 $ϵ$ = $1\times {10}^{-6}$ $\gamma$ = 0.03125	C = 625 $\lambda$ = 0.03125	max_depth = 7 min_samples_leaf = 1 min_samples_split = 2 n_estimators = 50	depth = 10 l2_leaf_reg = 100.0 lr = 0.01	max_depth = 5 num_leaves = 50 reg_alpha = 0.01 reg_lambda = 1	gamma = 0.0 lr = 0.1 max_depth = 5 min_child_weight = 1 reg_alpha = 1
MAPE	5.0417	5.8555	44.2395	22.0456	37.6811	20.8846	28.8258	5.1669	13.1710	5.9160	15.6317
MAE	87.7030	102.3284	803.9523	428.3851	692.8386	343.4988	462.9180	91.8243	246.8932	105.8958	277.2572
RMSE	110.8020	131.6346	869.9062	540.1201	767.0836	410.9314	571.3807	127.5901	318.3963	136.3436	371.4026
MSE	12,277.0771	17,327.6641	756,736.8125	291,729.7188	588,417.2500	168,864.6442	326,475.9121	16,279.2250	101,376.2171	18,589.5845	137,939.8906
MAAPE	0.0503	0.0584	0.4135	0.2132	0.3569	0.2011	0.2660	0.0514	0.1299	0.0590	0.1521
NRMSE	0.0939	0.1115	0.7368	0.4575	0.6497	0.3481	0.4840	0.1081	0.2697	0.1155	0.3146
RMSPE	0.0626	0.0727	0.4534	0.2593	0.3924	0.2577	0.3731	0.0708	0.1603	0.0730	0.2040
SMAPE	5.0855	6.0200	57.8681	26.1360	47.5717	19.0293	24.1186	5.2382	13.8647	5.8904	16.0521
U1	0.0314	0.0375	0.3202	0.1746	0.2713	0.1123	0.1484	0.0360	0.0936	0.0383	0.1070
U2	0.0624	0.0741	0.4899	0.3042	0.4320	0.2314	0.3218	0.0719	0.1793	0.0768	0.2092
IA	0.9720	0.9666	0.3690	0.4518	0.3935	0.3917	0.3762	0.9640	0.5246	0.9638	0.6462
$R^2$	0.8883	0.8423	−5.8872	−1.6551	−4.3553	−0.5369	−1.9713	0.8518	0.0774	0.8308	−0.2554

. It is clear that the predicted values of the CNN-BiLSTM are the closest to the raw data. Additionally, it can be seen that the predicted curve of the BiLSTM is similar to those of the CNN-BiLSTM. This is easy to explain as the BiLSTM is one of the layers of the CNN-BiLSTM. However, the BiLSTM failed to forecast the last two points. It is clear that the CNN-BiLSTM outperforms the CNN, which shows that the hybrid model can effectively improve the prediction performance of the single model. The predicted values of the other deep learning models are similar to constants. They absolutely failed to forecast the raw data, as these models are overfitting. For machine learning models, RF and LGBM perform well in this case, but the CatBoost and the XGBoost only catch the rough trend of the raw data. Even the SVR and LSSVR obtain the wrong trends due to overfitting.

It is clear that the CNN-BiLSTM outperforms all the other models, as all its metrics are the best. It can be seen that, although the predicted values of the RF and LGBM are similar to those of the CNN-BiLSTM, their metrics are worse than those of the CNN-BiLSTM. Additionally, prediction accuracy of the CNN-BiLSTM is obviously better than the single CNN and the BiLSTM. For the other deep learning models, the MAPE of the CNN-BiLSTM is at least four times smaller than that of them. For machine learning models, the LSSVR performs the worst, and the SVR has a similar performance to it. In general, the CNN-BiLSTM performs best in this case.

4.3.2. Case II: End-Use Energy Consumed by the Residential Sector

In the 24 multi-step ahead forecasting of the end-use energy consumed by the residential sector, the out-of-sample predicted values are plotted in Figure 6, and the corresponding evaluation metrics are presented in

Table 3. 24 multi step out-of-sample forecasting metrics of all models in Case II.

	CNN-BiLSTM	BiLSTM	CNN-LSTM	CNN	LSTM	SVR	LSSVR	RF	CatBoost	LGBM	XGBoost
	lr = 0.01 ℏ = 50 $\kappa$ = 2	lr = 0.01 ℏ = 40	lr = 0.01 ℏ = 65 $\kappa$ = 3	lr = 0.1 $\kappa$ = 2	lr = 0.1 ℏ = 45	C = 5 $ϵ$ = $1\times {10}^{-6}$ $\gamma$ = 1.72844	C = 15625 $\lambda$ = 0.13446	max_depth = 7 min_samples_leaf = 1 min_samples_split = 5 n_estimators = 50	depth = 8 l2_leaf_reg = 0 lr = 0.01	max_depth = 5 num_leaves = 50 reg_alpha = 0.001 reg_lambda = 0.1	gamma = 0.0 lr = 0.5 max_depth = 5 min_child_weight = 1 reg_alpha = 1
MAPE	5.0292	5.3492	40.6992	21.2876	28.0344	11.0438	13.8404	9.0470	7.8271	7.3418	7.3984
MAE	47.1754	50.2112	453.6898	227.9078	273.9267	111.5945	135.6584	89.3437	76.8737	72.0425	74.2770
RMSE	68.3823	83.3169	555.3765	292.4198	339.5989	143.5384	166.9033	110.3985	97.0129	95.2964	96.7993
MSE	4676.1406	6941.7031	308,443.0938	85,509.3438	115,327.4141	20,603.2779	27,856.7086	12,187.8240	9411.4937	9081.4098	9370.1064
MAAPE	0.0501	0.0528	0.3781	0.2067	0.2621	0.1093	0.1366	0.0899	0.0779	0.0730	0.0736
NRMSE	0.0676	0.0824	0.5491	0.2891	0.3358	0.1419	0.1650	0.1091	0.0959	0.0942	0.0957
RMSPE	0.0692	0.0916	0.4404	0.2482	0.3513	0.1322	0.1622	0.1071	0.0926	0.0908	0.0918
SMAPE	5.0129	5.6868	54.0280	23.7873	27.1095	11.8818	14.7027	9.5339	8.1516	7.5961	7.7344
U1	0.0329	0.0402	0.3546	0.1549	0.1689	0.0721	0.0848	0.0545	0.0476	0.0465	0.0476
U2	0.0660	0.0805	0.5364	0.2824	0.3280	0.1386	0.1612	0.1066	0.0937	0.0920	0.0935
IA	0.9895	0.9840	0.4640	0.6817	0.5731	0.9491	0.9192	0.9717	0.9786	0.9795	0.9780
$R^2$	0.9546	0.9327	−1.9923	0.1704	−0.1188	0.8001	0.7298	0.8818	0.9087	0.9119	0.9091

. It can be seen that the predicted points of the first 12 steps of the CNN-BiLSTM almost coincide with the original points. It is obvious that, although the prediction accuracy of most models becomes worse with the increase in the number of forecasting steps, the CNN-BiLSTM presents the best stability. It is clear that the CNN-BiLSTM outperforms the CNN and the BiLSTM, even though the predicted values of the BiLSTM are closest to those of it. The performance of the BiLSTM for the later points is worse than the CNN-BiLSTM, which implies that the stability of the BiLSTM is worse than the CNN-BiLSTM. The CNN-LSTM presents the worst performance among all the models, as the model is overfitting. The CNN and the LSTM only capture the rough trends. The predicted curves of the machine learning models, except the SVR and LSSVR, are quite similar, but obviously, their predicted values are not as close to the raw data as the CNN-BiLSTM.

It is clear that of all the models, the CNN-BiLSTM has the best metrics while the CNN-LSTM has the largest error. It is interesting to see that the performance of the CNN-BiLSTM is better than the single CNN and BiLSTM, which further explains that the hybrid model with the convolution layer and BiLSTM layer has a stronger prediction ability. And it can be seen that the prediction performance of the LSTM is still far worse than the BiLSTM. Although the predicted curves of most machine learning models look pretty close to the raw data, their metrics are poor. The LGBM presents the best prediction accuracy among the machine learning models, while it is still worse than the CNN-BiLSTM. And the SVR and the LSSVR have the worst performance among the machine learning models.

4.3.3. Case III: Primary Energy Consumed by the Commercial Sector

In the case of the primary energy consumed by the commercial sector multi-step ahead forecasting, 24-step out-of-sample predicted values are shown in Figure 7, and the corresponding evaluation metrics are presented in

Table 4. 24 multi step out-of-sample forecasting metrics of all models in Case III.

	CNN-BiLSTM	BiLSTM	CNN-LSTM	CNN	LSTM	SVR	LSSVR	RF	CatBoost	LGBM	XGBoost
	lr = 0.01 ℏ = 55 $\kappa$ = 2	lr = 0.01 ℏ = 25	lr = 0.01 ℏ = 70 $\kappa$ = 2	lr = 0.01 $\kappa$ = 2	lr = 0.1 ℏ = 70	C = 20 $ϵ$ = $1\times {10}^{-6}$ $\gamma$ = 0.04500	C = 625 $\lambda$ = 0.40171	max_depth = 7 min_samples_leaf = 1 min_samples_split = 2 n_estimators = 100	depth = 7 l2_leaf_reg = 1 lr = 0.01	max_depth = 5 num_leaves = 50 reg_alpha = 0.01 reg_lambda = 1	gamma = 0.0 lr = 0.5 max_depth = 3 min_child_weight = 2 reg_alpha = 0.01
MAPE	5.4774	12.6515	46.7455	27.8494	19.9738	15.4584	13.0736	13.0576	12.0870	14.0688	16.0561
MAE	21.9951	47.7852	217.1153	93.8588	76.8539	69.5279	54.7455	56.1637	49.5175	56.9160	69.7948
RMSE	27.1889	60.9706	273.3944	103.2772	90.8298	94.1978	74.6178	76.6894	65.7881	77.1039	96.7247
MSE	739.2354	3717.4133	74,744.4766	10,666.1846	8250.0557	8873.2195	5567.8181	5881.2619	4328.0769	5945.0158	9355.6621
MAAPE	0.0547	0.1248	0.4225	0.2650	0.1952	0.1515	0.1287	0.1285	0.1194	0.1382	0.1569
NRMSE	0.0565	0.1267	0.5681	0.2146	0.1888	0.1958	0.1551	0.1594	0.1367	0.1602	0.2010
RMSPE	0.0625	0.1545	0.5176	0.3233	0.2237	0.1887	0.1604	0.1611	0.1445	0.1719	0.1976
SMAPE	5.5394	11.5774	66.3500	32.5211	20.7827	17.2753	14.5297	14.5490	13.2565	15.6626	18.3556
U1	0.0322	0.0680	0.4539	0.1319	0.1092	0.1204	0.0943	0.0975	0.0828	0.0972	0.1253
U2	0.0642	0.1439	0.6453	0.2438	0.2144	0.2223	0.1761	0.1810	0.1553	0.1820	0.2283
IA	0.9930	0.9680	0.4525	0.8915	0.9243	0.8941	0.9408	0.9360	0.9545	0.9381	0.8952
$R^2$	0.9720	0.8590	−1.8351	0.5954	0.6871	0.6634	0.7888	0.7769	0.8358	0.7745	0.6451

. In general, the utilized CNN-BiLSTM model is well-performed with all multi-step ahead predicted values close to the raw samples and outperforms the other benchmarked models. One can find that only the CNN-BiLSTM model successfully captured the fluctuations from July-September 2019, whereas other models are weak in dealing with those fluctuations. Besides, the CNN-BiLSTM holds the advantage of following the trend without less trouble in time delay, while all the predicted values of other models are carried with the “time delay” characteristic. Compared with the other single benchmarked deep learning models (including the BiLSTM, CNN, and LSTM), the CNN-BiLSTM model improves the predictive ability a lot, and it is also interesting to see that the CNN-LSTM model has quite poor performance with a nearly horizontal forecasting line in this case due to the overfitting. On the other hand, the CNN-BiLSTM is more capable of coping with the peaking point and time delay than the machine learning models.

From the perspective of the out-of-sample evaluation metrics in Table 4, the CNN-BiLSTM model consistently produces the lowest metrics (where IA and $R^2$ are the largest) than other benchmarked deep learning models and machine learning models, of which the MAPE is lower from 6.6096% to 41.2681%. As we can see, the $R^2$ for the other deep learning models are relatively small, especially the CNN-LSTM model with an $R^2$ value of −1.8351 and there also no $R^2$ value above 0.9, whereas the $R^2$ of the CNN-BiLSTM model is up to 0.972, revealing that the CNN-BiLSTM is capable of the out-of-sample forecasting.

4.3.4. Case IV: End-Use Energy Consumed by the Commercial Sector

In the case of the End-Use energy consumed by the commercial sector multi-step ahead forecasting, 24-step out-of-sample predicted values are shown in Figure 8, and the corresponding evaluation metrics are presented in

Table 5. 24 multi step out-of-sample forecasting metrics of all models in Case IV.

	CNN-BiLSTM	BiLSTM	CNN-LSTM	CNN	LSTM	SVR	LSSVR	RF	CatBoost	LGBM	XGBoost
	lr = 0.01 ℏ = 125 $\kappa$ = 2	lr = 0.001 ℏ = 150	lr = 0.01 ℏ = 5 $\kappa$ = 2	lr = 0.1 $\kappa$ = 3	lr = 0.01 ℏ = 10	C = 100 $ϵ$ = $1\times {10}^{-6}$ $\gamma$ = 1.72844	C = 390625 $\lambda$ = 0.09336	max_depth = 11 min_samples_leaf = 2 min_samples_split = 10 n_estimators = 100	depth = 6 l2_leaf_reg = 1 lr = 0.01	max_depth = 5 num_leaves = 50 reg_alpha = 1 reg_lambda = 1	gamma = 0.0 lr = 0.05 max_depth = 3 min_child_weight = 1 reg_alpha = 1
MAPE	4.0034	18.0789	53.1444	24.0272	44.4674	9.0418	10.6892	11.9223	10.8338	12.0526	11.0120
MAE	30.5129	138.1454	424.3060	176.0921	353.9534	75.3807	85.4468	95.7201	86.7298	96.8677	88.6650
RMSE	37.4739	144.8520	449.4491	197.1012	374.8922	96.5099	111.9645	119.9051	106.6557	124.0479	106.3436
MSE	1404.2955	20,982.1133	202,004.5000	38,848.8867	140,544.1250	9314.1634	12,536.0451	14,377.2259	11,375.4350	15,387.8716	11,308.9697
MAAPE	0.0400	0.1784	0.4859	0.2318	0.4159	0.0898	0.1057	0.1178	0.1074	0.1190	0.1093
NRMSE	0.0754	0.2913	0.9039	0.3964	0.7540	0.1941	0.2252	0.2412	0.2145	0.2495	0.2139
RMSPE	0.0491	0.1885	0.5391	0.2751	0.4538	0.1097	0.1355	0.1446	0.1273	0.1472	0.1261
SMAPE	4.0839	20.0636	73.4016	23.1385	58.0351	9.6521	11.0102	12.5102	11.4853	12.7886	11.5984
U1	0.0240	0.1005	0.3939	0.1240	0.3088	0.0641	0.0727	0.0788	0.0704	0.0818	0.0704
U2	0.0475	0.1836	0.5696	0.2498	0.4751	0.1223	0.1419	0.1520	0.1352	0.1572	0.1348
IA	0.9840	0.7973	0.3550	0.4935	0.4142	0.8586	0.7996	0.8019	0.8464	0.7894	0.8249
$R^2$	0.9330	−0.0004	−8.6318	−0.8523	−5.7013	0.5559	0.4023	0.3145	0.4576	0.2663	0.4608

. It is obvious that the CNN-BiLSTM model is more efficient in the out-of-sample multi-step ahead forecasting as reflected in Figure 8. For the predicted values based on the deep learning models (plotted in (a)–(e)), the CNN-LSTM model and the LSTM model hold very poor performance in the 24-step ahead forecasting, as the predicted values are far away from the raw samples, and the reason for this phenomenon is that the models are overfitting. Besides, the CNN model also fails to catch the continuance of periodicity and presents an uncorrected predictive growing linear tendency as the CNN is overfitted. Compared to the single BiLSTM model, the forecasting ability of the BiLSTM combined with CNN is improved, in terms of handling the fluctuations and better following the original trend. In terms of the machine learning models, there is a common phenomenon that the predicted values by machine learning models delay with more steps, and it is difficult for the machine learning models to predict the values nearby the peak points. In contrast, the CNN-BiLSTM model shows the advantage of efficiently peaking points, with predicted values closely following the original data and less “lag” characteristic in out-of-sample multi-step ahead forecasting.

From what has been discussed above, the out-of-sample metrics of all models provided in Table 5 also present more precise numerical evidence that the utilized CNN-BiLSTM model, which performs better in the out-of-sample time series forecasting than the popular machine learning models and the single deep learning models, as well as the combined CNN-LSTM models. In terms of benchmark deep learning models, the MAPE of CNN-BiLSTM is almost five times smaller than these models, with the value reduced between 14.0755% and 49.1410%, and the $R^2$ of CNN-BiLSTM is 0.9330, while the $R^2$ for other deep learning models are negative. On the other hand, the out-of-sample multi-step predictive ability of the machine learning models is inferior to that of the CNN-BiLSTM model with MAPE exceeding 10%. In brief, the CNN-BiLSTM model always yields a favorable forecasting performance.

5. Discussion

For further discussion, one can find that the CNN-BiLSTM model maintains the preferable performance in all the cases, with the out-of-sample predicted values close to the raw data in an accurate trend, as well as the superior metrics than all the other benchmarked deep learning models and several popular machine learning models. More detailed analyses are presented as follows in terms of the predictive ability of the model and application performance.

5.1. Comparisons of the CNN-BiLSTM Model and Benchmarked Machine Learning Models

The kernel models (such as the SVR and LSSVR model) and tree models (such as RF, CatBoost, etc.) used in this work always produce favorable forecasting performance in recent years, whereas it can be easily found that those models hold homogenous drawbacks, which are unable to follow the raw trends tightly with apparent “lag” characteristic in the multi-step ahead forecasting (noticeable in Case III and Case IV of the commercial sector). Additionally, the machine learning models also waken in handling the top points, with predicted values upward or down from the raw data (in Case I and Case II of the residential sector). However, the deep learning CNN-BiLSTM model is more adaptable to deal with those problems and presents a good stable performance in all cases, showing strength in extracting temporal features and simultaneously processing the time series features. The metrics results also reveal that the CNN-BiLSTM outperforms all the machine learning models in the four cases. The best-performing machine model provides RMSEs of 127.5901, 95.2964, 65.7881, and 95.5099, respectively, while the RMSEs of the CNN-BiLSTM are 110.8020, 68.3823, 27.1889, and 37.4739, respectively.

5.2. Comparisons of the CNN-BiLSTM Hybrid Model and Benchmarked Deep Learning Models

In terms of the deep learning models, the results in our work coincide with the existing works, which compare the performance of the BiLSTM model and LSTM model in the time series forecasting and indicate that the BiLSTM model consistently outperforms the LSTM model in energy consumption forecasting (The LSTM offers the MAPEs of 37.6811, 28.0344, 19.9738, and 44.4674, while the BiLSTM provides the MAPEs of 5.8555, 5.3492, 12.6515, and 18.0989) [57], as the forward LSTM of BiLSTM can extract the past data information of the input sequence and the reverse LSTM of BiLSTM can extract the future data information of the input sequence, so as to realize the dual training of the forward and reverse LSTM of the time series, and further improve the global and complete feature extraction ability. On the other hand, the combination of the CNN and BiLSTM models has improved the predictive ability, compared to the single CNN and BiLSTM models (especially in Case IV, in which the MAPEs of the CNN and BiLSTM are 24.0272 and 18.0789, respectively, while the CNN-BiLSTM offers a MAPE of 4.0034), which is more capable of reaching the peak points and enhanced the predictive ability, with predicted values tightly following the raw trend.

5.3. Performance Analysis of the Models Considering the COVID-19 Stay-at-Home Orders

In terms of the time period of the application aspect, the in-sample data used for model fitting based on the optimal hyperparameters (from January 1973 to May 2018) are the total without an influence of COVID-19, and the out-of-sample data to validate the model forecasting accuracy include both the data (from June 2018 to December 2019, the first 19-step ahead forecasting), without an influence of the COVID-19 and data (from January 2020 to May 2020, the 20-th to 24-th step ahead forecasting) affected by the COVID-19. It can be observed from the figures that the developed CNN-BiLSTM model holds not only a superior performance in the unaffected 19-step ahead forecasting but also a more efficient adaption to the decreased data influenced by COVID-19. In contrast, the other models seem to be less adaptable than the CNN-BiLSTM model in the COVID-19 influenced period, even with optimistic forecasting results in the previous period, such as the BiLSTM and the tree models shown in Figure 5. In detail, as known that the United States government delivered mandatory stay-at-home orders in March-May 2020, and the energy consumed by the residential and commercial sectors has been severely affected by the lockdown. It can be observed that the CNN-BiLSTM model is more capable of dealing with the duration influenced by the COVID-19 stay-at-home orders, with the 22-th to 24-th step ahead predicted values close to the raw data during the lockdown time in 2020, and shows no significant differences in the multi-step ahead forecasting in 2018-2019. In contrast, the other benchmarked deep learning and machine learning models are relatively inferior in effectively predicting the samples during lockdown time.

6. Conclusions

Forecasting the energy consumed by the residential and commercial sectors is of great significance. However, it is hard for traditional machine learning models and deep learning models to obtain accurate forecasting results since the time series of the energy consumed by the residential and commercial sectors has obvious temporal and spatial characteristics. Hence, a hybrid model CNN-BiLSTM based on the convolution neural network and bidirectional long short-time memory is proposed to forecast the energy consumed by the residential and commercial sectors in this paper. The convolution layer of the CNN-BiLSTM is utilized to capture the spatial features of the time series. Additionally, the BiLSTM layer is used to catch the characteristics of long-term evolution and short-term change of the time series. The CNN-BiLSTM can be fully extrapolated by using the recursive multi-step prediction strategy. Additionally, the grid search is utilized to search the optimal hyperparameters of the CNN-BiLSTM.

The case study focuses on the energy consumed by the residential and commercial sectors in the US, including primary energy consumption, end-use energy consumption, and total energy consumption, with raw data from January 1973 to July 2022, which covers 595 points. For the four cases, the proposed CNN-BiLSTM offers improvements for the MAPEs of CNN of 16.6286, 16.2584, 22.372, and 20.0238, respectively; the improvements for the MAPEs of BiLSTM are 0.8138, 0.32, 7.1741, and 14.0755, respectively. What’s more, the MAPEs of the CNN-BiLSTM are 0.1252, 2.3126, 6.6096, and 5.0384, smaller than the MAPEs of the best-performing model among other models, respectively. The forecasting and metrics results of the four cases show that the CNN-BiLSTM obviously improves the adaptability and stability of the single CNN and BiLSTM and has the best forecasting performance among the other deep learning and machine learning models. The CNN-BiLSTM can make accurate short/medium-term forecasting for the energy consumed by the residential and commercial sectors in the US with the above discussions. Additionally, it is expected to be used for more kinds of energy forecasting in the future.

A possible limitation of the CNN-BiLSTM is that this model may not be suitable for small sample data modeling, as this model has complex layers, which require a lot of data to train parameters. The traditional grid search is adopted to optimize the CNN-BiLSTM in this paper, which may result in local optimization. From this perspective, future work can also be extended by improving the parameter optimization algorithm.