Short-Term Load Forecasting Using EMD-LSTM Neural Networks with a Xgboost Algorithm for Feature Importance Evaluation

Abstract

Accurate load forecasting is an important issue for the reliable and efficient operation of a power system. This study presents a hybrid algorithm that combines similar days (SD) selection, empirical mode decomposition (EMD), and long short-term memory (LSTM) neural networks to construct a prediction model (i.e., SD-EMD-LSTM) for short-term load forecasting. The extreme gradient boosting-based weighted k-means algorithm is used to evaluate the similarity between the forecasting and historical days. The EMD method is employed to decompose the SD load to several intrinsic mode functions (IMFs) and residual. Separated LSTM neural networks were also employed to forecast each IMF and residual. Lastly, the forecasting values from each LSTM model were reconstructed. Numerical testing demonstrates that the SD-EMD-LSTM method can accurately forecast the electric load.

1. Introduction

Short-term load forecasting (STLF), which ranges from one hour to one week ahead, plays an important role in the control, power security, market operation, and scheduling of reasonable dispatching plans for smart grids. However, achieving high accuracy is difficult because of the complicated effects of a variety of attributes on the load.

Over the past few decades, scholars have developed many modus to improve the accuracy of STLF that can mainly be divided into three methods, namely, traditional, similar day (SD), and artificial intelligence (AI)-based methods. Traditional methods are based on mathematical models, including multiple linear regression [1], stochastic time series [2], exponential smoothing [3], and knowledge-based methods [4]. Traditional methods often perform poorly at nonlinear forecasting, and STLF is a nonlinear problem. Accordingly, the prediction accuracy of traditional methods is insufficient for STLF.

The SD method is based on the selection of historical days that have similar features to the forecasted days [5,6,7,8,9]. Mandal et al. [7] selected SDs based on the calculation of the Euclidean norm of factors between historical and forecasted days. Chen et al. [8] required SDs to have the same weekday index and similar weather to the forecasted days. Mu [9] applied a weighted average model for the historical day to determine the influence of most SDs on the forecasted day. However, using this method solely cannot sufficiently obtain high prediction accuracy. The selection of input variables plays a crucial role when modelling time series and thus should be treated as a generalization problem. Arahal [10] proposed a method consists on calculating the difference index for all variables.

The AI-based methods, such as the artificial neural networks [11,12,13,14], support vector machine [15,16,17,18], expert system models [19], fuzzy logistic methods [20], and Bayesian neural networks [21], are extensively used to handle many forecasting problems. Although wide-ranging research has been conducted, an accurate STLF remains a challenge due to its non-stationary load data and long-term dependencies forecasting horizon. Hence, we applied the long short-term memory (LSTM) [22,23], which is a special type of recurrent neural network (RNN) architecture [24], to solve the STLF problem. The vanishing gradient point is a problem for RNNs in handling time series; LSTM cells can address this issue by incorporating memory cells in the hidden layer of RNN. LSTM performs well in long time horizon forecasting than other artificial intelligence methods based on the past load data that determine the effect and relationship among time series.

However, two inherent defects exist in neural networks: slow convergence and presence of a local minimum. After extensively analyzing the structure of neural networks, scholars [25] proposed a model that combines data decomposition with neural networks to address these two defects. Empirical Mode Decomposition (EMD) [26,27,28,29] can facilitate the determination of the characteristics of the complex non-linear time series. EMD is based on local characteristics of the signal sequence to complete the signal decomposition, in other words, the method do not require any base function pre-defined. Compared with wavelet decomposition, EMD method can be applied in theory to the decomposition of any type of signal since it has the characteristics of intuitive, direct, posterior and adaptive. Because it is essentially quite different from wavelet decomposition methods based on the wavelet basis function. Briefly, The EMD takes advantage of the multi-resolution and overcomes the difficulty of selecting wavelet basis function in wavelet transformation.

Given the preceding discussion, this study presents a generic framework that combines extreme gradient boosting (Xgboost) and k-means on SD selection, empirical mode decomposition (EMD), and LSTM neural networks to forecast short-term load (i.e., SD-EMD-LSTM model). Simulation experiments that use hourly load data from New England-ISO are conducted to verify the performance of the proposed STLF framework. We compare the SD-EMD-LSTM model with several other classical STLF models, and the results demonstrate that the proposed model outperforms the others in one-day ahead and one-week ahead tests. Moreover, numerical testing confirms that the proposed SD-EMD-LSTM model is capable of forecasting accuracy, robustness, and stability.

The main contributions of this study are as follows.

• Although the temperature, humidity, and day type have been extensively used as input features in STLF, we also recognize that STLF is sensitive to the day-ahead peak load, which has to be a supplemental input feature to the SD selection and LSTM training processes.
• Extending from our previous work on data analysis, we independently learned the feature candidate weights for the SD selection framework based on the Xgboost algorithm to overcome the dimensionality limitation in clustering. Thus, the proposed Xgboost-based k-means framework can deal with the SD selection tasks beyond pure clustering.
• Numerical testing demonstrates that data decomposition-based LSTM neural networks can outperform most of the well-established forecasting methods in the longer-horizon load forecasting problem.

The rest of this paper is organized as follows. Section 2 discusses the factors that affect electricity forecasting, including temperature, day-type, and day-ahead peak load factors. Section 3 presents a generic SD selection framework that combines the Xgboost and k-means algorithms. Section 4 presents the forecasting framework, which combines the EMD and LSTM neural networks. Section 5 presents the experimental design and numerical test results. Lastly, Section 6 provides the conclusions of this study.

2. Data Analysis

The analysis of the relationship between the load data and external variables that affect the electric load is necessary to achieve high forecasting accuracy. This analysis is based on the electricity load data (provided by ISO New England) measured at one-hour intervals from 2003 to 2016. This section describes the major load-affecting factors, including temperature and day-type index. We also analyze the relationship between the daily and day-ahead peak loads.

Evidently, temperature changes are the primary cause of electricity load changes. In particular, the temperature variation range often determines the variation range of the electricity load. The variation in the interval-valued load with respect to the interval-valued temperature is shown in Figure 1. In the summer season, the higher the temperature is, the larger the electricity load value becomes (see Figure 1a). That is, a positive correlation exists between the load and temperature. By contrast, this correlation becomes negative in the winter season (Figure 1b). The preceding analysis indicates the necessity of discussing the effect of temperature on electricity load from one season to another.

Different day-types have different daily load curves, and the load of different day-types, such as weekends, holidays, and working days, are also different. The load of a working day is often higher than that on the weekend due to the decrease in industrial load on weekends. Accordingly, load on Saturdays is lower compared with those on other days (see Figure 2). Mondays and Tuesdays typically have the largest energy consumption over the week. Evidently, non-working days have considerably low energy consumption. Therefore, day-types are an important feature that cannot be ignored.

Although we have already identified several features that affect load forecasting, prediction errors may still be large during peak hours in the STLF process. Thus, we suppose that the day-ahead peak load is an important feature for forecasting. The Figure 3 is a scatter plot between the day-ahead peak load and daily load from 1 March 2003 to 31 October 2016. We determine that these two variables are closely related with the correlation coefficient = 0.8754. This result confirms the necessity of one day-ahead peak load to be a supplemental input feature to the SD selection and LSTM training processes.

Precipitation and wind speed also have a bearing on the electricity load. Load on a sunny day is significantly higher than that on a rainy day. Therefore, improving the prediction accuracy is possible by selecting SD and maximizing the historical and features data.

3. Similar Day Selection: Improved K-Means with Extreme Gradient Boosting

If exogenous features, such as temperature, are included, then the traditional load forecasting model could lead to slow convergence and poor prediction accuracy. Thus, we select the SD load as the input data to improve the prediction power.

Clustering based on the feature values of the data and similar samples gathered in the same cluster can substantially improve the selection of SDs with the forecasting day. The performance of the clustering algorithm depends on the distance between records. “It is misleading to calculate the distance by measuring all attributes equally. The distance between neighbors will be dominated by the large number of irrelevant attributes, which sometimes leads to the dimensionality curse” [30]. An effective method to overcome this problem is to add a weighted parameter for each feature. Hence, the more relevant the feature is, the larger the impact of this feature becomes on the clustering results.

This section presents an alternative to SD selection that calculates the weights of the features using the Xgboost algorithm and integrates the weighted features using the k-means clustering.

3.1. Feature-Weight Learning Algorithm: Extreme Gradient Boosting

Xgboost [31] is an improved algorithm based on the gradient boosting decision tree and can construct boosted trees efficiently and operate in parallel. The boosted trees in Xgboost are divided into regression and classification trees. The core of the algorithm is to optimize the value of the objective function.

Unlike the use of feature vectors to calculate the similarity between the forecasting and history days, gradient boosting constructs the boosted trees to intelligently obtain the feature scores, thereby indicating the importance of each feature to the training model. The more a feature is used to make key decisions with boosted trees, the higher its score becomes. The algorithm counts out the importance by “gain”, “frequency”, and “cover” [32]. Gain is the main reference factor of the importance of a feature in the tree branches. Frequency, which is a simple version of gain, is the number of a feature in all constructed trees. Cover is the relative value of a feature observation. In this study the feature importance is set by “gain”.

For a single decision tree T, Breiman et al. [33] proposed

$$w_{\ell }^2\left(T\right)=\sum _{t=1}^{J-1}{\widehat{\tau }}_t^2$$

(1)

as a score of importance for each predictor feature $X_{\ell }$. The decision tree has $J-1$ internal nodes, and partitions the region into two subregions at every node t by the prediction feature $X_{\ell }$. The selected feature is the one that provides maximal estimated improvement ${\widehat{\tau }}_t^2$ in the squared error risk over that for a constant fit over the entire region. The squared importance of the feature $X_{\ell }$ is the sum of such squared improvement over the $J-1$ nodes, for which it was selected as the splitting feature. The following formula represents the importance calculation over the additive M trees.

$$w_{\ell }^2\left(T\right)=\frac{1}{M}\sum _{\mathrm{m}=1}^M{\widehat{\tau }}_t^2\left(T_m\right)$$

(2)

The importance of a feature depends on whether the prediction performance changes considerably when such feature is replaced with random noise. Given the data analysis in the previous section, we take several features as input for the Xgboost algorithm to calculate the feature importance with the electricity load. We can obtain how each feature contributes to the prediction performance in the training course of the Xgboost algorithm. Evidently, the electricity load is sensitive to temperature variables (see Figure 4). Moreover, the supplement features (i.e., day-ahead-peak load) are an important feature for load forecasting. This conclusion is consistent with the results of the data analysis. We have now derived the important values of all features, which will be used as a priori knowledge of the subsequent clustering algorithm.

3.2. K-Means Clustering Based on Feature-Weight

K-means, which was first proposed by MacQueen in 1967 [34], is extensively applied in many fields and sensitive to the selection of the initial cluster centroids. We selected the initial cluster centers with the maximum distance method to diminish the probability of converging to a local optimum. This section improves the k-means clustering by computing the initial cluster centers and utilizing the new distance calculation method. The steps are presented as follows.

• Given a data set $X=\{x_1,x_2,...,x_n\}$ and an integer value K. The data set is normalized as follows:

  $$x_i=\frac{x_i-x_{i\min }}{x_{i\max }-x_{i\min }}$$

  (3)

  where $x_{i\min }$ and $x_{i\max }$ denote the minimum and maximum values, respectively, of each input factor.
• The forecasting day is selected as the first center $u_0$
• The next center $u_j$ is selected, where $u_j$ is the farthest point from the previously selected cluster centers $\{u_0,u_1,...,u_{j-1}\}$. Steps 2 and 3 are repeated until the K centers have been identified.
• The feature weights are calculated using the Xgboost algorithm. Thereafter, the weights are attributed to each feature, thereby providing them with different levels of importance. Let $w_p$ be the weight associated with the feature p. The norm is presented as follows.

  $$d(x_i,x_j)=\sqrt{w_1{(x_{i1}-x_{j1})}^2+···+w_p{(x_{ip}-x_{jp})}^2+···+w_n{(x_{in}-x_{jn})}^2}$$

  (4)
  (1) Each data point is assigned to the nearest cluster.
  (2) The clusters are updated by recalculating the cluster centroid. The algorithm repeatedly executes (1) and (2) until convergence is reached.

The key idea in selecting SDs is to determine the attribute weights using the Xgboost algorithm and calculating the distance between the selected day and the day that relies on measuring different attributes in different weights. In Figure 5, the horizontal coordinate-axis presents the time (hour), whereas the longitudinal coordinate-axis presents the load curves. The color that changes from light to dark means that the electric load values change from large to small. Figure 5a shows the heat map for the original load data set, where every curve is evidently different in shape. Figure 5b,c are the heat maps for the original load data after the simple k-means clustering and weight k-means clustering, respectively. Our proposed Xgboost-k-means method can merge SDs into one cluster more effectively than the simple k-means algorithm does. Therefore, the SD can be the input data for subsequent load forecasting.

4. LSTM with Empirical Mode Decomposition

Neural networks are extensively employed in time series forecasting. However, determining the structure is difficult and often falls into the local minimum. The EMD method can facilitate the determination of the characteristics of the complex non-linear or non-stationary time series, i.e., it can divide the singular values into separated IMFs and determines the general trend of the real time series. This can effectively reduce the unnecessary interactions among singular values and improve the performance when a single kernel function is used in forecasting. Thus, this section proposes a model that combines the EMD and LSTM neural networks for STLF.

4.1. Empirical Mode Decomposition

EMD is a new signal processing method proposed by Huang et al. in 1998 [26]. The original signal was derived from the data’s characteristics and can be decomposed into the intrinsic mode functions (IMF) by EMD. Thus, EMD can effectively decompose the singular values and avoid trapping into a local optimum, thereby improving the performance and robustness of the model.

All IMFs must meet the following conditions:

• For a set of data sequences, the number of extremal points must be equal to the number of zero crossings or, at most, differ by one.
• For any point, the mean value of the envelope of the local maxima and local minima must be zero.

For the original signal $x\left(t\right)$, EMD decomposes $x\left(t\right)$ through the “sifting” process, which is described as follows.

• Identify all the maxima and minima of signal $x\left(t\right)$.
• Through the cubic spline interpolation fitting out the upper envelope $u\left(t\right)$ and lower envelope $l\left(t\right)$ of signal $x\left(t\right)$. The mean of the two envelopes can be the average envelope curve $m_1\left(t\right)$ :

  $$m_1\left(t\right)=\frac{u\left(t\right)+l\left(t\right)}{2}$$

  (5)
• Subtraction of m from $x\left(t\right)$ to obtain an IMF candidate :

  $$h_1\left(t\right)=x\left(t\right)-m_1\left(t\right)$$

  (6)
• If $h_1\left(t\right)$ does not satisfy the two conditions of the IMF, then it should take $h_1\left(t\right)$ as original signal and repeat above calculate k times. At this point, $h_{1k}\left(t\right)$ could be as shown in Equation (7):

  $$h_{1k}\left(t\right)=h_{1(k-1)}\left(t\right)-m_{1k}\left(t\right)$$

  (7)

  $h_{1(k-1)}\left(t\right)$ and $h_{1k}\left(t\right)$ present the signal after shifting $k-1$ times and k times,respectively. $m_{1k}\left(t\right)$ is the average envelope of $h_{1k}\left(t\right)$
• If $h_{1(k-1)}\left(t\right)$ satisfies the conditions of the IMF, define $h_{1k}\left(t\right)$ as $c_1\left(t\right)$. Standard deviation is defined by Equation (8):

  $${\mathrm{S}}_d={\sum _{t=0}^T\frac{|h_{1(k-1)}\left(t\right)-h_{1k}\left(k\right)|}{{h_{1k}}^2\left(t\right)}}^2\in (0.2,0.3)$$

  (8)
• Subtraction of $c_1\left(t\right)$ from $x\left(t\right)$ to obtain new signal $r_1\left(t\right)$

  $$r_1\left(t\right)=x\left(t\right)-c_1\left(t\right)$$

  (9)
• Repeat previous steps 1 to 6 until the $r_n\left(t\right)$ cannot be decomposed into the IMF. $r_n\left(t\right)$ is the residual of the original data $x\left(t\right)$. Finally, the original signal $x\left(t\right)$ can be presented as a collection of n components $u_i\left(t\right)$$(\mathrm{i}=1,2,...,\mathrm{n})$ and a residual $r_n\left(t\right)$:

  $$\mathrm{x}\left(t\right)=\sum _{i=1}^n{u}_i\left(t\right)+r_n\left(t\right)$$

  (10)

The preceding steps show that the EMD method is employed to decompose the SD load at low and high frequencies, respectively. Figure 6 evidently shows the decomposition of the eight IMF extractions and residuals. Furthermore, all graphs in Figure 6 are shown in the same scale, thereby enabling the assessment of the contribution of each extracted IMF.

4.2. Lstm-Based Rnn for Electric Load Forecasting

LSTM was proposed by Hochreiter et al. in 1997 [35] as a type of efficient RNN architecture, and has been extensively applied in various fields. Moreover, LSTM is a popular time series forecasting model and can expertly deal with long-term dependencies data.

A. Recurrent Neural Networks (RNNs)

RNNs are designed to operate over the non-linear time-varying problem [24]. The RNN internal connections enable signals to travel forward and backward, thereby making RNNs substantially suitable for time series prediction.

RNNs can mine the rules from the time sequences to predict the data that have yet to occur [36,37]. The reason for this characteristic lies in the feedback connections that can facilitate updating the weights based on the residual in each forward step (Figure 7). The forecasting day load in STLF is bound up with the SD load. Therefore, if we provided the SD time sequences, then obtaining high accuracy on the forecasting day becomes possible.

RNN proves to be suitable for this problem [38]. However, RNNs tend to suffer heavily from gradient vanishing, which may increase indefinitely and eventually cause the network to break down. Therefore, simple RNNs may not be the ideal option for forecasting problems with long-term dependencies.

B. LSTM-Based RNN Forecasting Scheme

LSTM was mainly motivated and designed to overcome the vanishing gradients problem of the standard RNN when dealing with long term dependencies.This section leads to the long short-term memory neural network. The LSTM model add the input gate,output gate and forgetting gate to the neurons in RNN. Such a structure can effectively mitigate the vanishing gradient problem [39]. This makes LSTM an architecture suitable for problems with long term dependencies.

The major innovation of LSTM is its memory cell, which essentially acts as an accumulator of the state information. First, as shown in Figure 8, the forget gate is applied to decide what information to get rid of the cell state. A sigmoid function is used to calculate the activation of the forget gate $f_t$.

$$f_t=\sigma (W_f·[h_{t-1},x_t]+b_f)$$

(11)

The second step is to determine what new information should be stored in the cell state. To start with, a sigmoid layer named the “input gate layer” decides which information should be update. Then, a tanh layer creates a vector ${\tilde{c}}_t$ of new candidate values should be updated in the next state.

$$\begin{array}{c}{i}_t=\sigma (W_i·[h_{t-1},x_t]+b_i)\hfill \end{array}$$

(12)

$${\tilde{c}}_t=\tanh (W_c·[h_{t-1},x_t]+b_c)$$

(13)

Next, we will update the old cell state $c_{t-1}$ into the new cell state $c_t$. We multiply $c_{t-1}$ by $f_t$ for throw away the information from old cell. Then we add $i_t*{\tilde{c}}_t$. There are the new candidate values, scaled by how much information should be updated to each state value.

$$c_t=f_t*c_{t-1}+i_t*{\tilde{c}}_t$$

(14)

Lastly, we need to decide the output. This has two parts: we run a sigmoid layer as output gate to filter the cell state firstly. Then, we put the cell state through $\tanh (·)$ and multiply it by the output $o_t$ to calculate the desired information.

$$o_t=\sigma (W_o·[h_{t-1},x_t]+b_o)$$

(15)

$$h_t=o_t*\tanh \left(c_t\right)$$

(16)

In Equations (11)–(16), $W_i$, $W_f$, $W_c$, $W_o$ represents the appropriate weight matrices. The vectors $b_i$, $b_f$, $b_c$, $b_o$ denote the corresponding bias vectors.

This study presents the experiments that apply the separated LSTM neural networks scheme for the SD load’s IMF and residual forecasting. The training process inputs include the temperature, day-ahead-peak, humidity, day-type index, precipitation, wind speed, and IMF component of the SD load. The model framework is shown in Figure 9. In order to further improve the accuracy and practicability of the prediction model, we establish the architecture based on LSTM, named sequence to sequence (S2S). Sequence to sequence structure can adjust the length of the input and output sequences flexibly, that is appropriate to perform different time scales load forecasting.

It is obvious that this architecture consists of two parts: encoder and decoder. The encoder aims to prediction the load we have already known, yet $y_{[M-1]}$ and $f_{\left[M\right]}$ present the load’s IMF component of previous hour and the features of this hour, respectively. $f_{\left[M\right]}$ can be expressed as:

$$\begin{array}{c}{f}_{\left[M\right]}=[Temperatur{e}_{\left[M\right]}Humidit{y}_{\left[M\right]}Day-typ{e}_{\left[M\right]}Precipitatio{n}_{\left[M\right]}\hfill \\ Windspee{d}_{\left[M\right]}Day-ahead-peakloa{d}_{\left[M\right]}]\hfill \end{array}$$

(17)

Then, the decoder generates an output sequence $\widehat{y}=\left\{{\widehat{y}}_{\left[M\right]},{\widehat{y}}_{[M+1]},...,{\widehat{y}}_{[M+n]}\right\}$, that is the prediction of IMF component of the load for the next $n+1$ hours.

Standard backpropagation can be applied to train the network using a gradient based method called Stochastic Gradient Descent (SGD).

Table 1. MAPE(%) FOR S2S ARCHITECTURE (one-day-ahead load forecasting).

Layers	Units	MAPE (Training)	MAPE (Testing)
1	5	1.13	1.026
1	20	1.079	1.043
1	50	1.023	1.072
1	100	0.944	1.115
2	5	1.095	1.028
2	20	1.021	1.061
2	50	0.962	1.012
2	100	1.078	1.103
3	5	1.083	1.116
3	20	1.05	1.051
3	50	0.958	1.138
3	100	0.934	1.342

shows the Mean Absolute Percentage Error (MAPE) on training and testing datasets for different number of layers and units using the S2S architecture for data with one-day-ahead forecasting.

It can be seen that the proposed architecture is able to produce very low errors on training dataset. Further, it is noticed that increasing the capacity of the network by increasing the number of layers and units only improves error on training dataset. The model performs well on training dataset using a 2-layer network with 50 units in each layer. However, increasing the capacity of the network does not improve performance on testing data. In order to improve accuracy on testing data Dropout is used as regularization methodology.

4.3. The Full Procedure of SD-EMD-LSTM Model

The complete procedure of the proposed SD-EMD-LSTM model is presented as follows and illustrated in Figure 10.

• Similar days selection. Calculate the features weight by xgboost method, then combined with K-means algorithm to determine the similar days cluster.
• Data decomposition. Take similar days load as input data, and decompose the input data into several intrinsic mode functions with EMD algorithm.
• Forecasting. Separated LSTM neural networks employed to forecast each IMF and residual, respectively. Reconstruct the forecasting values from each single LSTM model.

5. Numerical Experiments

This section presents the forecasting performance of the proposed SD-EMD-LSTM model. The hourly electric load data from NE-ISO 2003 to 2016 is employed for the models. The forecasting has been conducted in two time scales, namely, one day ahead (24 h) and one week ahead (168 h).

First, we present the experiments on applying the weighted k-means-based SD selection algorithm for load forecasting, as well as analyze the optimum value of the clusters k. Second, we verify the clustering effect of the proposed SD selection method and the need of using the supplemental feature. Third, experiments in two time scales are conducted to compare the proposed model with the standalone LSTM, SD-LSTM, and EMD-LSTM models to show the fitting effect of the hybrid model. Lastly, we compare the forecasting performance with three other models (i.e., ARIMA, BPNN, and SVR) to illustrate the forecasting accuracy and stability of the SD-EMD-LSTM model. The structure of BPNN model comprises of 3 layers viz. input, hidden and output layers(6-20-1), where the transfer functions of hidden layer and output layer are tansig and purelin, respectively. While the training function uses traingdm, the learning function of threshold and weights use learnged. SVR with LIBSVM package with $C=8.4065$, $\gamma =0.0869335$, $\epsilon =0.000118$.

5.1. Evaluation Indices for the Forecasting Performance

The mean absolute percentage error (MAPE) is employed as a criterion of error evaluation to analyze the forecasting performance.

$$MAPE=\frac{1}{m}\sum _{j=1}^m\left|\frac{X_j-{\widehat{X}}_j}{X_j}\right|$$

(18)

where ${\widehat{X}}_j$ is the forecasting value, $X_j$ is the actual value, and m is the total number of forecasting points. For the two forecasting time scales, m is set at 24 h and 168 h.

5.2. Empirical Results and Analysis

We perform simulations of the four examples to verify the predictive ability of the proposed method:

Example 1: Through the enumeration method, k ranges from 5 to 12, the run is repeated several times in each k value using the Xgboost-k-means-based SD-EMD-LSTM model. Thereafter, the prediction accuracy of each k is calculated. Experiments of the 24-h-ahead forecasting in different seasons are performed to analyze the best k value with the highest prediction accuracy. Figure 11 shows that when the number of clusters equals 9, the prediction curve most closely follows the raw curve in four days, including 30 October 2016, 5 July 2015, 13 April 2014, and 22 February 2013, which represent the four seasons.

MAPE can also be used to determine the ideal number of clusters. Comparison results (see

Table 2. MAPE(%) for the different number of clusters in one-day ahead prediction.

k	5	6	7	8	9	10	11	12
MAPE(%)	3.81	3.45	3.08	2.05	0.97	1.42	1.94	2.12

) show that the proposed model with 9 clusters outperformed all the other cluster numbers with the smallest forecasting MAPE of 0.97%. That is, the proposed Xgboost-k-means method can effectively merge SDs into one cluster. Consequently, we define k = 9 as a priori knowledge in the proposed SD-EMD-LSTM model to select SD for the subsequent load forecasting.

Example 2: This example includes two cases. Case 1 verifies the clustering effect of the proposed SD selection method, and the simple k-means algorithm is used to the SD selection for comparison with the proposed Xgboost-k-means model. Case 2 demonstrates the importance of using the supplemental feature, namely, day-ahead peak load. The training period in this example is from 2003 to 2015, and the prediction period is 2016.

Case 1: EMD-LSTM is combined with the proposed SD selection method and simple k-means algorithm respectively to verify its performance. In the one-day ahead load forecasting as shown in Figure 12, Xgboost-k-means hybrid with the EMD-LSTM model fits the raw data better than the simple k-means clustering algorithm. That is, the Xgboost-k-means algorithm could merge SDs into one cluster more effectively, thereby improving the prediction accuracy.

Table 3. Monthly MAPE(%) of validation phase.

Model	January	February	March	April	May	June	July	August	September	October	November	December
SD-EMD-LSTM	1.03	0.96	0.83	1.09	0.77	0.93	0.96	0.88	1.06	1.12	0.91	1.12
kmeans-EMD-LSTM	2.11	2.17	1.98	2.56	1.83	2.16	1.96	2.03	2.08	1.79	2.04	2.18

also verifies this scenario, which shows that the SD-EMD-LSTM model achieved an improved forecasting performance with a considerably small MAPE, as well as agrees with the conclusion presented in Section 3. The reason lies in that the Xgboost algorithm has considerable ability to access each feature’s weight, the limitation of the dimensionality is generally reduced, and the models are obtained with increasing the forecasting accuracy.

Case 2: The SD-EMD-LSTM model is used with and without the supplemental feature (i.e., day-ahead peak load) to analyze the prediction accuracy on the one-day ahead load forecasting. Further details are shown in Figure 13.

The most significant forecasting errors often occur at the peak points of the forecast load curve. The reason is that the proposed model with the supplemental feature (i.e., day-ahead peak load) can achieve an improved forecasting performance at the peak points.

The hourly mean absolute percentage errors listed in

Table 4. 24-h forecast MAPE (%) of validation phase.

Hour	1	2	3	4	5	6	7	8	9	10	11	12
SD-EMD-LSTM	1.77	1.42	1.73	2.00	1.87	1.63	0.81	0.20	1.09	1.99	1.75	0.96
SD’-EMD-LSTM	2.20	2.25	2.26	2.21	2.44	2.41	0.93	0.61	1.33	2.13	1.99	0.82
13	14	15	16	17	18	19	20	21	22	23	24	Average
0.49	1.57	1.22	0.47	0.27	0.71	0.29	0.46	0.25	1.33	1.48	0.52	1.10
0.12	1.01	1.46	1.05	1.41	2.40	2.32	1.07	0.15	1.73	1.02	0.33	1.44

indicate that the proposed model with supplemental input feature (i.e., day-ahead peak load) obtained an average MAPE of 1.10%. This value is lower than the 1.44% obtained in the model without the supplemental input feature. Furthermore, SD-EMD-LSTM with supplemental input feature has good prediction accuracy during peak hours (i.e., from 15:00 to 20:00). Therefore, the day-ahead peak load should be the supplemental input feature for load forecasting.

Example 3: This example compares the forecasting value of the proposed SD-EMD-LSTM model to the single LSTM, SD-LSTM, and EMD-LSTM models. The experiments have a forecasting horizon of h = 24 h and h = 168 h. The training period is from 2003 to 2014, and the prediction period is 2015.

Figure 14a,b show the one-day ahead and one-week ahead forecasting results of the single LSTM, SD-LSTM, EMD-LSTM, and proposed SD-EMD-LSTM models, respectively.

We can conclude from Figure 14 that the forecasting curve of the proposed SD-EMD-LSTM model follows the raw data better than the other alternative models for the two forecasting horizon in Example 3. Evidently, comparing the LSTM curve with those of SD-LSTM and EMD-LSTM shows that the SD selection can generally enhances the accuracy of the load forecasting in the one-day-ahead and one-week-ahead forecasting. EMD can also effectively determine the general trend of the real time series.

Table 5. Mape(%) for Load Forecasting in Example 3.

	24-h Ahead				168-h Ahead
	LSTM	SD-LSTM	EMD-LSTM	SD-EMD-LSTM	LSTM	SD-LSTM	EMD-LSTM	SD-EMD-LSTM
January	5.97	2.7	4.39	1.19	9.86	3.37	6.46	1.57
February	4.3	2.13	3.73	0.98	7.88	2.43	5.69	1.22
March	6.89	1.65	4.58	0.66	6.26	2.62	4.77	1.25
April	5.19	2.46	3.63	0.72	7.93	4.34	4.93	2.08
May	6.85	3.97	5.17	1.26	9.65	3.21	6.65	1.79
June	5.64	1.02	3.4	0.83	7.49	3.1	5.12	0.91
July	5.9	2.72	4.09	1.24	9.19	3.13	7.11	1.04
August	4.59	2.16	2.44	1.12	8.3	4.97	6.72	1.63
September	4.31	1.88	3.11	0.85	8.69	4.08	6.07	1.59
October	6.82	2.89	5.94	1.65	8.53	5.02	6.69	1.87
November	4.2	2.84	3.15	1.35	11.81	6.16	8.03	2.31
December	4.46	2.01	3.68	1.05	9.31	3.88	7.96	1.79
Average	5.43	2.37	3.94	1.08	8.74	3.86	6.35	1.59

shows the MAPE values per month of all the models in Example 3. The last row of Table 4 lists the average MAPE values for the experiment based on 12 months. The LSTM neural networks combined with the Xgboost-k-means-based SD selection method is better than the LSTM neural networks combined with the EMD model but is slightly inferior to the SD-EMD-LSTM model. The evaluation results of the MAPE indexes and prediction curves for the four models tend to be consistent.

Example 3 enables us to conclude the following points.

(1)

The fitting effect of the hybrid model is evidently better than that of the single LSTM neural networks model in both time scales.

(2)

The Xgboost-k-means method can effectively merge SDs into one cluster and prevent the LSTM neural networks from being trapped into a local optimum, thereby substantially improving the prediction accuracy.

(3)

The data decomposition method divides the singular values into separated IMFs and determines the general trend of the real time series, thereby effectively improving the performance and robustness of the model.

In general, the SD-EMD-LSTM model significantly outperforms the three other methods and achieves a good prediction effect in STLF.

Example 4: This example compares the forecasting results of ARIMA, BPNN, SVR, and the proposed SD-EMD-LSTM model. For a fair comparison, we compare their performance with the same input data sets (i.e., SDs). Figure 15a,b show the one-day ahead and one-week ahead forecasting results, respectively. The training period is from 2003 to 2013, and the prediction period is 2014.

Figure 15 shows that the forecasting curve of the proposed SD-EMD-LSTM model is closer to the raw load curve than the other alternative models in Example 4. The performance results of the three other methods are insufficient for STLF.

From the MAPE values in

Table 6. MAPE(%) For Load Forecasting in Example 4.

	24-h Ahead				168-h Ahead
	SD-ARIMA	SD-BPNN	SD-SVR	SD-EMD-LSTM	SD-ARIMA	SD-BPNN	SD-SVR	SD-EMD-LSTM
January	5.47	4.27	4.29	1.23	11.55	7.52	8.3	2.36
February	4 .43	1.79	1.96	0.75	6.97	9.03	2.29	1.05
March	3.34	2.84	3.5	1.12	5.04	3.42	5.25	1.29
April	4.05	3.74	6.08	1.04	7.94	4.56	7.52	1.09
May	7.88	4.43	2.43	0.95	9.36	5.26	2.41	1.83
June	5.49	5.68	2.93	0.82	6.44	9.26	6.96	1.92
July	3.61	1.92	2.07	1.12	8.5	3.81	3.18	1.69
August	5.86	2.68	3.58	1.87	10.65	9.12	6.29	2.03
September	4.51	1.86	3.68	0.98	9.72	2.19	4.33	1.26
October	6.07	4.28	6.51	0.76	7.98	5.77	8.85	1.28
November	8.51	3.85	2.13	0.99	6.11	4.52	3.72	1.75
December	10.05	5.71	3.47	0.87	11.73	5.44	9.5	1.22
Average	5.48	3.39	3.55	1.04	8.50	5.83	5.72	1.56

, the experiment results indicate that the proposed model is significantly superior to the SVR, ARIMA, and BPNN models. MAPE of the SD-EMD-LSTM model is the lowest among all the models. Its prediction accuracy also reaches 98.96% and 98.44% in the 24-h-ahead and 168-h-ahead forecasting, respectively. ARIMA has the maximum MAPE value. Although the three other models determined the general trend of the raw data, their forecasting errors were extremely high.

The comparison between the two different forecasting time scales demonstrate that the accuracy of the proposed hybrid model exhibit minimal changes because the LSTM neural networks can maximize the long-term dependencies in the electric load time series for substantially accurate forecasting. That is, the SD-EMD-LSTM model can perform longer-horizon load forecasting. Overall, the proposed hybrid model provides a powerful method that can outperform many other forecasting methods in the challenging STLF problem.

6. Conclusions

This study presents an LSTM neural network model hybridized with the SD selection and EMD methods for STLF. The key idea in selecting SDs is to determine the attribute weights using the Xgboost algorithm and calculate the distance between the selected day and the day that relies on the different measured attributes in different weights. Thereafter, the k-means algorithm merges SDs into one cluster as input data for the subsequent forecasting based on the Xgboost distance. EMD eventually determines the key features of the SD load at low and high frequencies. Lastly, the separated LSTM neural networks are used to forecast the future values in low-frequency and high-frequency time series. The proposed method has been compared with the LSTM, SD-LSTM, EMD-LSTM, ARIMA, BPNN, and SVR models in real-load data obtained from the NE-ISO for the one-day ahead and one-week ahead load forecasting. Comparison results demonstrate that the proposed Xgboost-k-means method can effectively merge SDs into one cluster. Moreover, the EMD-LSTM model has a good ability to accurately forecast the complex non-linear electric load time series over a long horizon. The aforementioned analysis implies that the proposed SD-EMD-LSTM framework can be a promising alternative approach to STLF.