1. Introduction
Confident predictions are necessary to build reliable, fully Artificial Intelligence (AI) automated smart energy grid systems [
1,
2,
3,
4,
5,
6]. Load predictions produced without any information about their uncertainty are unsafe [
7] when critical autonomous decisions are being made in energy production and distribution. Current forecasting models normally do not provide information about uncertainty in their predictions, which can lead to costly and risky decisions and affect current efforts to produce a reliable smart energy grid system [
8]. This is a problem for many AI applications, such as autonomous vehicles, where the lack of consideration for uncertainty can be dangerous [
9]. Therefore, for safe, reliable, autonomous artificial intelligence systems and for safe prediction-based decisions, evaluating the degree of uncertainty is paramount [
10].
At the same time, the efficient allocation of resources is a primary goal of the electric power industry, which includes the production, transportation, and distribution of energy. An accurate forecast of the future electric load is integral to achieving this, as it allows utility companies to make informed decisions for planning and operations [
2,
3,
11]. Accurately predicting the hourly load, known as Short-Term Electric Load Forecasting (STELF), allows utilities to make these decisions in real-time and saves millions of dollars normally wasted due to overload or underload costs as well as reducing CO
2 emissions [
4,
12,
13,
14].
Obtaining confident and accurate estimates for learning models through data-driven methods has been the subject of intensive research [
4]. A recent survey [
13] showed that previous studies primarily used artificial neural networks (ANNs) and support vector machines (SVMs), with decision trees and various statistical methods making up the remainder. Of these, ANNs have become the most common choice due to advances in deep learning and the development of neural net architectures such as the long short-term memory (LSTM) network [
15], designed for sequence modeling and time series tasks. LSTM networks have been employed successfully for a wide range of applications, including short-term electric load forecasting [
13,
16,
17,
18,
19]. At the same time, tree-based algorithms such as gradient-boosted trees have become widely used in industry and machine learning competitions due to lower computation requirements, ease of interpretation and tuning, and better performance on smaller datasets, and have recently been applied to load forecasting as well [
20,
21,
22].
In this paper, we present the implementation and testing of two methods for estimating uncertainty of short-term electrical load prediction, based on two popular machine learning algorithms: LSTM and gradient boosting regression (GBR). The contributions are fourfold. First, we developed two energy forecasting computational models using deep learning techniques. LSTM represents a class of advanced artificial recurrent neural networks (RNN) that has been shown to be efficient in many applications. However, to our knowledge, this is the first time that LSTM has been applied together with Monte Carlo dropout regularization to quantify the prediction uncertainty for energy forecasting. Second, we applied the optimization algorithm GBR to energy forecasting. More importantly, using the properties of the GBR to accept an arbitrary differentiable loss function, we were able to construct prediction intervals to measure the uncertainty of the prediction. Third, the applicability and effectiveness of the models are verified and analyzed using real-world data–a benchmark dataset on electricity load forecasting. The impact of implementing such methods on predictive accuracy, as well as the effect of input features on the produced uncertainty are also examined. Lastly, experimental results show that the models achieved high prediction accuracies in terms of the Mean Absolute Percent Error (MAPE) and that uncertainty estimates can be obtained with minimal cost to accuracy.
The rest of the paper is organized as follows. In
Section 2, we introduce the algorithms, uncertainty estimation, and the design of the learning models.
Section 3 describes the processing of the dataset, implementation, and experimental results. Finally, some conclusions and future work are presented in
Section 4.
3. Experiment and Results
We now discuss our implementations of the methods described above, as well as empirical results.
3.1. Model Implementation
We use the Python-based scikit-learn package [
36] and the TensorFlow and Keras libraries [
16,
24,
37,
38] to implement three learning models. The first model is the standard LSTM architecture as used in practice [
17,
26], consisting of the input layer, three LSTM hidden layers with 50 neurons each, and a single neuron in the output layer.
The second model implements uncertainty estimation as described above, by performing Monte Carlo sampling with dropout to approximate a Bayesian posterior distribution for the predictions (Algorithm 1). We then construct a 90% interval based on the sample variance as the uncertainty estimate. The Mean Absolute Error (MAE) loss function and the Adam version of stochastic gradient descent were used with Keras and Tensorflow to optimize the LSTM models.
The third model is a gradient boosting regressor implemented using the
GradientBoos tingRegressor function from the sci-kit-learn package [
36] and optimized for Mean Absolute Error (MAE) in MWh unit. Two auxiliary models are implemented the same way, but optimized using quantile loss for the 95th and 5th quantiles. The 90% prediction interval is constructed using the outputs from the auxiliary models, while the main model provides the prediction.
Table 1 lists the configuration parameters for the implementation.
Algorithm 1: MC dropout Algorithm for model
uncertainty. |
|
3.2. Data and Feature Selection
The data used in this paper was obtained from the European Network on Intelligent Technologies (EUNITE) 2001 competition on electricity load forecasting [
39]. The data was collected by the Eastern Slovakian Electricity Corporation for two years from 1 January 1997 until 31 December 1998, and contains the following features:
- Half-hourly electricity load in MWh.
- Daily average temperature in Celsius.
- Annual holidays.
The objective of the original competition was to predict the daily peak electricity load for the 31 days of January 1999 using the given historical data. For this paper, our objective is to instead predict the electricity load for each half-hour for the same time period and compare the forecast with actual consumption.
The EUNITE competition data has been studied by many authors [
40,
41] to investigate the relationship between load data and other features such as temperature, weekdays, and annual holidays.
3.2.1. Temperature Effects
References [
40,
41] show from the load and average daily temperature data that the electricity load has high consumption in the winter (September–March) due to heater usage in the winter, and low consumption in the summer (April–August). This behavior indicate a strong relationship between the electricity load and the weather conditions represented by temperature. A negative correlation with coefficient −0.8676 was found between the daily peak load and daily average temperature over the two years [
40,
41], showing this strong relation.
3.2.2. Weekday Effects
Examination of load data during the weekdays shows that a periodic weekly pattern exists in the consumption. References [
40,
41] show that on most Saturdays and Sundays (the weekends), the load is lower than on weekdays (Monday–Friday). In addition, electricity peak load usually occurs on a Wednesday.
3.2.3. Holiday Effects
The analysis for the two-year historical load data shows that the load usually reduces on holidays [
40,
41]. The load demand also depends on the type of holiday; for example, on Christmas or New Year, the electricity consumption is affected more compared to other public holidays.
3.2.4. Feature Selection
For high-accuracy load forecasting, our model needs the most descriptive input features that have the highest correlation with the output forecasting variable and the highest degree of linear independence.
From the above data analysis, the best candidate input features are the past half-hourly loads, the average daily temperatures, and the type of day [
40,
41].
3.2.5. Half-Hourly Load
Past half-hourly load is a very good indicator of the next timestep’s load. Including the past half-hourly loads as a key feature is very important for high-accuracy forecasting.
3.2.6. Average Daily Temperature
From the above data analysis, electricity load and the average daily temperature have a high negative correlation. Therefore, the average daily temperature is a crucial feature for our forecasting model.
3.2.7. Weekday Indicator
The weekday effect on electricity load was shown in the data analysis as a pattern in references [
40,
41]. Therefore, the type of day can be encoded numerically as a feature in our forecasting model to enhance its performance. This can be done in Python by using the method, date.weekday(), which returns the day of the week as an integer, where Monday is 0 and Sunday is 6.
3.2.8. Public Holidays
Public holidays have a significant effect on users’ behaviors for electricity consumption. Hence, inclusion of a binary indicator 0/1 for public holidays in our forecasting model will benefit its accuracy.
3.2.9. Time Series Modeling
After encoding indicators of both weekdays and public holidays as numerical values, the EUNITE competition data was framed as multivariate time series data consisting of sequences of records ordered by date and time, as shown in
Table 2.
3.2.10. Data Normalization
The range of values for the EUNITE competition data varies widely with feature, which affects both the accuracy and the stability of our forecasting model [
16,
24,
37,
38]. Therefore, the ranges of all features have been normalized by rescaling each feature to lie within
. This scaling was done in Python using the
MinMaxScalers method from the sklearn package [
24,
37].
3.2.11. Convert Data to Input–Output Pairs
The main task for our models in a supervised machine learning setting is to learn the function that maps inputs to an output, based on examples from labeled training data consisting of known input–output pairs [
16,
24].
The EUNITE competition data must be reframed as a supervised machine learning problem in order to be handled by our models for forecasting electricity load.
Therefore, the multivariate time series was converted to input–output pairs using a sliding window method, in which prior time steps are used as features for predicting the next time step [
38,
42,
43].
3.3. Splitting Data into Train and Test Sets
The two years of historical data was split into training and testing sets, with
for training and
for testing. The training and testing sets were split into input and output variables, and finally, the inputs were reshaped into the 3D format expected by our models, namely, [samples, time steps, features] [
16,
24,
37,
38]. The training set was used first to train our models, then the testing set was used to evaluate it.
3.4. Experimental Results
The three models—namely, the standard LSTM Model, the Bayesian LSTM model, and the GBR model—were trained and used to forecast the electricity load for the month of January 1999. The accuracy of the load forecast is measured by way of two error metrics: the Root Mean Square Error (RMSE) in MWh units and the Mean Absolute Percent Error (MAPE). The RMSE is the standard deviation of the prediction errors and is defined as follows [
40,
41]:
where
is the target value,
is the predicted value, and
N is the total number of values. The MAPE is calculated by taking the average of the unsigned percentage errors as follows [
40,
41]:
The behavior of these metrics during the training sessions for each model is shown in
Figure 3 and
Figure 4. We note that the loss drops with each epoch and stabilizes over time, indicating that the models are sufficiently trained without overfitting.
Results from the standard LSTM model are shown in
Figure 5. We can see that the model is efficient in comparing the prediction mean and the actual values. In fact, the error percentage measured as MAPE is less than 3%, as shown in
Table 3.
The January forecast obtained from the Bayesian LSTM model is shown for 50 timesteps in
Figure 6. We note that the uncertainty band is narrow due to this only being a measure of the model uncertainty and not including other sources of error such as noise.
Figure 7 shows the same prediction using quantile regression with the GBR model. As this method does not differentiate between sources of uncertainty but instead captures all types, we note that the prediction intervals are wider than those for the LSTM model.
Table 3 shows the RMSE and MAPE values for each model for the January 1999 testing set. The simple LSTM model shows high accuracy with a MAPE value of 2.3 percent. Introducing MC dropout to the neural network reduces the accuracy of the LSTM model by a small amount. GBR achieves similar accuracy or slightly better accuracy compared to the LSTM model, and as producing a prediction interval does not require any modifications to the main predictive model itself, there is no loss in performance. Overall, the performance of all three models is fairly similar, showing that uncertainty estimates can be obtained with minimal cost to accuracy.
3.5. Uncertainty Analysis
We perform some analysis for the obtained uncertainty values.
Figure 8 shows the frequency of the calculated uncertainty as a percentage of the actual load values in the prediction of January 1999 loads. We observe that the frequency decreases for higher percentage values of uncertainty and almost no uncertainty estimates rise above 10%. This behavior reflects the relatively high confidence and low error percentage of our forecasting model.
Table 4 shows the correlation of the percentage uncertainty with the input features used in the prediction of January 1999 load, namely, the past half-hourly loads, the average daily temperatures, type of weekdays, and public holidays. From the figure, the uncertainty has a −0.05 correlation with the past half-hourly loads, a 0.03 correlation with average daily temperatures, and zero correlation with the type of weekdays or public holidays. We see that the model uncertainty is generally independent of the features used as it is a result of the model architecture.
4. Conclusions
With the fast development of machine learning algorithms and their extensive application, estimating model uncertainty has become a significant research area. We implemented and evaluated two methods in quantifying the uncertainty for two popular supervised learning algorithms. This work is one of the few that studies uncertainty for both RNN and tree based algorithms in the area of energy forecasting. An uncertainty measure is critical for applications such as designing reliable AI-enabled autonomous energy smart grids [
10]. In addition, uncertainty is also an important metric for energy production and distribution as it allows utility companies to assess the trustworthiness of forecasts before making decisions. Our results demonstrate the effectiveness of the learning techniques, LSTM and GBR, and the uncertainty evaluation methods, using the EUNITE electricity load dataset. Both methods are shown to provide accurate predictions.
There remains much to be investigated in the field of uncertainty estimation and its application to electricity forecasting. Although we only described methods to produce uncertainty estimates for two algorithms used in STELF, these methods can also be extended to other forecasting algorithms. Monte Carlo methods can be used to estimate the degree of uncertainty that can be attributed to the model parameters, by estimating the variance of the posterior. Quantile methods which aim to train separate models to predict upper and lower bounds can provide an overall estimate of the expected distribution. Both techniques provide useful estimates of the risk in real-world scenarios and easy computational implementations for AI-enabled autonomous smart grids.