Next Article in Journal
The Biodiesel of Microalgae as a Solution for Diesel Demand in Iran
Next Article in Special Issue
Short-Term Electricity Demand Forecasting Using a Functional State Space Model
Previous Article in Journal
Energy-Saving Potential of China’s Steel Industry According to Its Development Plan
Previous Article in Special Issue
Identifying Health Status of Wind Turbines by Using Self Organizing Maps and Interpretation-Oriented Post-Processing Tools
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Stacking Ensemble Learning for Short-Term Electricity Consumption Forecasting

1
Division of Computer Science, Universidad Pablo de Olavide, ES-41013 Seville, Spain
2
Faculty of Computer Science, University of Namur, B-5000 Namur, Belgium
*
Author to whom correspondence should be addressed.
Energies 2018, 11(4), 949; https://doi.org/10.3390/en11040949
Submission received: 2 February 2018 / Revised: 4 April 2018 / Accepted: 9 April 2018 / Published: 16 April 2018
(This article belongs to the Special Issue Data Science and Big Data in Energy Forecasting)

Abstract

:
The ability to predict short-term electric energy demand would provide several benefits, both at the economic and environmental level. For example, it would allow for an efficient use of resources in order to face the actual demand, reducing the costs associated to the production as well as the emission of CO 2 . To this aim, in this paper we propose a strategy based on ensemble learning in order to tackle the short-term load forecasting problem. In particular, our approach is based on a stacking ensemble learning scheme, where the predictions produced by three base learning methods are used by a top level method in order to produce final predictions. We tested the proposed scheme on a dataset reporting the energy consumption in Spain over more than nine years. The obtained experimental results show that an approach for short-term electricity consumption forecasting based on ensemble learning can help in combining predictions produced by weaker learning methods in order to obtain superior results. In particular, the system produces a lower error with respect to the existing state-of-the art techniques used on the same dataset. More importantly, this case study has shown that using an ensemble scheme can achieve very accurate predictions, and thus that it is a suitable approach for addressing the short-term load forecasting problem.

1. Introduction

The world energy demand is increasing day by day. As pointed out in [1], it is estimated that the world energy consumption will increase from 549 quadrillion British thermal unit (Btu) in 2012 to 629 quadrillion Btu in 2020. A further 48% increase (to 815 quadrillion Btu) is expected by 2040. More than half of the increase will correspond to Asian countries that do not belong to the Organization for Economic Co-operation and Development (OECD), including China and India.
Several factors are contributing to such growing energy demand, e.g., the rapid grow of the human population and increasing energy required by buildings and technology applications. Therefore, the development of efficient energy management systems and predictive models for forecasting energy consumption are becoming important in decision-making for effective energy saving and development in particular areas, in order to decrease both the costs associated to it and the environmental impact this consumption presents. Governments are also taking actions into these matters. For example, the European Commission is constantly developing measures to increase the EU’s energy-efficiency targets and to make them legally binding. Under the current energy plan, EU countries will have to adopt a set of minimum energy efficiency requirements in order to achieve an increment of at least 20 % in the energy efficiency [2]. Moreover, all EU countries have reached an agreement in order to reach an increment of at least 27 % by 2020, , to be reviewed by 2020 with the potential to raise the target to 30 % by 2030.
Electric energy consumption forecasting algorithms can provide several benefits in this sense. For example, in [3,4] forecasting is used to assess what fraction of the generated power should be stored locally for later use and what fraction of it can instead be fed to the loads or injected into the network. Generally, forecasting can be divided into three categories, depending on the prediction horizon, i.e., the time scale of the predictions. Short-term load forecasting, characterised by prediction horizons going from one hour up to a week, medium-term load forecasting, with prediction from one month up to a year, and long-term load forecasting, for prediction involving a prediction horizon of more than one year [5].
Short-term load forecasting is an important problem. In fact, with reliable and precise prediction of short-term load, schedules can be generated in order to determine the allocation of generation resources, operational limitations, environmental and equipment usage constraints. Knowing the short-term energy demand can also help in ensuring the power system security since accurate load prediction can be used to determine the optimal operational state of power systems. Moreover, the predictions can be helpful in preparing the power systems according to the future predicted load state. Precise predictions also have an economic impact, and may improve the reliability of power systems. The reliability of a power system is affected by abrupt variations of the energy demand. Shortage of power supply can be experienced if the demand is underestimated, while resources may be wasted in producing energy if such energy demand is overestimated. From the above observations, we can understand why short-term load forecasting has gained popularity. The work presented in this paper lies among the short-term load forecasting.
Basically, there are two main approaches to forecasting energy consumption, conventional methods, such as [6,7] and, more recently, methods based on machine learning. Conventional methods, including statistical analysis, smoothing techniques such as the autoregressive integrated moving average (ARIMA) and exponential smoothing and regression-based approaches, can achieve satisfactory results when solving linear problems. Machine learning strategies, in contrast to traditional methods, are also suitable for non-linear cases. Among the machine learning strategies approaches, strategies such as Artificial Neural Networks (ANN) or Support Vector Machines (SVM) have been successfully (and increasingly) exploited to forecast power consumption data, e.g., [8,9,10]. Although machine learning techniques provide effective solutions for time series forecasting, these methods tend to get stuck in a local optimum. For instance, ANN and SVM may get trapped in a local optimum if the configurations parameters are not properly set.
In order to overcome such limitations, in this paper we propose an approach based on ensemble learning [11,12,13], and more specifically, we propose a two-layer ensemble scheme. Ensemble learning is a machine learning paradigm where multiple learners are trained to solve the same problem. In contrast to ordinary machine learning approaches, which try to learn one hypothesis from training data, ensemble methods try to construct a set of hypotheses and combine them. This approach usually yields better results than the use of a single strategy, since it provides better generalizations, i.e., adaptation to unseen cases, better capability of escaping from local optima and superior search capabilities. In this paper, we propose a novel ensemble scheme, which is based on two layers. On the bottom layer, three learning algorithms are used, and their predictions are used by another strategy at the top level.
In order to assess the performances of our proposal, we use a dataset regarding the electricity consumption in Spain registered over a period of more than nine years. We use a fixed prediction horizon of four hours, while we vary the historical window size, i.e., the amount of historical data used in order to make the predictions. Experimental results shows that an ensemble scheme can achieve better results than single methods, obtaining more precise predictions than other state of the art methods. Therefore, we can summarize the contributions of this work as follows:
  • Propose to explore the short-term electrical consumption forecasting by using ensemble learning;
  • Analyse electricity consumption data from Spain by means of the proposed ensemble scheme.
The rest of the paper is organised as follows. In Section 2, we provide a brief overview of the state of the art on prediction of time series, with a special focus on prediction of energy consumption. Section 3 describes the data used in this paper and the proposed strategy. In particular, in Section 3.3 we describe the particular ensemble learning scheme used. Results are discussed in Section 4. Finally in Section 5, we draw the main conclusions and discuss possible future works.

2. Time Series Forecasting

This section provides a basic background on time series. We refer the reader to [14] for a more extensive introduction to time series analysis. Moreover, in Section 2.1 we present an overview of relevant works on time series forecasting.
A time series is a sequence of time-ordered observations measured at equal intervals of time. In a time series consisting of T real value samples x 1 , , x T , x i ( 1 i T ) represents the recorded value at time i. We can then define the problem of time series forecasting as the problem of predicting the values of x w + 1 , , x w + h , given the previous x 1 , , x w ( w + h T ) samples, with the objective of minimizing the error between the predicted value x ^ w + i and the actual value x w + i ( 1 i h ). Here, we refer to w as the historical window, i.e., how many values we consider in order to produce the predictions, and to h as the prediction horizon, which represents how far in the future one aims to predict.
Traditionally, time series are decomposed into the three components [14]:
  • Trend—This term refers to the general tendency exhibited by the time series. A time series can present different types of trends, such as linear, logarithmic, exponential power, polynomial, etc.
  • Seasonality—This is a pattern of changes that represents periodic fluctuations of constant length. This variations are originated by effects that are stable along with time, magnitude and direction.
  • Residual—This component represents the remaining, mostly unexplainable, parts of the time series. It also describes random and irregular influences that, in case of being high enough, can mask the trend and seasonality.
More decomposition patterns can be included in order to represent long-run cycles, e.g., holiday effects. However, real-world time series are challenging to forecast due to meaningful irregular components they incorporate.
An important aspect is also to determine if a time series is stationary. This means to verify whether or not the mean and variance of the time series are constant over time. If a time series is not stationary, some transformation techniques must be applied before one can apply some forecasting methods.
According to the number of variables involved, time series analysis can be divided into univariate and multivariate analysis [15]. In the univariate case, a time series consists of a single observation recorded sequentially. In contrast, in multivariate time series the values of more than one variables are recorder at each time stamp. The interaction among such variables should be taken into account.
There are different techniques that can be applied to the problem of time series forecasting. Such approaches can be roughly divided into two categories, linear and non-linear methods [16]. Linear methods try to model the time series using a linear function. The basic idea is that even if the random component of a time series may prevent one from making any precise predictions, the strong correlation among data allows to assume that the next observation can be determined by a linear combination of the preceding observations, except for additive noise.
Non-linear methods are currently in use in the machine learning domain. These methods try to extract a model, that can be non-linear, which describe the observed data, and then use the so obtained model in order to forecast future values of the time series. Machine learning techniques have gained popularity in the forecasting field, due to the fact that while conventional methods can achieve satisfactory results in linear problems, machine learning methods are suitable also for non-linear modelling [15].

2.1. Related Work

The number of studies addressing the electricity consumption forecasting is increasing due to several reasons, such as gaining knowledge about the demand drivers [17], or comprehending the different energy consumption patterns in order to adopt new policies according to demand response scenarios [18], or, again, measuring the socio-economic and environmental impact of energy production for a more sustainable economy [19].
In the conventional approach, the Auto-Regressive and Moving Average (ARMA) is a very common technique that arises as a mix of the Auto-Regressive (AR) and the Moving Average (MA) models. In [6] Nowicka-Zagrajek and Weron applied the ARMA model to the California power market. In another work, Chujai et al. [20] compared the Auto-Regressive Integrated Moving Average (ARIMA) with ARMA on household electric power consumption. The results showed that the ARIMA model performed better than ARMA at forecasting longer periods of time, while ARMA is better at shorter periods of time. The ARIMA methods were applied in [21] by Mohanad et al. to predict short-term electricity demand in Queensland (Australia) market. ARMA is usually applied on stationary stochastic processes [6] while ARIMA on non-stationary cases [22].
Regression based methods are also popular in energy consumption studies. The use of the simple regression model of the ambient temperature was proposed by Schrock and Claridge [23], where the authors investigated a supermarket’s electricity use. In later studies, however, the use of multiple regression analysis is preferred, due to the capability to handle more complex models. Lam et al. [24] used such an approach to analyse office buildings in different climates in China. In another work, Braun et al. [25] performed multiple regression analysis on gas and electricity usage in order to study how the change in the climate affects the energy consumption in buildings. In a more recent work Mottahedia et al [26] investigated the suitability of the multiple-linear regression to model the effect of building shape on total energy consumption in two different climate regions.
As stated in the previous section, a significant part of recent studies in the literature is focussed on time series forecasting using machine learning techniques. Among these techniques, Artificial Neural Networks (ANN) have been extensively applied. In an early work presented by Nizami and Ai-Garni [27], the authors developed a two-layered fed-forward ANN to analyse the relation between electric energy consumption and weather-related variables. In another work, Kelo and Dudul [28] proposed to use a wavelet Elman neural network to forecast short-term electrical load prediction under the influence of ambient air temperature. In [29] Chitsaz et al. combined the wavelet and ANN for short-term electricity load forecasting in micro-grids. In a more recent work, Zheng et al. [30] developed a hybrid algorithm that combines similar days selection, empirical mode decomposition, and long short-term memory neural networks to construct a prediction model for short-term load forecasting. Other recent examples of using ANN for the problem of energy consumption prediction are [31,32,33].
Despite the popularity of ANN, other novel-techniques are lately gaining attention. For instance, Talavera-Llames et al. [34] adapted a Nearest Neighbours-based strategy to address the energy consumption forecasting problem in a Big Data environment. Torres et al. [35] developed a novel strategy based on Deep Learning to predict times series and tested such strategy on electricity consumption data recorded in Spain from 2007 to 2016. Zheng et al. [36] also presents a Deep Learning approach to deal with forecasting short term electric load time series. Galicia et al. [37] compared Random Forest with Decision Trees, Linear Regression and the gradient-boosted trees on Spanish electricity load data with a ten-minute frequency. Furthermore, Evolutionary Algorithms have been applied to short-term forecasting energy demand by Castelli et al. in [38,39]. Burger and Moura [40] tackled the forecasting of electricity demand by applying an ensemble learning approach that uses Ordinary Least Squares and k-Nearest Neighbors. In [41], Papadopoulos and Karakatsanis explore the ensemble learning approach and compare four dfferent mehtods: seasonal autoregressive moving average (SARIMA), seasonal autoregressive moving average with exogenous variable (SARIMAX), random forests (RF) and gradient boosting regression trees (GBRT). Finally, Li et al. [42] proposed a novel ensemble method for load forecasting based on wavelet transform, extreme learning machine (ELM) and partial least squares regression.
For a more exhaustive review of the state of the art in the field of time series forecasting, we refer the reader to, for example, Martínez-Álvarez et al. [16], where an extensive review of machine learning methods is proposed, while Daut et al. [43] and Deb et al. [15] review conventional and artificial intelligence methods.

3. Materials and Methods

In this section we will provide details about the data and the methods used in this paper.

3.1. Data

The dataset used in this work records the general electricity consumption in Spain (expressed in MW) over a period of 9 years and 6 months, with a 10 min period between each measurement. Thus, what is measured is the electricity consumption taken as a whole, not relative to a specific sector. In total, the dataset is composed by 497.832 measurements, which go from 1 January 2007 at midnight till 21 June 2016 at 11:40 p.m. The dataset is available on request.
Figure 1 shows both the AutoCorrelation Function (ACF) and the Partial AutoCorrelation Function (PACF) for the dataset considered in this paper. Both graphs have a few significant lags but these die out quickly, so we can conclude our series is stationary. In order to support this conclusions, we have run different tests, namely the Ljung-Box, the Augmented Dickey–Fuller (ADF) and the Kwiatkowski-Phillips-Schmidt-Shin (KPSS). All the test have return a very low p-value, confirming the stationarity of the series.
The original dataset has been pre-processed in order to be used, as in [35]. First, the attribute corresponding to consumption has been extracted, and a consumption vector has been obtained. After that, the consumption vector has been redistributed in a matrix depending on a historical window, w, and a prediction horizon, h. The historical window, or data history (w) represents the number of previous entries taken into consideration in order to train a model that will be used to predict the subsequent values (h). This process is detailed in Figure 2.
In this study, the prediction horizon (h) has been set to 24, corresponding to a period of 4 h. Moreover, different values of the data history have been used. In particular, w has been set to the values 24, 48, 72, 96, 120, 144 and 168, corresponding to 4, 8, 12, 16, 20, 24 and 28 h, respectively. The resulting datasets have been divided into 70 % for the training set and 30 % for the test set. Table 1 provides the details of each dataset. Notice that for all the obtained datasets, the last 24 columns represent the values to be predicted, and thus are not considered for training purposes.

3.2. Ensemble Learning

In the last few years, ensemble models are taking more relevance due to the good performance obtained in several tasks like classification or regression problems [44]. These methods consist in combining different learning models in order to improve the results obtained by each individual model.
The earliest works on ensemble learning were carried out in 1990s, e.g., [45,46,47], where it was proven that multiple weak learning algorithms could be converted into a strong learning algorithm. In a nutshell, ensemble learning [48,49] is a procedure where multiple learner modules are applied on a data set to extract multiple predictions. Such predictions are then combined into one composite prediction.
Usually two phases are employed. In a first phase a set of base learners are obtained from training data, while in the second phase the learners obtained in the first phase are combined in order to produce a unified prediction model. Thus, multiple forecasts based on the different base learners are constructed and combined into an enhanced composite model superior to the base individual models. This integration of all good individual models into one improved composite model generally leads to higher accuracy levels.
According to [48] there are three main reasons why ensemble learning is successful in ML. The first reason is statistical. Models can be seen as searching a hypothesis space H to identify the best hypothesis. However, since usually the datasets are limited, we can find many different hypotheses in H which can fit reasonably well, and we cannot establish a priori which one will generalize better, i.e., will perform the best on unseen data. This makes it difficult to choose among the hypotheses. It follows that the use of ensemble methods can help to avoid this issue by using several models to get a good approximation of the unknown true hypothesis.
The second reason is computational. Many models work by performing some form of local search to minimize error functions. These searches can get stuck in local optima. An ensemble constructed by starting the local search from many different points may provide a better approximation to the true unknown function.
The third argument is representational. In many situations, the unknown function we are looking for may not be included in H. However, a combination of several hypotheses drawn from H can enlarge the space of representable functions, which could then also include the unknown true function.
The most used and well-known of the basic ensemble methods are bagging, boosting and stacking.
  • Bagging in this scheme, a number of models are built, the results obtained by these models are considered equally, and a voting mechanism is used in order to settle on the majority result. In case of regression the average predictions is usually the final output.
  • Boosting is similar to bagging, but with one conceptual modification. Instead of assigning equal weighting to models, boosting assigns different weights to classifiers, and derives its ultimate result based on weighted voting. In case of regression a weighted average is usually the final output.
  • Stacking builds its models using different learning algorithms and then a combiner algorithm is trained to make the ultimate predictions using the predictions generated by the base algorithms. This combiner can be any ensemble technique.
In this paper we have used a stacking approach, since we believe it to be the most suitable in case of the regression problem considered in this work. Figure 3 shows a general scheme of such approach. In the following section we will specify which learning algorithms have been used in the scheme we propose. We can define a stacking ensemble scheme more formally in the following way. Given a set of N different learning algorithms L k , k = 1 , , N and the pair < x , y > , with x = ( x 1 , , x w ) representing the w recorded values and y = ( x w + 1 , , x w + h ) the h values to predict. Let m k j , k = 1 , , N , j = 1 , , h be the model induced by the learning algorithm L k on x to predict x w + j , and let f j be the generalizer function responsible for combining the models for predicting such value. f j can be a generic function, such as the average, or a model induced by a learning algorithm. Then, the estimated x ^ w + j value is given by the expression:
x ^ w + j = f j ( m 1 j , , m N j )
Ensemble methods have been successfully applied for solving pattern classification, regression and forecasting in time series problems [50,51]. For example, Adhikari [52] proposed a linear combination method for time series forecasting that determines the combining weights through a novel neural network structure. Bagnal et al. [53] proposed a method using an ensemble of classifiers on different data transformations in order to improve the accuracy of time-series classification. Authors demonstrated that the simple combination of all classifiers in one ensemble obtained better performance than any of its components. Jin and Dong [51] proposed a deep neural network-based ensemble method that integrates filtering views, local views, distorted views, explicit and implicit training, subview prediction, and Simple Average for classification of biomedical data. In particular, they used the Chinese Cardiovascular Disease cardiograms database. Chatterjee et al. [54] developed an ensemble support vector machine algorithm for reliability forecasting of a mining machine. This method is based on least square support vector machine (LS-SVM) with hyper parameters optimized by a Genetic Algorithm (GA). The output of this model was generalized from a combination of multiple SVM predicted results in time series dataset. Additionally, the advantages of ensemble methods for regression from different viewpoints such as strength-correlation or biasvariance was also demonstrate in the literature [55].
Ensemble learning based methods have been also applied in energy time series forecasting context. For example, Zang et al. [56] proposed a method, called extreme learning machine (ELM), which was successfully applied on the Australian National Electricity Market data. Another example was presented by Tan et al. in [57] where the authors proposed a price forecasting method based on wavelet transform combined with ARIMA and GARCH models. The method was applied on Spanish and PJM electricity markets. Fan et al. [58] proposed a ensemble machine learning model based on Bayesian Clustering by Dynamics (BCD) and SVM. The proposed model was trained and tested on the data of the historical load from New York City in order to forecasts the hourly electricity consumption. Tasnim et al. [59] proposed a cluster-based ensemble framework to predict wind power by using an ensemble of regression models on natural clusters within wind data. The method was tested on a large number of wind datasets of locations across spread Australia.
Ensembles of ANNs have been recently applied in the literature with the aim of energy consumption or price forecasting. For instance, the authors in [60] presented a building-level neural network-based ensemble model for day-ahead electricity load forecasting. The method showed that it outperforms the previously established best performing model by up to 50%, in the context of load data from operational commercial and industrial sites. Jovanovic et al. [61] used three artificial neural networks for prediction of heating energy consumption of a university campus. The authors tested the neural networks with different parameter combinations, which, when used in an ensemble scheme, achieved better results.

3.3. Methods

As already stated in Section 3.2, in our proposal we used a stacking ensemble scheme. In particular, we employed a scheme formed by three base learning methods and a top method. The basic learning methods are regression trees based on Evolutionary Algorithms, Artificial Neural Networks and Random Forests. At the top level, we have used the Generalized Boosted Regression Models in order to combine the predictions produced by the bottom level. The employed scheme is graphically shown in Figure 4.
In the following, we provide some basic notions regarding the methods used in the ensemble scheme.
  • Evolutionary Algorithms (EAs) for Regression Trees EAs [62] are population-based strategies that use techniques inspired by evolutionary biology such as inheritance, mutation, selection and crossover. Each individual i of the population represents a candidate solution to a given problem and is assigned a fitness function, which is a measure of the quality of the solution represented by i. Typically EAs start from an initial population consisting of randomly initialised individuals. Each individual is evaluated in order to determine its fitness value. Then a selection mechanism is used in order to select a number of individuals. Usually the selection is based on the fitness, so that fitter individuals have more probabilities of being selected. Selected individuals generate offspring, i.e., new solutions, by means of the application of crossover and mutation operators. This process is repeated over a number of generations or until a good enough solution is found. The idea is that better and better solutions will be found at each generation. Moreover, the use of stochastic operators, such as mutation, allows EAs to escape from local optima. For the problem tackled in this paper, each individual encodes a regression tree. A regression tree is a decision tree similar to a classification tree [63]. Both classification and regression trees aim at modeling a response variable Y by a vector of P predictor variables X = ( X 1 , , X P ) . The different is that for classification trees, Y is qualitative and for regression trees Y is quantitative. In both cases X i can be continuous and/or categorical variables.
    Regression trees are commonly used in regression-type problems, where we attempt to predict the values of a continuous variable from one or more continuous and/or categorical predictor variables. An advantage of using regression trees is that results can be easier to interpret. Other greedy strategies have been used in order to obtained regression trees, for example [64,65]. The main challenge of such strategiesis that the search space is typically huge, rendering full-grid searches computationally infeasible. Due to their search capabilities, EAs have proven that they can overcome this limitation.
    In this paper, we have used the R evtree package (from now on EVTree) [66], with the following parameters:
    • minbucket: 8 (minimum number of observations in each terminal node)
    • minsplit: 100 (minimum number of observations in each internal node)
    • maxdepth: 15 (maximum tree depth)
    • ntrees: 300 (number of tree in the population)
    • niterations: 1000 (maximum number of generations)
    • alpha: 0.25 (complexity part of the cost function)
    • operatorprob: with this parameter, we can specify, in list or vector form, the probabilities for the following variation operators:
      pmutatemajor: 0.2 (Major split rule mutation, selects a random internal node r and changes the split rule, defined by the corresponding split variable v r , and the split point s r [66])
      pmutateminor: 0.2 (Minor split rule mutation is similar to the major split rule mutation operator. However, it does not alter v r and only changes the split point s r by a minor degree, which is defined by four cases describes in [66])
      pcrossover: 0.8 (Crossover probability)
      psplit: 0.2 (Split selects a random terminal-node and assigns a valid, randomly generated, split rule to it. As a consequence, the selected terminal node becomes an internal node r and two new terminal nodes are generated)
      pprune: 0.4 (Prune chooses a random internal node r, where r > 1, which has two terminal nodes as successors and prunes it into a terminal node [66])
  • Artificial Neural Networks (ANNs) ANNs [67] are computational models inspired by the structure and functions of biological neural networks. The basic unit of computation is the neuron, also called node, which receives input from other nodes or from an external source and computes an output. In order to compute such output, the node applies a function f called the Activation Function, which has the purpose of introducing non-linearity into the output. Furthermore, the output is produced only if the inputs are above a certain threshold.
    Basically, an ANN creates a relationship between input and output values and is composed of interconnected nodes grouped in several layers. Among such layers we can distinguish the outer ones, called input and output layers, from the “internal” ones, called hidden layers. In contrast to biological neurons networks, ANNs usually consider only one type of node, in order to simplify the model calculation and analysis.
    The intensity of the connection between nodes is determined by weights, which are modified during the learning process. Therefore, the learning process consists in adapting the connections to the data structure that model the environment and to characterize its relations.
    According to the structure, there are different types of ANN. The suitability of the structure depends on several factors as, for example, the quality and the volume of the input data. The simplest type of ANN is the so called feedforward neural network. In such networks, nodes from adjacent layers are interconnected and each connection has a weight associated to it. The information moves forward from the input to the output layer through the hidden nodes. There is only one node at the output layers, which provides the final results of the network, being it a class label or a numeric value.
    In this paper we have used the nnet package of R [68], a package for feed-forward neural networks with a single hidden layer, and for multinomial log-linear models.
    The following parameters were used in this paper:
    • size: 10 (number of hidden units)
    • skip: true (add skip-layer connections from input to output)
    • MaxNWts: 10,000 (maximum number of weights allowed)
    • maxit: 1000 (maximum number of iterations)
  • Random Forests (RF) The term Random Forest was introduced by Breinman and Cutle in [69], and refers to a set of decision trees which form an ensemble of predictors. Thus, RF is basically an ensemble of decision trees, where each tree is trained separately on a idependent randomly selected training set. It follows that each tree depends on the values of an input dataset sampled independently, with the same distribution for all trees.
    In other words, the trees generated are different since they are obtained from different training sets from a bootstrap subsampling and different random subsets of features to split on at each tree node. Each tree is fully grown, in order to obtain low-bias trees. Moreover, at the same time, the random subsets of features result in low correlation between the individual trees, so the algorithm yields an ensemble that can achieve both, low bias and low variance [70]. For classification, each tree in the RF casts a unit vote for the most popular class at input. The final result of the classifier is determined by a majority vote of the trees. For regression, the final prediction is the average of the predictions from the set of decision trees.
    The method is less computationally expensive than others tree-based classifiers that adopt bagging strategies, since each tree is generated by taking into account only a portion of the input features [71].
    In this paper, we have used the implementation from the randomForest package of R [72], which provides a R interface to the original implementation by Breiman and Cutle. For this study, the algorithm is used with the following parameters:
    • ntree : 100 (number of trees to be built by algorithm).
    • maxnodes: 100 (maximum number of terminal nodes trees in the forest can have).
  • Generalized Boosted Regression Models (GBM) [73,74]. This method iteratively trains a set of decision trees. The current ensemble of trees is used in order to predict the value of each training example. The prediction errors are then estimated, and poor predictions are adjusted, so that in the next iterations the previous mistakes are corrected. Gradient boosting involves three elements:
    • A loss function to be optimised. Such function is problem dependent. For instance, for regression a squared error can be used and for classification we could use logarithmic loss.
    • A weak learner to make predictions. Regression trees are used to this aim, and a greedy strategy is used in order to build such trees. This strategy is based on using a scoring function used each time a split point has to be added to the tree. Other strategies are commonly adopted in order to constrain the trees. For examples one may limit the depth of the tree, the number of splits or the number of nodes.
    • An additive model to add trees to minimise the loss function. This is done in a sequential way, and the trees already contained in the model built so far are not changed. In order to minimise the loss during this phase, a gradient descend procedure is used. The procedure stops when a maximum number of trees has been added to the model or once there is no improvement in the model.
    Overfitting is common in gradient boosting, and usually, some regularisation methods are used in order to reduce it. These methods basically penalise various parts of the algorithm. Usually some mechanisms are used in order to impose constraints on the construction of decision trees, for example limit the depth of the trees, the number of nodes or leafs or the number of observation per split.
    Another mechanism is shrinkage, which is basically weighting the contribution of each tree to the sequential sum of the predictions of the trees. This is done with the aim of slowing down the learning rate of the algorithm. As a consequence the training takes longer, since more trees are added to the model. In this way a trade-off between the learning rate and the number of trees can be reached.
    In this paper we have used the GBM package of R [75] with the following parameters:
    • distribution: Gaussian (function of the distribution to use)
    • n.trees: 3000 (total number of trees, i.e., the number of gradient boosting iteration)
    • interaction.depth: 40 (maximum depth of variable interactions)
    • shrinkage: 0.9 (learning rate)
    • n.minobsinnode: 3 (minimum number of observations in the trees terminal nodes)
All the parameters used in this paper were set after running preliminary experiments on the data.
We have selected the strategies forming our ensemble scheme based on their popularity and good results achieved in similar problems. Moreover, we have selected algorithms that base the predictions on decision trees, and complemented the possible weakness of such methods by using Artificial Neural Networks. In particular, both RF and GBM are based on an ensemble of decision trees, but the set of trees is obtained in a different way, with RF building each tree independently. EAs provide the ability of escaping local optima, thus we believe that these methods complement each other. Moreover, in order to overcome possible representation limitations of decision trees, we have used NNs, which can handle very well non-linear learning and are tolerant to noise. GBM training generally takes longer than RF, since trees are built sequentially. Moreover, decision trees obtained are prone to overfitting, so we have used it on the top layer, where the predictions are based on three columns, i.e., the output of the three base learners.
The final ensemble scheme we proposed is depicted in Figure 5. We can see that the training set is used in order to obtain the predictions of the base level, consisting of RF, NN and EVTree. The so obtained predictions are then used by the top layer (GBM) in order to produce the final predictions for each problem.
Then, according to the notation introduced in Section 3.2, L 1 = EVTree, L 2 = ANN and L 3 = RF, m 1 j , m 2 j and m 3 j are the models induced by EVTree, ANN and RF, respectively, while f j is the model produced by GBM. Thus the final predictions are produced by GBM, which builds the model using the predictions generated by the three bottom layer methods.

4. Results

In this section we provide the results obtained on the dataset described in Section 3.1 and draw the main conclusions. In order to assess the performances of both the ensemble scheme and the base methods, we used five measures commonly used in regression: the mean relative error (MRE), the mean absolute error (MAE), the symmetric mean absolute percentage error (SMAPE), the coefficient of determination R 2 , and the root mean squared error (RMSE), which are defined as [16]:
M R E = 1 n i = 1 n | Y i Y ^ i | Y i
M A E = 1 n i = 1 n | Y i Y ^ i |
s M A P E = 1 n i = 1 n 2 | ˙ Y i Y ^ i | | Y i + Y ^ i |
R 2 = 1 i = 1 n ( Y i Y ^ i ) 2 | Y i + Y ¯ i |
R M S E = 1 n i = 1 ( Y i Y ^ i ) 2
In the above equations, Y ^ i is the predicted value, Y i the real value and Y ¯ i is the mean of the observed data.
Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10 show the results obtained on all the problems (h) for each historical window (w) used by both the algorithms employed in the bottom layer (EVTree, NN and RF) and by the top layer of the ensemble scheme (GBM). The average results obtained are also shown in the bar graphs. The detailed results obtained by the ensemble scheme and by the base methods can be found in Appendix A, and in particular in Table A1, Table A2, Table A3, Table A4 and Table A5, where results are grouped by the size of the historical window used, as indicated by the first row.
The first and main conclusion that we can draw from these graphs is that the best results were obtained when all the predictions of the baseline methods were combined by GBM at the top level of the ensemble scheme. In particular, when using a historical window of 168 measuraments, the average MRE obtained was 1.88 , while the R 2 was 0.97 , the MAE was 513.50 , the RMSE 714.56 and the average sMAPE was 0.02 . In order to assess the significance of the results with respect to the results obtained, we applied a statistical paired two-tailed t-test with confidence level of 1 % . According to this test, all the results are significantly different, a part from the MRE, MAE, RMSE and sMAPE obtained by EVTree and NN when historical windows of 120 , 96 and 48 were used. Moreover, when w was set to 24, all the results obtained by the bottom layer methods were not significantly different, as far as MRE, MAE and sMAPE are concerned. When we consider MAE, RMSE and sMAPE, results obtained by RF and NN are considered equal for a historical window of 168. The same holds when R 2 is considered, moreover, in the case of R 2 , results obtained by these two methods are not considered significantly different for a historical window of 120 as well. Considering again R 2 , results obtained by EVTree and NN are considered equal for historical windows of size 120 and 72. The results produced by the top layer were always significantly better in all the cases but when considering R 2 for a historical window equal to 120. In this case results obteind by RF are not significantly different. Results obtained by RF and NN are not considered different for historical window values of 144 and 120 as far as RMSE is concerned.
Results are summarized in Table 2, where a ranking of the methods is shown, according to the MRE obtained.
In general, we can also notice the degradation of performances of NN when the historical window used is reduced. In fact, for a historical window of 168, NN obtains the best results among the three bottom layer methods, while for smaller historical windows, starting from 120 measurements, the predictions obtained by this method are always worst than those obtained by RF, and are comparable or worst than the predictions produced by EVTree. Similar consideration can be extracted for the other measures.
We can also notice that the predictions are less and less accurate for increasing values of p, meaning that it is easier to predict the very near future demand than the medium-far future demand. In this sense, we can also observe that NN performs really well on the first two problems. In fact, for values of p equals to 1 or 2, in many cases the predictions obtained by NN are superior to those obtained by the top layer. However, as the value of p increments, the results obtained by the top layer are much better than the results achieved by the three bottom methods. Basically the real difference is made when the problems become harder and harder.
Finally, in Figure 11, we present a comparison of the real and predicted values for a subset of the time series when a historical window of 168 was used. For readability reasons, we have selected two subsets of 250 and 1000 readings, respectively shown in Figure 11a,b. We have included the figure regarding 250 in order to provide a more detailed view of the predictions. We can notice that the predictions are very accurate, and that they can describe in a very precise way the original time series.
In order to globally assess the performance of our proposal, we have compared the results achieved by our ensemble scheme with the results obtained by the single components used in our proposal, i.e., Random Forest (RF), Neural Networks (NN) and Evolutionary Decition Trees (EV), and the results obtained by other four state of the art methods: linear regression (LR), ARMA and ARIMA, Deep Learning (DL) and a decision tree algorithm (DT). In particular we have taken into account linear regression, as a reference time series forecasting strategy [76,77]. The well-known stochastic gradient descent method has been used to minimise the mean square error for the training set in order to obtain the model. We have used a decision tree greedy algorithm [78] that performs a recursive binary partitioning of the feature space in order to build a decision tree. This algorithm uses the information gain in order to build the decision trees, and we have used the default parameter as in the package rpart of R [79]. For the conventional methods ARMA and ARIMA, we have used a tool [80] for determining the order of auto-regressive (AR) terms (p), the degree of differencing (d) and the order of moving-average (MA) terms (q). The values obtained are p = 4 , d = 1 and q = 3 . The deep learning-based methodology has been designed using H 2 O framework of R [81]. This framework implements a Feed Forward Neural Network (also called multi-layer perceptron) that can be launched in distributed environments. The network is trained with stochastic gradient descend using back-propagation algorithm. In order to set the parameters for this algorithm, we have used a grid search approach. As a consequence, we have used a hyperbolic tangent function as activation function, the number of hidden layer was set to 3 and the number of neurons to 30. The distribution function was set to Poisson and in order to avoid overfitting, one of two regularization parameter (Lambda) has ben set to 0.001. The other parameter were set as default as in [35].
Table 3 shows the results of such comparison for each value of historical window considered. We can notice that our proposal outperforms all the other methods, obtaining the best results in all the historical window values considered. This is particularly noticeable for smaller values of w. Another conclusion that we can draw from the table is that LR and NN obtains good results, which are comparable, especially on higher values of the historical window w. Among the single methods, RF obtains, in general, good results, especially for smaller values of historical. It can be noticed that RF achieves better results than LR and NN strategies in all cases except for w values 144 and 168, while it outperforms DL and EV in all cases. In general, the classical strategies ARMA and ARIMA do not perform well on this problem. DT does not perform well on this problem either. This is probably due to the greedy strategy used by this algorithm, which may cause it to get stuck in some local optima. The same considerations may be done for GBM, even if the ensemble nature of this algorithm provides an advantage over DT, and so the results obtained are better. In general we can conclude that the results obtained on this problem by the ensemble scheme are satisfactory, as they achieve more accurate predictions for this short-term electricity consumption forecast problem.

5. Conclusions and Future Works

Accurate short-term forecasting regarding the electric energy demand would provide several benefits, both economical and environmental. For instance, predictions can be taken into account in order to reduce the costs of energy productions, decreasing in the same way the impact on the environment. The predictions are made by taking into consideration data regarding the past demand of electricity, i.e., taking into account historical data. In short-term forecasting, the aim is to be able to predict the near future demand.
In this paper, we have approached the electric energy short-term forecasting problem with a methodology based on ensemble learning. Ensemble learning allows to combine the predictions made by different learning mechanisms in order to achieve predictions that are usually more accurate. More specifically, we have used a stacking ensemble learning scheme, where two levels of learning methods are used. The prediction made by the first level methods are passed to a top method which combines them in order to produce the final forecastings. In this paper, we have used three base learning methods, i.e, regression trees based on Evolutionary Computation, Random Forest and Artificial Neural Networks. At the top layer we have used an algorithm based on Gradient Boosting. We have considered different historical windows, i.e., different amount of historical data used in order to obtain a prediction, and we have focused on predicting the electricity demand of the following four hours. We have compared the results obtained by the ensemble method with the results obtained by the single methods and by linear regression and a decision tree algorithm. Predictions obtained by the ensemble scheme were always superior to the result of the other methods. We have also observed that some methods, like NN, are able to make very precise predictions in the very near future, but that results degenerates the further in the future we aim to predict. Moreover, when the size of historical windows used is small, results are significantly improved when the ensemble scheme is employed. This is due to the degradation in performances of single methods that need the support of more historical data in order to achieve acceptable results.
As for future works, we intend to explore other ensemble schemes, using different methods, for example methods based on support vector machines, deep learning [82] or methods based on SP Theory of Intelligence [83]. Moreover, we are planning on using other datasets, both regarding the electric energy consumption, but also other kind of time series, in order to check if our approach can be generalized to other kind of problems.

Acknowledgments

This work was supported by the Spanish Ministry of Economic and Competitiveness and the European Regional Development Fund, under the grant TIN2015-64776-C3-2-R (MINECO/FEDER). Authors are also thankful to the Andalusian Scientific Computer Science Centre (CICA) for allowing us to use their computing infrastructures.

Author Contributions

Federico Divina proposed the concept of this research and has been involved in the whole experimentation phase. Aude Gilson has carried out the analysis using the evolutionary algorithm and contributed to the building and experimentation of the ensemble learning scheme. Francisco A. Goméz Vela has been involved in the experimentation, while José F. Torres has preprocessed the datasets. Miguel García Torres has provided overall guidance and has been involved in the experimentation. All the authors were involved in preparing the manuscript.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A. Detailed Results of the Ensemble Scheme

Table A1. MRE obtained for each historical window considered and each problem. In the table EV stands for EVTree, RF for Random Forest, NN for Neural Network. GBM stand for the Gradient Boost Models, the method used at the top level, and thus represent the final MRE obtained by the ensemble method.
Table A1. MRE obtained for each historical window considered and each problem. In the table EV stands for EVTree, RF for Random Forest, NN for Neural Network. GBM stand for the Gradient Boost Models, the method used at the top level, and thus represent the final MRE obtained by the ensemble method.
w16814412096724824
h EVRFNNGBMEVRFNNGBMEVRFNNGBMEVRFNNGBMEVRFNNGBMEVRFNNGBMEVRFNNGBM
11.150.660.530.570.980.670.550.591.080.580.570.641.010.700.570.631.110.690.590.640.930.730.590.640.940.740.590.64
21.630.960.980.751.580.980.780.821.441.030.880.931.361.050.880.951.301.060.890.971.391.090.910.961.431.150.921.00
31.811.181.100.951.751.211.011.011.831.271.281.211.791.341.271.231.771.321.331.311.861.421.341.311.791.531.351.37
42.101.281.141.092.091.331.191.131.971.431.541.382.081.451.561.412.101.481.631.432.101.601.641.502.091.791.661.65
52.311.461.261.162.111.531.381.272.301.651.871.542.221.701.851.582.291.721.981.632.341.861.981.732.582.112.001.85
62.571.541.401.322.461.631.511.382.341.802.151.692.531.862.131.762.371.902.301.782.792.112.291.952.762.422.342.14
72.711.691.551.433.231.761.641.472.702.022.341.872.732.112.421.942.772.142.511.992.872.402.642.153.172.762.662.38
82.631.831.671.542.801.911.801.562.732.212.652.062.922.352.702.102.992.382.792.223.232.682.942.373.493.112.972.76
92.901.911.841.632.992.001.991.692.982.282.962.212.992.433.002.303.202.503.232.293.372.823.192.503.903.343.362.89
102.922.072.011.753.452.132.121.813.152.433.212.333.332.663.322.373.302.703.522.463.583.053.582.663.753.713.683.07
113.232.212.101.863.212.232.211.883.402.583.432.533.572.813.562.483.272.853.772.493.863.363.852.834.154.004.023.30
123.242.242.201.903.582.282.281.963.472.633.452.533.622.923.722.703.662.994.142.744.303.704.103.044.554.334.333.56
133.262.362.251.983.212.392.362.023.712.753.692.633.723.073.772.793.903.204.442.824.193.924.543.224.794.694.713.84
143.582.502.302.083.292.482.422.103.502.813.862.633.753.204.192.754.123.374.563.004.594.214.923.294.994.955.043.90
153.452.432.372.163.412.562.482.213.592.834.092.664.053.244.432.933.913.415.092.934.794.425.193.385.635.195.424.29
163.542.532.472.373.512.622.572.303.882.914.232.804.243.344.573.064.253.515.163.055.214.655.353.535.405.465.764.24
173.522.582.482.363.892.702.582.363.772.984.402.884.243.404.903.124.243.615.733.205.074.865.873.705.745.826.094.51
183.752.612.552.273.702.692.652.283.782.994.482.864.023.424.843.184.123.675.933.375.375.116.143.776.036.006.454.75
193.552.702.622.413.652.772.722.474.202.994.533.013.973.475.183.214.463.776.223.355.375.296.403.966.286.195.744.91
203.752.782.782.553.992.812.872.563.953.124.663.124.243.575.193.384.503.876.493.545.555.386.544.086.146.456.895.17
214.022.912.892.724.053.012.932.804.103.234.643.284.943.855.133.754.894.096.263.745.495.636.574.186.556.796.845.45
224.022.953.012.634.063.093.082.854.143.344.763.324.794.125.263.884.934.396.553.955.986.037.144.726.857.167.165.63
234.383.053.162.834.143.163.242.954.403.430.853.445.144.335.384.075.014.706.714.035.716.087.244.637.217.627.536.10
244.133.083.302.934.463.213.363.044.213.485.033.484.894.435.394.094.985.007.374.375.686.307.594.787.527.977.866.60
avg3.092.152.081.883.152.222.161.943.192.453.152.383.422.783.552.573.482.934.132.643.983.704.272.954.494.394.393.58
stdev0.840.690.750.670.900.710.780.690.950.801.410.811.151.041.560.971.181.162.051.001.521.712.161.201.912.142.231.65
Table A2. R 2 obtained for each historical window considered and each problem. In the table EV stands for EVTree, RF for Random Forest, NN for Neural Network. GBM stand for the Gradient Boost Models, the method used at the top level, and thus represent the final R 2 obtained by the ensemble method.
Table A2. R 2 obtained for each historical window considered and each problem. In the table EV stands for EVTree, RF for Random Forest, NN for Neural Network. GBM stand for the Gradient Boost Models, the method used at the top level, and thus represent the final R 2 obtained by the ensemble method.
w16814412096724824
h EVRFNNGBMEVRFNNGBMEVRFNNGBMEVRFNNGBMEVRFNNGBMEVRFNNGBMEVRFNNGBM
10.991.001.001.000.991.001.001.000.991.001.001.000.991.001.001.000.991.001.001.000.991.001.001.000.991.001.001.00
20.980.991.001.000.980.991.001.000.990.991.000.990.990.990.990.990.990.990.990.990.990.990.990.990.990.990.990.99
30.980.990.990.990.980.990.990.990.980.990.990.990.980.990.990.990.980.990.990.990.980.980.990.990.980.980.990.98
40.970.990.990.990.970.990.990.990.970.980.980.990.970.980.980.990.970.980.980.980.970.980.980.980.970.970.980.98
50.960.980.990.990.970.980.990.990.960.980.970.980.970.980.970.980.960.980.970.980.960.970.970.980.950.970.970.97
60.960.980.990.990.960.980.980.990.960.980.970.980.960.970.970.980.960.970.960.980.950.960.960.970.950.950.960.96
70.950.980.980.990.930.980.980.990.950.970.960.970.950.970.960.970.950.970.950.970.940.950.950.960.930.940.950.96
80.950.980.980.980.950.980.980.980.950.970.950.970.940.960.950.970.940.960.940.960.930.940.940.960.910.930.940.94
90.940.970.970.980.940.970.970.980.940.960.940.960.940.960.940.960.930.950.940.960.920.940.930.960.890.910.920.93
100.940.970.970.980.920.970.970.980.930.960.930.960.930.950.930.960.930.950.920.960.910.930.920.950.900.900.910.93
110.940.970.970.980.930.970.960.980.930.960.930.960.920.950.920.960.930.940.910.960.900.920.910.940.880.880.900.92
120.930.970.970.980.920.970.960.970.920.950.920.960.920.940.920.950.910.940.900.950.880.910.900.940.860.870.900.91
130.930.970.970.970.940.960.960.970.920.950.920.950.910.940.910.950.900.930.890.950.880.900.890.930.850.860.880.90
140.920.960.970.970.930.960.960.970.920.950.920.960.910.940.900.950.890.930.880.940.860.890.890.930.840.840.870.90
150.930.960.960.970.930.960.960.970.920.950.900.960.900.930.900.950.900.930.870.950.860.880.860.930.800.830.850.88
160.920.960.950.960.910.960.960.960.910.950.900.950.890.930.880.940.880.920.850.940.830.880.850.920.810.820.860.88
170.930.960.960.960.900.950.950.960.910.950.890.950.890.930.880.940.890.920.830.940.840.870.850.910.790.800.830.87
180.910.960.960.970.910.950.950.960.910.950.890.950.900.930.870.940.890.910.820.930.810.860.820.910.770.790.800.86
190.920.960.950.960.910.950.950.960.890.940.880.940.900.930.890.940.880.910.800.920.820.850.810.900.750.770.780.84
200.910.950.940.960.900.950.940.950.900.940.880.940.880.920.870.930.870.910.790.920.790.830.770.890.740.760.760.82
210.890.950.940.950.890.940.940.940.890.930.880.920.830.910.850.910.830.890.790.900.790.820.800.880.720.720.730.80
220.890.940.940.950.890.930.930.930.890.930.870.920.840.900.840.900.830.880.770.900.770.800.710.850.700.690.700.79
230.870.940.930.940.880.930.920.940.870.920.850.920.830.890.830.890.820.870.710.890.780.780.690.850.660.660.670.74
240.880.930.930.940.850.920.920.930.870.920.840.910.840.880.820.890.830.850.690.870.770.770.660.840.630.630.630.69
average0.930.970.970.980.930.960.960.970.930.960.920.960.910.940.910.950.910.940.880.950.880.900.880.930.840.850.870.89
stdev0.030.020.020.020.040.020.020.020.040.020.050.020.050.030.050.030.050.040.090.030.070.070.100.050.110.110.110.08
Table A3. sMAPE obtained for each historical window considered and each problem. In the table EV stands for EVTree, RF for Random Forest, NN for Neural Network. GBM stand for the Gradient Boost Models, the method used at the top level, and thus represent the final sMAPE obtained by the ensemble method.
Table A3. sMAPE obtained for each historical window considered and each problem. In the table EV stands for EVTree, RF for Random Forest, NN for Neural Network. GBM stand for the Gradient Boost Models, the method used at the top level, and thus represent the final sMAPE obtained by the ensemble method.
w16814412096724824
h EVRFNNGBMEVRFNNGBMEVRFNNGBMEVRFNNGBMEVRFNNGBMEVRFNNGBMEVRFNNGBM
10.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.01
20.020.010.010.010.020.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.01
30.020.010.010.010.020.010.010.010.020.010.010.010.020.010.010.010.020.010.010.010.020.010.010.010.020.020.010.01
40.020.010.010.010.020.010.010.010.020.010.020.010.020.010.020.010.020.010.020.010.020.020.020.010.020.020.020.02
50.020.010.010.010.020.020.010.010.020.020.020.020.020.020.020.020.020.020.020.020.020.020.020.020.030.020.020.02
60.030.020.010.010.020.020.020.010.020.020.020.020.030.020.020.020.020.020.020.020.030.020.020.020.030.020.020.02
70.030.020.020.010.030.020.020.010.030.020.020.020.030.020.020.020.030.020.030.020.030.020.030.020.030.030.030.02
80.030.020.020.020.030.020.020.020.030.020.030.020.030.020.030.020.030.020.030.020.030.030.030.020.030.030.030.03
90.030.020.020.020.030.020.020.020.030.020.030.020.030.020.030.020.030.020.030.020.030.030.030.020.040.030.030.03
100.030.020.020.020.030.020.020.020.030.020.030.020.030.030.030.020.030.030.040.020.040.030.030.030.040.040.040.03
110.030.020.020.020.030.020.020.020.030.030.030.030.040.030.040.020.030.030.040.020.040.030.040.030.040.040.040.03
120.030.020.020.020.040.020.020.020.030.030.030.030.040.030.040.030.040.030.040.030.040.040.040.030.040.040.040.04
130.030.020.020.020.030.020.020.020.040.030.040.030.040.030.040.030.040.030.040.030.040.040.050.030.050.050.050.04
140.040.020.020.020.030.020.020.020.030.030.040.030.040.030.040.030.040.030.050.030.050.040.050.030.050.050.050.04
150.030.020.020.020.030.030.020.020.040.030.040.030.040.030.040.030.040.030.050.030.050.040.050.030.060.050.050.04
160.040.030.030.020.040.030.030.020.040.030.040.030.040.030.050.030.040.030.050.030.050.050.050.040.050.050.050.04
170.030.030.030.020.040.030.030.020.040.030.040.030.040.030.050.030.040.040.060.030.050.050.050.040.060.060.060.04
180.040.030.030.020.040.030.030.020.040.030.040.030.040.030.050.030.040.040.060.030.050.050.060.040.060.060.060.05
190.040.030.030.020.040.030.030.020.040.030.050.030.040.030.040.030.040.040.060.030.050.050.060.040.060.060.070.05
200.040.030.030.030.040.030.030.030.040.030.050.030.040.040.050.030.040.040.060.040.050.050.060.040.060.060.070.05
210.040.030.030.030.040.030.030.030.040.030.040.030.050.040.050.040.050.040.060.040.050.060.060.040.060.070.070.05
220.040.030.030.030.040.030.030.030.040.030.050.030.050.040.050.040.050.040.060.040.060.060.070.050.070.070.070.06
230.040.030.030.030.040.030.030.030.040.030.050.030.050.040.050.040.050.050.070.040.060.060.070.050.070.070.070.06
240.040.030.030.030.040.030.030.030.040.030.050.030.050.040.050.040.050.050.070.040.060.060.070.050.070.080.080.07
average0.030.020.020.020.030.020.020.020.030.020.030.020.030.030.040.030.030.030.040.030.040.040.040.030.040.040.040.04
stdev0.010.010.010.010.010.010.010.010.010.010.010.010.010.010.020.010.010.010.020.010.010.020.020.010.020.020.020.02
Table A4. MAE obtained for each historical window considered and each problem. In the table EV stands for EVTree, RF for Random Forest, NN for Neural Network. GBM stand for the Gradient Boost Models, the method used at the top level, and thus represent the final sMAPE obtained by the ensemble method.
Table A4. MAE obtained for each historical window considered and each problem. In the table EV stands for EVTree, RF for Random Forest, NN for Neural Network. GBM stand for the Gradient Boost Models, the method used at the top level, and thus represent the final sMAPE obtained by the ensemble method.
w16814412096724824
h EVRFNNGBMEVRFNNGBMEVRFNNGBMEVRFNNGBMEVRFNNGBMEVRFNNGBMEVRFNNGBM
1315.06183.84141.20154.12269.76186.59148.87159.90293.55191.03157.33174.33276.20194.46157.72172.48304.90197.39159.39175.95257.80202.16160.19176.54258.43203.61161.48175.66
2444.69266.97190.80204.59425.13272.91209.06222.03395.66288.96239.10258.01371.90294.44240.53261.22355.27289.69249.19268.96384.45306.45247.31263.48391.60319.37252.14274.50
3492.27327.51257.04260.41488.19339.33278.43278.55503.03359.57356.13338.13497.62367.68350.54342.75489.06371.16367.66366.09521.11397.33376.64362.84495.70427.91374.66382.52
4580.65358.72299.64299.16575.93373.70327.27312.64546.29405.41436.83385.62578.36411.12434.09393.85577.60413.16441.61399.27582.53451.24461.76420.55584.82500.47468.88461.21
5633.24405.73345.38318.82591.43428.35374.48349.61646.47464.54520.46433.05609.88477.01509.80442.70636.29477.60533.51458.29648.89521.81556.55487.17719.36585.39562.29518.34
6708.86435.43382.18361.47689.03457.39412.02380.15650.78507.12597.92475.11697.23528.56584.98496.39653.91530.68647.32497.37781.15592.72652.18549.64768.48689.36670.26603.58
7749.47467.70413.95392.33889.64492.57447.50404.27752.55570.38651.44526.94758.87595.00678.23543.33768.36597.30735.48559.00800.87679.71744.49601.35882.54777.91750.64668.62
8727.03509.72458.95422.22776.29530.44487.05431.16755.20625.97748.03577.38807.79642.63750.07585.73822.29669.54819.31619.75893.67755.07829.21660.66970.28871.32837.41774.29
9819.41532.20511.86451.38833.20563.57549.43469.61842.12649.88840.47625.19830.74679.61828.03648.79891.82701.85831.44642.54937.45797.36894.48703.201094.66943.37953.01816.23
10810.18568.26553.47485.61952.72596.12575.04504.84879.64694.50905.87659.69941.04733.60919.11670.39917.64754.63999.09694.16990.46873.23984.90746.001051.811044.241037.51864.31
11891.89610.32581.75509.68880.88619.85605.61516.43943.19729.25867.36714.20985.22781.221002.23698.04912.41798.291063.46700.091084.70930.091091.99796.091155.791124.081121.68926.88
12894.93608.21603.37520.96980.53631.34633.63544.41977.36739.77980.95715.08997.67811.461036.33756.641013.89845.351152.37769.421196.851010.001175.20850.541266.591197.121176.201000.95
13896.87644.11619.53543.39883.43658.74655.42555.721019.40772.301052.90744.521036.05849.831103.87778.871084.55895.181240.51786.211169.421101.201264.23899.781332.771303.181299.241067.64
14980.38669.09632.18567.50910.84677.33667.50578.08974.63791.251017.97736.471030.03887.581161.53770.371136.91923.951319.33830.441254.911155.721304.93917.161383.111377.411395.141088.89
15946.78665.24656.31592.82948.26707.41687.23604.83995.99797.771138.44749.471120.29912.031209.66824.111078.36952.711344.57818.241325.751223.901443.44948.721562.891448.231504.661205.16
16973.29694.25790.18642.25967.61723.83704.26632.211077.44816.181190.03784.791145.19930.931304.47852.281176.16979.541505.64852.821420.221277.151533.88982.401500.131516.811478.191190.47
17966.73697.08691.12640.491068.00740.52713.60646.131040.81835.821238.88803.801162.00953.571333.90867.071185.671010.281578.23893.401379.961299.291530.591025.861575.811587.221614.371257.03
181026.92707.66707.79617.421021.08734.31717.33620.931054.04826.921236.60793.851119.67950.621386.93883.761135.041018.891651.35939.361453.181394.121661.931038.761660.721621.781751.411325.89
19974.38723.85727.31649.871009.15752.29759.06676.861153.30845.251278.66827.001094.84957.741233.45882.071228.421027.761713.54929.811451.421412.651702.211082.611701.791689.891824.471357.30
201016.35738.07825.27686.701091.03756.30776.08696.341077.74843.781263.89854.551159.40972.471339.16934.351208.891034.461677.34973.781486.861452.521773.521112.471650.191722.461858.161410.17
211084.94784.41780.49737.431100.86813.31806.97756.251130.04891.351216.28898.951361.371035.031418.941037.131338.841110.021625.711031.831474.941499.521615.871142.601757.341817.201877.551486.30
221088.14800.66812.81710.881112.60833.29837.48771.221133.53903.251255.31912.971302.741105.471437.691069.701357.541207.391697.421089.461607.871549.451900.731283.691839.991925.601940.771531.37
231184.70815.42852.09761.961122.55851.76858.68796.521197.91921.171365.60941.511388.611159.521471.601111.571367.941245.601928.571110.241533.301611.611976.611260.101938.542006.352036.471670.22
241128.05841.73882.38792.661216.07866.71910.49818.011149.93943.401386.94953.711346.511203.821490.431116.211352.311337.191978.071198.211524.721641.112053.841303.852032.202119.412148.371808.10
average847.30585.67571.54513.50866.84608.67589.27530.28882.94683.95914.31661.85942.47768.14974.30714.16958.09807.901135.84733.531090.101005.641164.03817.341232.311200.821212.29994.40
stdev226.72182.05217.91181.09244.39188.85211.57184.26258.51212.99372.27220.79312.53278.34420.29266.36321.24311.53551.56274.41402.84443.22570.18324.38512.40564.23605.33450.33
Table A5. RMSE obtained for each historical window considered and each problem. In the table EV stands for EVTree, RF for Random Forest, NN for Neural Network. GBM stand for the Gradient Boost Models, the method used at the top level, and thus represent the final sMAPE obtained by the ensemble method.
Table A5. RMSE obtained for each historical window considered and each problem. In the table EV stands for EVTree, RF for Random Forest, NN for Neural Network. GBM stand for the Gradient Boost Models, the method used at the top level, and thus represent the final sMAPE obtained by the ensemble method.
w16814412096724824
h EVRFNNGBMEVRFNNGBMEVRFNNGBMEVRFNNGBMEVRFNNGBMEVRFNNGBMEVRFNNGBM
1403.05240.06180.01195.26340.41241.79190.81203.65374.04249.38201.59222.75350.47256.89201.74220.97387.66259.57204.07224.33324.29268.58205.41224.69326.92268.82206.69224.45
2570.43344.83245.47264.11552.53352.93271.37286.43496.22372.30309.00367.03474.68381.78311.36360.10450.67376.86323.04348.90494.99404.74320.03348.44499.66418.64324.50353.70
3645.60444.80347.52349.54629.05464.02382.62377.39660.13494.19477.56459.06647.98512.66470.66464.50644.11518.45495.65503.78687.24551.33513.93483.85653.13588.23508.01539.42
4758.17489.79411.10403.01769.84508.63463.12444.81729.35558.13591.19531.76781.90573.95591.02535.21774.86586.52606.89544.60770.35632.08636.53576.53795.75699.18645.11645.90
5847.81549.89477.37430.77776.48578.44542.78472.12870.09627.79712.21598.05828.54666.25703.98620.70846.34671.81739.76638.31888.91740.57775.82674.54980.61828.64778.73735.76
6924.32598.55541.68502.21920.27630.40608.10511.67882.87695.24828.50666.16949.97749.28815.61699.63877.78760.32890.08694.091050.03858.18911.27764.111049.12989.98922.26854.51
71007.64637.70586.40533.551227.41677.53660.12553.991016.41774.09913.02744.111029.67828.63947.91764.621055.54842.271015.23786.671119.70974.201029.19848.701216.511114.181030.94955.43
8968.98684.48646.39561.791034.58719.48712.52584.791026.90827.731030.04794.831107.53877.281046.38818.151149.43920.491121.49865.991221.531075.731129.39912.251339.711239.581133.131094.88
91097.95728.63750.24614.091136.86772.11818.41635.591125.66876.271159.09870.291150.87947.791157.99914.341212.04984.681157.62897.491280.931130.341220.43973.441510.111347.181291.911174.28
101117.05767.29789.16679.711275.63812.86863.84686.981190.81928.091227.45897.151268.501013.421258.38926.301240.671053.771347.09955.951387.061223.431324.011026.041467.221482.081377.531216.61
111154.29812.06824.40675.661205.01837.45894.95693.921239.51973.131195.60986.631318.391056.691345.94956.721265.691101.421397.34958.751509.761293.601421.721097.221640.481587.601455.151320.04
121224.45816.16850.80696.541359.76856.17926.20751.541348.851000.591326.27973.411335.621110.461359.951050.501399.641167.861495.721059.411655.091383.651499.211171.551731.251667.971502.991400.90
131214.64858.52865.70750.601195.35893.13952.12760.301345.111046.961369.201010.101400.161145.881430.371059.681481.961224.141579.361060.251622.801491.071579.581227.311843.961785.341611.931457.79
141361.89893.76883.53810.851249.99921.21946.59810.731324.061057.991345.66990.181384.871189.511498.751043.231593.561252.221661.321153.611738.171558.041600.941256.091884.661871.801707.611518.87
151293.15906.16925.68833.341277.38974.27985.54854.011341.751075.761470.711005.831525.861231.281539.371106.561485.431300.731725.741112.571797.091634.501774.401272.772131.741960.211830.891645.12
161326.64940.901064.73950.521393.31995.601007.61900.861435.811086.051514.001069.041605.611246.041650.601128.691632.781337.411874.661162.351956.051683.761871.561322.232082.902041.051819.211649.38
171300.90946.44962.58927.541484.791014.451019.28932.401410.071109.941573.611068.881603.191279.161669.351176.161612.511374.851946.831206.971925.751705.301869.411442.042164.982121.911961.901693.13
181407.34967.12985.02868.391431.291022.811023.39889.161394.621099.831570.921056.101474.101274.541731.621201.321574.251384.522028.611267.562068.551799.982018.371457.122268.872152.052102.831803.97
191304.73993.921018.15917.361408.021053.061089.48970.301563.211135.271613.111133.371505.021280.031577.371196.011659.721401.422097.581293.511984.521828.492064.941482.202326.912238.312195.141861.62
201375.071011.701122.04964.371475.621064.621097.931004.951431.091134.201611.171159.841586.671295.381678.081255.851684.511395.482086.251328.332096.411877.302194.601558.712338.572278.372272.221951.11
211490.471055.951085.261002.911493.001126.731129.521077.961504.161199.461561.241245.301874.161365.601770.821388.731840.221472.702043.771419.072070.401937.392027.481572.932395.712374.732337.822037.80
221510.811082.001132.06996.701507.351156.411186.801166.151519.431224.901597.021253.531820.971442.391804.071439.541835.181577.252145.141454.302161.512005.992399.641765.562477.742488.622445.002072.15
231642.401106.391185.061090.471578.631196.241232.771114.491602.671253.781738.571274.361823.461501.161858.591460.091898.161612.452402.931503.512102.392102.842501.011708.492625.272599.102567.152275.37
241545.301143.791213.591130.071702.801220.671274.421203.261605.221279.141791.281342.221802.031562.131882.761484.211810.281711.112465.831624.302126.312151.322593.831791.812716.302716.842705.662480.75
average1145.55792.54795.58714.561184.39837.13845.01745.311184.92920.011197.00905.001277.091032.841262.61969.661308.871095.351452.171002.691501.661346.351478.451123.271686.171619.181530.601373.46
stdev324.05248.77303.12266.00352.66269.50303.06278.77354.34287.27464.15300.87431.31358.21513.00351.15450.01402.20664.60372.10569.11559.10686.62448.52702.20719.97726.11611.06

References

  1. U.S. Energy Information Administration—International Energy Outlook. 2016. Available online: https://www.eia.gov/outlooks/ieo/index.php (accessed on 22 May 2016).
  2. Energy 2020, A Strategy for Competitive, Sustainable and Secure Energy. Available online: http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52010DC0639&from=EN (accessed on January 2018).
  3. Narayanaswamy, B.; Jayram, T.S.; Yoong, V.N. Hedging strategies for renewable resource integration and uncertainty management in the smart grid. In Proceedings of the 3rd IEEE PES International Conference and Exhibition on Innovative Smart Grid Technologies Europe, Berlin, Germany, 14–17 October 2012; pp. 1–8. [Google Scholar]
  4. Haque, R.; Jamal, T.; Maruf, M.N.I.; Ferdous, S.; Priya, S.F.H. Smart management of PHEV and renewable energy sources for grid peak demand energy supply. In Proceedings of the 2015 International Conference on Electrical Engineering and Information Communication Technology (ICEEICT), Dhaka, Bangladesh, 21–23 May 2015; pp. 1–6. [Google Scholar]
  5. Raza, M.Q.; Khosravi, A. A review on artificial intelligence based load demand forecasting techniques for smart grid and buildings. Renew. Sustain. Energy Rev. 2015, 50, 1352–1372. [Google Scholar] [CrossRef]
  6. Nowicka-Zagrajek, J.; Weron, R. Modeling electricity loads in California: ARMA models with hyperbolic noise. Signal Process. 2002, 82, 1903–1915. [Google Scholar] [CrossRef]
  7. Huang, S.; Shih, K. Short-term load forecasting via ARMA model identification including non-Gaussian process considerations. IEEE Trans. Power Syst. 2003, 18, 673–679. [Google Scholar] [CrossRef]
  8. Bonetto, R.; Rossi, M. Machine Learning Approaches to Energy Consumption Forecasting in Households. arXiv, 2017; arXiv:1706.09648. [Google Scholar]
  9. Gajowniczek, K.; Ząbkowski, T. Short Term Electricity Forecasting Using Individual Smart Meter Data. Procedia Comput. Sci. 2014, 35, 589–597. [Google Scholar] [CrossRef]
  10. Min, Z.; Qingle, P. Very Short-Term Load Forecasting Based on Neural Network and Rough Set. In Proceedings of the 2010 International Conference on Intelligent Computation Technology and Automation, International Conference on (ICICTA), Changsha, China, 11–12 May 2010; Volume 3, pp. 1132–1135. [Google Scholar]
  11. Sewell, M. Ensemble Learning; Technical Report; University College London: London, UK, 2008. [Google Scholar]
  12. Zhou, Z.H. Ensemble Methods: Foundations and Algorithms, 1st ed.; Chapman & Hall CRC: Boca Raton, FL, USA, 2012. [Google Scholar]
  13. Zhang, C.; Ma, Y. Ensemble Machine Learning: Methods and Applications; Springer Science & Business Media: Boston, MA, USA, 2012. [Google Scholar]
  14. Brockwell, P.J.; Davis, R.A. Introduction to Time Series and Forecasting; Springer: Heidelberg, Germany, 2002. [Google Scholar]
  15. Deb, C.; Zhang, F.; Yang, J.; Lee, S.E.; Shah, K.W. A review on time series forecasting techniques for building energy consumption. Renew. Sustain. Energy Rev. 2017, 74, 902–924. [Google Scholar] [CrossRef]
  16. Martínez-Álvarez, F.; Troncoso, A.; Asencio-Cortés, G.; Riquelme, J.C. A Survey on Data Mining Techniques Applied to Electricity-Related Time Series Forecasting. Energies 2015, 8, 13162–13193. [Google Scholar] [CrossRef]
  17. Ranjan, M.; Jain, V.K. Modelling of electrical energy consumption in Delhi. Energy 1999, 24, 351–361. [Google Scholar] [CrossRef]
  18. Campillo, J.; Wallin, F.; Torstensson, D.; Vassileva, I. Energy Demand Model Design for Forecasting Electricity Consumption and Simulating Demand Response Scenarios in Sweden. In Proceedings of the International Conference on Applied Energy, Suzhou, China, 5–8 July 2012. [Google Scholar]
  19. Medina, A.; Cámara, A.; Monrobel, J.R. Measuring the Socioeconomic and Environmental Effects of Energy Efficiency Investments for a More Sustainable Spanish Economy. Sustainability 2016, 8, 1039. [Google Scholar] [CrossRef]
  20. Chujai, P.; Kerdprasop, N.; Kerdprasop, K. Time Series Analysis of Household Electric Consumption with ARIMA and ARMA Models. In Proceedings of the International Multi Conference of Engineers and Computer Scientists, Hong Kong, China, 13–15 March 2013; Volume 1. [Google Scholar]
  21. Al-Musaylhab, M.S.; Deo, R.C.; Adamowskic, J.F.; Li, Y. Short-term electricity demand forecasting with MARS, SVR and ARIMA models using aggregated demand data in Queensland, Australia. Adv. Eng. Inform. 2018, 35, 1–16. [Google Scholar] [CrossRef]
  22. Zhang, G.P. Time series forecasting using a hybrid ARIMA and neural network model. Neurocomputing 2003, 50, 159–175. [Google Scholar] [CrossRef]
  23. Schrock, D.W.; Claridge, D.E. Predicting Energy Usage in a Supermarket. In Proceedings of the 6th Symposium on Improving Building Systems in Hot and Humid Climates, Dallas, TX, USA, 3–4 October 1989. [Google Scholar]
  24. Lam, J.C.; Wan, K.K.W.; Liu, D.; Tsang, C.L. Multiple regression models for energy use in air-conditioned office buildings in different climates. Energy Convers. Manag. 2010, 51, 2692–2697. [Google Scholar] [CrossRef]
  25. Braun, M.R.; Altan, H.; Beck, S.B.M. Using regression analysis to predict the future energy consumption of a supermarket in the UK. Appl. Energy 2014, 130, 305–313. [Google Scholar] [CrossRef]
  26. Mottahedia, M.; Mohammadpour, A.; Amirib, S.S.; Rileyb, D.; Asadib, S. Multi-linear Regression Models to Predict the Annual Energy Consumption of an Office Building with Different Shapes. Procedia Eng. 2015, 118, 622–629. [Google Scholar] [CrossRef]
  27. Nizami, J.; Ai-Garni, A.Z. Forecasting electric energy consumption using neural networks. Energy Policy 1995, 23, 1097–1104. [Google Scholar] [CrossRef]
  28. Kelo, S.; Dudul, S. A wavelet Elman neural network for short-term electrical load prediction under the influence of temperature. Int. J. Electr. Power Energy Syst. 2012, 43, 1063–1071. [Google Scholar] [CrossRef]
  29. Chitsaz, H.; Shaker, H.; Zareipour, H.; Wood, D.; Amjady, N. Short-term electricity load forecasting of buildings in microgrids. Energy Build 2015, 99, 50–60. [Google Scholar] [CrossRef]
  30. Zheng, H.; Yuan, J.; Chen, L. Short-Term Load Forecasting Using EMD-LSTM Neural Networks with a Xgboost Algorithm for Feature Importance Evaluation. Energies 2017, 10, 1168. [Google Scholar] [CrossRef]
  31. Ruiz, L.G.B.; Cuéllar, M.P.; Calvo-Flores, M.D.; Jiménez, M.D.C.P. An Application of Non-Linear Autoregressive Neural Networks to Predict Energy Consumption in Public Buildings. Energies 2016, 9, 684. [Google Scholar] [CrossRef]
  32. Pîrjan, A.; Oprea, S.V.; Căruțașu, G.; Petroșanu, D.M.; Bâra, A.; Coculescu, C. Devising Hourly Forecasting Solutions Regarding Electricity Consumption in the Case of Commercial Center Type Consumers. Energies 2017, 10, 1727. [Google Scholar] [CrossRef]
  33. Buitrago, J.; Asfour, S. Short-Term Forecasting of Electric Loads Using Nonlinear Autoregressive Artificial Neural Networks with Exogenous Vector Inputs. Energies 2017, 10, 40. [Google Scholar] [CrossRef]
  34. Talavera-Llames, R.L.; Pérez-Chacón, R.; Martínez-Ballesteros, M.; Troncoso, A.; Martínez-Álvarez, F. A Nearest Neighbours-Based Algorithm for Big Time Series Data Forecasting. In Proceedings of the 11th International Conference on Hybrid Artificial Intelligent Systems, Seville, Spain, 18–20 April 2016; Springer International Publishing: Cham, Switzerland, 2016; pp. 174–185. [Google Scholar]
  35. Torres, J.F.; Fernández, A.M.; Troncoso, A.; Martínez-Álvarez, F. Deep Learning-Based Approach for Time Series Forecasting with Application to Electricity Load. In Biomedical Applications Based on Natural and Artificial Computing—International Work, Proceedings of the Conference on the Interplay between Natural and Artificial Computation, IWINAC 2017, Corunna, Spain, 19–23 June 2017; Ferrández Vicente, J.M., Álvarez-Sánchez, J.R., de la Paz López, F., Toledo Moreo, J., Adeli, H., Eds.; Springer International Publishing: Cham, Switzerland, 2017; pp. 203–212. [Google Scholar]
  36. Zheng, J.; Xu, C.; Zhang, Z.; Li, X. Electric load forecasting in smart grids using Long-Short-Term-Memory based Recurrent Neural Network. In Proceedings of the 51st Annual Conference on Information Sciences and Systems (CISS), Baltimore, MD, USA, 22–24 March 2017. [Google Scholar]
  37. Galicia, A.; Torres, J.F.; Martínez-Álvarez, F.; Troncoso, A. Scalable Forecasting Techniques Applied to Big Electricity Time Series. In Proceedings of the 14th International Work-Conference on Artificial Neural Networks, Cadiz, Spain, 14–16 June 2017; Springer International Publishing: Cham, Switzerland, 2017; pp. 165–175. [Google Scholar]
  38. Castelli, M.; Vanneschi, L.; De Felice, M. Forecasting short-term electricity consumption using a semantics-based genetic programming framework: The South Italy case. Energy Econ. 2015, 47, 37–41. [Google Scholar] [CrossRef]
  39. Castelli, M.; Trujillo, L.; Vanneschi, L. Energy Consumption Forecasting Using Semantic-based Genetic Programming with Local Search Optimizer. Intell. Neurosci. 2015, 2015, 57. [Google Scholar] [CrossRef] [PubMed]
  40. Burger, E.M.; Moura, S.J. Gated ensemble learning method for demand-side electricity load forecasting. Energy Build. 2016, 109, 23–34. [Google Scholar] [CrossRef]
  41. Papadopoulos, S.; Karakatsanis, I. Short-term electricity load forecasting using time series and ensemble learning methods. In Proceedings of the 2015 IEEE Power and Energy Conference at Illinois (PECI), Champaign, IL, USA, 20–21 February 2015; pp. 1–6. [Google Scholar]
  42. Li, S.; Goel, L.; Wang, P. An ensemble approach for short-term load forecasting by extreme learning machine. Appl. Energy 2016, 170, 22–29. [Google Scholar] [CrossRef]
  43. Daut, M.A.M.; Hassan, M.Y.; Abdullah, H.; Rahman, H.A.; Abdullah, M.P.; Hussin, F. Building electrical energy consumption forecasting analysis using conventional and artificial intelligence methods: A review. Renew. Sustain. Energy Rev. 2017, 70, 1108–1118. [Google Scholar] [CrossRef]
  44. Ren, Y.; Zhang, L.; Suganthan, P.N. Ensemble classification and regression-recent developments, applications and future directions. IEEE Comput. Intell. Mag. 2016, 11, 41–53. [Google Scholar] [CrossRef]
  45. Hansen, L.K.; Salamon, P. Neural network ensembles. IEEE Trans. Pattern Anal. Mach. Intell. 1990, 12, 993–1001. [Google Scholar] [CrossRef]
  46. Ginzburg, I.; Horn, D. Combined neural networks for time series analysis. In Advances in Neural Information Processing Systems; MIT Press Ltd.: Cambridge, MA, USA, 1994; pp. 224–231. [Google Scholar]
  47. Perrone, M.P.; Cooper, L.N. When networks disagree: Ensemble methods for hybrid neural networks. In How We Learn; How We Remember: Toward an Understanding of Brain and Neural Systems: Selected Papers of Leon N Cooper; World Scientific: Singapore, 1995; pp. 342–358. [Google Scholar]
  48. Dietterich, T.G. Ensemble Methods in Machine Learning. In International Workshop on Multiple Classifier Systems; Springer: London, UK, 2000; pp. 1–15. [Google Scholar]
  49. Mendes-Moreira, J.; Soares, C.; Jorge, A.M.; Sousa, J.F.D. Ensemble Approaches for Regression: A Survey. ACM Comput. Surv. 2012, 45, 10:1–10:40. [Google Scholar] [CrossRef]
  50. Qiu, X.; Zhang, L.; Ren, Y.; Suganthan, P.N.; Amaratunga, G. Ensemble deep learning for regression and time series forecasting. In Proceedings of the 2014 IEEE Symposium on Computational Intelligence in Ensemble Learning (CIEL), Orlando, FL, USA, 9–12 December 2014; pp. 1–6. [Google Scholar]
  51. Jin, L.; Dong, J. Ensemble deep learning for biomedical time series classification. Comput. Intell. Neurosci. 2016, 2016. [Google Scholar] [CrossRef] [PubMed]
  52. Adhikari, R. A neural network based linear ensemble framework for time series forecasting. Neurocomputing 2015, 157, 231–242. [Google Scholar] [CrossRef]
  53. Bagnall, A.; Lines, J.; Hills, J.; Bostrom, A. Time-series classification with COTE: The collective of transformation-based ensembles. IEEE Trans. Knowl. Data Eng. 2015, 27, 2522–2535. [Google Scholar] [CrossRef]
  54. Chatterjee, S.; Dash, A.; Bandopadhyay, S. Ensemble support vector machine algorithm for reliability estimation of a mining machine. Qual. Reliab. Eng. Int. 2015, 31, 1503–1516. [Google Scholar] [CrossRef]
  55. Zhou, Z.H. Ensemble learning. In Encyclopedia of Biometrics; Springer: Boston, MA, USA, 2015; pp. 411–416. [Google Scholar]
  56. Zhang, R.; Dong, Z.Y.; Xu, Y.; Meng, K.; Wong, K.P. Short-term load forecasting of Australian National Electricity Market by an ensemble model of extreme learning machine. IET Proc. Gener. Transm. Distrib. 2013, 7, 391–397. [Google Scholar] [CrossRef]
  57. Tan, Z.; Zhang, J.; Wang, J.; Xu, J. Day-ahead electricity price forecasting using wavelet transform combined with ARIMA and GARCH models. Appl. Energy 2010, 87, 3606–3610. [Google Scholar] [CrossRef]
  58. Fan, S.; Mao, C.; Zhang, J.; Chen, L. Forecasting electricity demand by hybrid machine learning model. In Neural Information Processing; Springer: Berlin/Heidelberg, Germany, 2006; pp. 952–963. [Google Scholar]
  59. Tasnim, S.; Rahman, A.; Than Oo, A.M.; Haque, M.E. Wind power prediction using cluster based ensemble regression. Int. J. Comput. Intell. Appl. 2017, 16. [Google Scholar] [CrossRef]
  60. Jetcheva, J.G.; Majidpour, M.; Chen, W.P. Neural network model ensembles for building-level electricity load forecasts. Energy Build. 2014, 84, 214–223. [Google Scholar] [CrossRef]
  61. Jovanović, R.Ž.; Sretenović, A.A.; Živković, B.D. Ensemble of various neural networks for prediction of heating energy consumption. Energy Build. 2015, 94, 189–199. [Google Scholar] [CrossRef]
  62. Eiben, A.E.; Smith, J.E. Introduction to Evolutionary Computing; Springer: Berlin, Germany, 2003. [Google Scholar]
  63. Loh, W.Y. Classification and regression trees. Wiley Interdisciplin. Rev. Data Min. Knowl. Discov. 2011, 1, 14–23. [Google Scholar] [CrossRef]
  64. Breiman, L.; Friedman, J.; Olshen, R.; Stone, C. Classification and Regression Trees; Wadsworth and Brooks: Monterey, CA, USA, 1984. [Google Scholar]
  65. Quinlan, J.R. C4.5: Programs for Machine Learning; Morgan Kaufmann Publishers Inc.: San Francisco, CA, USA, 1993. [Google Scholar]
  66. Grubinger, T.; Zeileis, A.; Pfeiffer, K.P. Evtree: Evolutionary Learning of Globally Optimal Classification and Regression Trees in R. J. Stat. Softw. 2014, 61, 1–29. [Google Scholar] [CrossRef]
  67. Uhrig, R.E. Introduction to artificial neural networks. In Proceedings of the 1995 IEEE IECON 21st International Conference on Industrial Electronics, Control, and Instrumentation, FL, USA, USA, 6–10 November 1995; Volume 1, pp. 33–37. [Google Scholar]
  68. Venables, W.N.; Ripley, B.D. Modern Applied Statistics with S, 4th ed.; Springer: New York, NY, USA, 2002. [Google Scholar]
  69. Breiman, L. Random Forests. Mach. Learn. 2001, 45, 5–32. [Google Scholar] [CrossRef]
  70. Díaz-Uriarte, R.; Alvarez de Andrés, S. Gene selection and classification of microarray data using random forest. BMC Bioinform. 2006, 7, 3. [Google Scholar] [CrossRef] [PubMed]
  71. Gislason, P.O.; Benediktsson, J.A.; Sveinsson, J.R. Random forests for land cover classification. Pattern Recognit. Lett. 2006, 27, 294–300. [Google Scholar] [CrossRef]
  72. Liaw, A.; Wiener, M. Classification and Regression by randomForest. R News 2002, 2, 18–22. [Google Scholar]
  73. Friedman, J.H. Greedy Function Approximation: A Gradient Boosting Machine. Ann. Stat. 2000, 29, 1189–1232. [Google Scholar] [CrossRef]
  74. Friedman, J.H. Stochastic gradient boosting. Comput. Stat. Data Anal. 2002, 38, 367–378. [Google Scholar] [CrossRef]
  75. Ridgeway, G. Generalized boosted models: A guide to the gbm package. Update 2005, 1, 1–12. [Google Scholar]
  76. Hamilton, J.D. Time Series Analysis; Princeton University Press: Princeton, NJ, USA, 1994; Volume 2. [Google Scholar]
  77. Hurvich, C.M.; Tsai, C.L. Regression and time series model selection in small samples. Biometrika 1989, 76, 297–307. [Google Scholar] [CrossRef]
  78. Rokach, L.; Maimon, O. Top-down Induction of Decision Trees Classifiers—A Survey. Trans. Syst. Man Cybern. Part C 2005, 35, 476–487. [Google Scholar] [CrossRef]
  79. Therneau, T.M.; Atkinson, B.; Ripley, B. Rpart: Recursive Partitioning; Technical Report; Department of Health Science Research, Mayo Clinic: Rochester, MN, USA, 1997. [Google Scholar]
  80. Salles, R.; Assis, L.; Guedes, G.; Bezerra, E.; Porto, F.; Ogasawara, E. A Framework for Benchmarking Machine Learning Methods Using Linear Models for Univariate Time Series Prediction. In Proceedings of the 2017 International Joint Conference on Neural Networks (IJCNN), Anchorage, AK, USA, 14–19 May 2017. [Google Scholar]
  81. The H2O.ai Team. H2O: R Interface for H2O, R package version 3.1.0.99999. Available online: https://github.com/h2oai/h2o-3 (accessed on 11 April 2018).
  82. Goodfellow, I.; Bengio, Y.; Courville, A.; Bengio, Y. Deep Learning; MIT Press: Cambridge, MA, USA, 2016; Volume 1. [Google Scholar]
  83. Wolff, J.G. The SP Theory of Intelligence: Benefits and Applications. Information 2014, 5, 1–27. [Google Scholar] [CrossRef]
Figure 1. Correlation plots for the original time series. (a) AutoCorrelation Function (ACF); (b) Partial AutoCorrelation Function (PACF).
Figure 1. Correlation plots for the original time series. (a) AutoCorrelation Function (ACF); (b) Partial AutoCorrelation Function (PACF).
Energies 11 00949 g001
Figure 2. Dataset pre-processing. w determines the amount of the historical data used, while h determines the prediction horizon.
Figure 2. Dataset pre-processing. w determines the amount of the historical data used, while h determines the prediction horizon.
Energies 11 00949 g002
Figure 3. An example scheme of stacking ensemble learning.
Figure 3. An example scheme of stacking ensemble learning.
Energies 11 00949 g003
Figure 4. A graphical representation of the ensemble scheme used in this paper. NN: Artificial Neural Network; RF: Random Forests; GBM: Generalized Boosted Regression Models.
Figure 4. A graphical representation of the ensemble scheme used in this paper. NN: Artificial Neural Network; RF: Random Forests; GBM: Generalized Boosted Regression Models.
Energies 11 00949 g004
Figure 5. A scheme of the ensemble learning strategy used in this paper. w determines the size of the historical window used, while h determines the prediction horizon.
Figure 5. A scheme of the ensemble learning strategy used in this paper. w determines the size of the historical window used, while h determines the prediction horizon.
Energies 11 00949 g005
Figure 6. Comparison of the MRE obtained by the base algorithms and the ensemble scheme on each subproblem for each value of w used. (a) w = 168; (b) w = 144; (c) w = 120; (d) w= = 96; (e) w = 72; (f) w = 48; (g) w = 24; (h) average MRE.
Figure 6. Comparison of the MRE obtained by the base algorithms and the ensemble scheme on each subproblem for each value of w used. (a) w = 168; (b) w = 144; (c) w = 120; (d) w= = 96; (e) w = 72; (f) w = 48; (g) w = 24; (h) average MRE.
Energies 11 00949 g006
Figure 7. Comparison of the MAE obtained by the base algorithms and the ensemble scheme on each subproblem for each value of w used. (a) w = 168; (b) w = 144; (c) w = 120; (d) w= = 96; (e) w = 72; (f) w = 48; (g) w = 24; (h) average MAE.
Figure 7. Comparison of the MAE obtained by the base algorithms and the ensemble scheme on each subproblem for each value of w used. (a) w = 168; (b) w = 144; (c) w = 120; (d) w= = 96; (e) w = 72; (f) w = 48; (g) w = 24; (h) average MAE.
Energies 11 00949 g007
Figure 8. Comparison of the R2 obtained by the base algorithms and the ensemble scheme on each subproblem for each value of w used. (a) w = 168; (b) w = 144; (c) w = 120; (d) w= = 96; (e) w = 72; (f) w = 48; (g) w = 24; (h) average R2.
Figure 8. Comparison of the R2 obtained by the base algorithms and the ensemble scheme on each subproblem for each value of w used. (a) w = 168; (b) w = 144; (c) w = 120; (d) w= = 96; (e) w = 72; (f) w = 48; (g) w = 24; (h) average R2.
Energies 11 00949 g008
Figure 9. Comparison of the sMAPE obtained by the base algorithms and the ensemble scheme on each subproblem for each value of w used. (a) w = 168; (b) w = 144; (c) w = 120; (d) w= = 96; (e) w = 72; (f) w = 48; (g) w = 24; (h) average sMAPE.
Figure 9. Comparison of the sMAPE obtained by the base algorithms and the ensemble scheme on each subproblem for each value of w used. (a) w = 168; (b) w = 144; (c) w = 120; (d) w= = 96; (e) w = 72; (f) w = 48; (g) w = 24; (h) average sMAPE.
Energies 11 00949 g009
Figure 10. Comparison of the RMSE obtained by the base algorithms and the ensemble scheme on each subproblem for each value of w used. (a) w = 168; (b) w = 144; (c) w = 120; (d) w= = 96; (e) w = 72; (f) w = 48; (g) w = 24; (h) average rMSE.
Figure 10. Comparison of the RMSE obtained by the base algorithms and the ensemble scheme on each subproblem for each value of w used. (a) w = 168; (b) w = 144; (c) w = 120; (d) w= = 96; (e) w = 72; (f) w = 48; (g) w = 24; (h) average rMSE.
Energies 11 00949 g010
Figure 11. Comparison of real and predicted values for a subset of the time serie, for w = 168 . (a) 250 readings; (b) 1000 readings.
Figure 11. Comparison of real and predicted values for a subset of the time serie, for w = 168 . (a) 250 readings; (b) 1000 readings.
Energies 11 00949 g011
Table 1. Dataset information depending on the value of w.
Table 1. Dataset information depending on the value of w.
w#Rows#ColumnsFile Size (In MB)
2420,742486
4820,741729
7220,7409611.9
9620,73912014.9
12020,73814417.9
14420,73716820.9
16820,73619223.9
Table 2. Ranking of the methods according to their performances obtained on different values of w, according to the MRE.
Table 2. Ranking of the methods according to their performances obtained on different values of w, according to the MRE.
16814412096724824
GBMGBMGBMGBMGBMGBMGBM
NNNNRFRFRFRFRF,NN,EV
RFRFNN,EVNN,EVEVNN,EV
EVEV NN
Table 3. Average results for different historical window values. Standard deviation between brackets.
Table 3. Average results for different historical window values. Standard deviation between brackets.
w LRARMAARIMADLDTGBMRFEVNNENSEMBLE
24MRE4.44 (2.27)7.67 (5.37)8.82 (5.31)4.51 (0.52)9.52 (1.55)8.07 (3.82)4.39 (2.13)4.49 (1.91)4.39 (2.23)3.58 (1.65)
R 2 0.86 (0.11)0.56 0.45()0.50 (0.45)0.85 (0.03)0.52 (0.14)0.53 (0.37)0.85 (0.11)0.84 (0.11)0.87 (0.11)0.89 (0.08)
MAE1224.06 (613.04)2096.85 (1440.9)2335.87 (1339.15)1221.30 (153.89)2570.06 (419.80)2179.40 (1016.69)1200.82 (564.23)1232.31 (512.40)1212.29 (605.33)994.40 (450.33)
RMSE1541.10 (731.72)2564.07 (1691.68)2854.95 (1596.35)1712.24 (229.01)3167.26 (507.99)2847.41 (1322.07)1619.18 (719.97)1686.17 (702.20)1530.60 (726.11)1373.46 (611.06)
sMAPE0.04 (0.02)0.07 (0.05)0.08 (0.05)0.04 (0.01)0.09 (0.01)0.08 (0.04)0.04 (0.02)0.04 (0.02)0.04 (0.02)0.03 (0.02)
48MRE4.28 (2.15)8.67 (4.71)8.26 (4.73)3.46 (0.33)9.45 (1.48)6.59 (2.71)3.69 (1.71)3.98 (1.52)4.27 (2.16)2.95 (1.19)
R 2 0.87 (0.10)0.42 (0.48)0.56 (0.39)0.92 (0.02)0.53 (0.13)0.71 (0.19)0.91 (0.07)0.88 (0.07)0.88 (0.10)0.93 (0.05)
MAE2416.92 (1306.55)2162.29 (1177.52)1183.61 (583.02)940.30 (68.74)2533.13 (402.45)1767.13 (722.83)683.95 (212.99)882.94 (258.51)914.31 (372.27)661.85 (220.79)
RMSE3088.10 (1695.05)2673.49 (1451.80)1502.38 (702.01)1278.91 (96.08)3123.00 (484.50)2307.27 (938.98)1346.35 (559.10)1501.66 (569.11)1478.45 (686.62)1123.27 (448.52)
sMAPE0.04 (0.02)0.09 (0.05)0.08 (0.04)0.03 (0.01)0.09 (0.01)0.07 (0.03)0.04 (0.02)0.04 (0.01)0.04 (0.01)0.03 (0.01)
72MRE4.20 (2.11)8.08 (4.54)11.37 (10.43)3.39 (0.30)9.33 (1.39)5.73 (2.23)2.93 (1.16)3.48 (1.18)4.13 (2.05)2.64 (0.99)
R 2 0.88 (0.09)0.55 (0.43)−0.07 (1.97)0.91 (0.02)0.54 (0.11)0.77 (0.15)0.94 (0.04)0.91 (0.05)0.88 (0.09)0.94 (0.03)
MAE1160.92 (568.26)2240.28 (1253.74)2964.83 (2665.97)933.16 (57.97)2501.64 (382.32)1554.40 (616.75)807.90 (311.53)958.09 (321.24)1135.84 (551.56)733.53 (274.41)
RMSE1479.29 (680.59)2728.97 (1490.84)3562.36 (3098.46)1268.38 (70.18)3088.75 (457.78)2062.79 (819.53)1095.35 (402.20)1308.87 (450.01)1452.17 (664.60)1002.69 (372.10)
sMAPE0.04 (0.02)0.08 (0.05)0.10 (0.08)0.03 (0.01)0.09 (0.01)0.06 (0.02)0.03 (0.01)0.03 (0.01)0.04 (0.02)0.03 (0.01)
96MRE3.57 (1.57)4.66 (1.81)14.03 (13.00)3.12 (0.42)9.40 (1.45)5.33 (2.08)2.78 (1.04)3.42 (1.15)3.55 (1.56)2.57 (0.97)
R 2 0.91 (0.05)0.86 (0.08)−0.79 (3.57)0.92 (0.02)0.53 (0.12)0.79 (0.13)0.94 (0.03)0.91 (0.05)0.91 (0.05)0.95 (0.03)
MAE989.02 (429.51)1264.58 (479.21)3689.22 (3335.65)852.76 (82.42)2522.55 (397.32)1461.03 (555.86)768.15 (278.34)942.47 (312.53)974.30 (420.29)714.16 (266.36)
RMSE1279.62 (522.56)1619.73 (613.18)4562.39 (4075.72)1179.32 (114.33)3104.58 (472.31)1959.78 (735.49)1032.84 (358.21)1277.09 (431.31)1262.61 (513.00)969.66 (351.15)
sMAPE0.04 (0.02)0.04 (0.02)0.12 (0.09)0.03 (0.01)0.09 (0.01)0.05 (0.02)0.03 (0.01)0.03 (0.01)0.04 (0.01)0.02 (0.01)
120MRE3.33 (1.37)5.21 (1.87)6.79 (2.53)2.98 (0.28)9.08 (1.12)5.02 (1.81)2.45 (0.79)3.19 (0.95)3.15 (1.41)2.38 (0.81)
R 2 0.92 (0.05)0.84 (0.08)0.73 (0.14)0.94 (0.02)0.55 (0.10)0.81 (0.11)0.96 (0.02)0.93 (0.04)0.92 (0.05)0.96 (0.02)
MAE932.07 (381.65)1423.16 (499.93)1858.88 (682.36)814.90 (48.48)2440.62 (330.11)1368.47 (477.79)683.95 (212.99)882.94 (258.51)914.31 (372.27)661.85 (220.79)
RMSE1215.03 (475.89)1758.60 (623.36)2282.80 (817.39)1094.97 (72.06)3047.58 (430.15)1857.81 (655.27)920.01 (287.27)1184.92 (354.34)1197.00 (464.15)905.00 (300.87)
sMAPE0.03 (0.01)0.05 (0.02)0.07 (0.03)0.03 (0.01)0.09 (0.01)0.05 (0.02)0.02 (0.01)0.03 (0.01)0.03 (0.01)0.02 (0.01)
144MRE2.15 (0.77)2.57 (0.91)7.63 (2.54)2.32 (0.29)8.86 (1.01)4.49 (1.54)2.22 (0.71)3.15 (0.90)2.16 (0.78)1.94 (0.69)
R 2 0.96 (0.02)0.95 (0.03)0.66 (0.17)0.96 (0.01)0.57 (0.09)0.85 (0.09)0.96 (0.02)0.93 (0.03)0.96 (0.02)0.97 (0.02)
MAE589.62 (211.22)712.61 (247.39)2048.65 (653.32)624.32 (54.92)2366.77 (403.55)1220.77 (403.55)608.67 (188.85)866.84 (244.39)589.27 (211.57)530.28 (184.26)
RMSE845.03 (302.78)969.86 (335.40)2579.71 (835.12)852.89 (85.96)2966.20 (355.50)1694.63 (574.47)837.13 (269.50)1184.39 (352.66)845.01 (303.06)745.31 (278.77)
sMAPE0.02 (0.01)0.03 (0.01)0.07 (0.02)0.02 (0.010.08 (0.01)0.04 (0.02)0.02 (0.01)0.03 (0.01)0.02 (0.01)0.02 (0.01)
168MRE2.07 (0.77)2.43 (0.97)6.92 (2.97 )2.46 (0.29)8.79 (0.96)4.45 (1.56)2.15 (0.69)3.09 (0.84)2.08 (0.74)1.88 (0.67)
R 2 0.97 (0.02)0.96 (0.03)0.71 (0.19)0.96 (0.01)0.58 (0.08)0.85 (0.09)0.97 (0.02)0.93 (0.03)0.97 (0.02)0.98 (0.02)
MAE562.39 (209.83)660.20 (261.58)1836.62 (757.95)661.24 (51.93)2386.25 (296.70)1712.81 (611.79)585.67 (182.05)847.30 (226.72)571.54 (217.91)513.50 (181.09)
RMSE785.84 (296.05)889.18 (356.54)2332.30 (965.02)901.78 (84.22)2988.47 (370.41)1712.81 (611.78)792.58 (303.12)1145.55 (324.05)795.58 (303.12)714.56 (266.00)
sMAPE0.02 (0.01)0.02 (0.01)0.07 (0.02)0.02 (0.010.08 (0.01)0.04 (0.02)0.02 (0.01)0.03 (0.01)0.02 (0.01)0.02 (0.01)

Share and Cite

MDPI and ACS Style

Divina, F.; Gilson, A.; Goméz-Vela, F.; García Torres, M.; Torres, J.F. Stacking Ensemble Learning for Short-Term Electricity Consumption Forecasting. Energies 2018, 11, 949. https://doi.org/10.3390/en11040949

AMA Style

Divina F, Gilson A, Goméz-Vela F, García Torres M, Torres JF. Stacking Ensemble Learning for Short-Term Electricity Consumption Forecasting. Energies. 2018; 11(4):949. https://doi.org/10.3390/en11040949

Chicago/Turabian Style

Divina, Federico, Aude Gilson, Francisco Goméz-Vela, Miguel García Torres, and José F. Torres. 2018. "Stacking Ensemble Learning for Short-Term Electricity Consumption Forecasting" Energies 11, no. 4: 949. https://doi.org/10.3390/en11040949

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop