Electricity Spot Price Forecast by Modelling Supply and Demand Curve

Abstract

Electricity price forecasting has been a booming field over the years, with many methods and techniques being applied with different degrees of success. It is of great interest to the industry sector, becoming a must-have tool for risk management. Most methods forecast the electricity price itself; this paper gives a new perspective to the field by trying to forecast the dynamics behind the electricity price: the supply and demand curves originating from the auction. Given the complexity of the data involved which include many block bids/offers per hour, we propose a technique for market curve modeling and forecasting that incorporates multiple seasonal effects and known market variables, such as wind generation or load. It is shown that this model outperforms the benchmarked ones and increases the performance of ensemble models, highlighting the importance of the use of market bids in electricity price forecasting.

1. Introduction

Energy markets are commodity markets that deal exclusively with the supply and trade of energy, which can refer to electricity or any other source of energy.

Electricity stands out from other commodities because it has specific physical and economic characteristics. You cannot see, smell or touch it, and you cannot even guarantee its motion from one point to another in the same way as other commodities all because of the complex physics laws that govern it. It is economically non-storable, and the power system requires a constant balance between production and consumption. At the same time, its demand and production are dependent on many changeable variables, such as temperature, wind speed or solar radiation, or even the hours of the day or the days of the week where business and everyday activities are more intense. These unique characteristics lead to a very volatile market of high importance for governments, industry and consumers, with price dynamics that are not observable in any other market.

In the current deregulated scenario, the forecasting of electricity price has emerged as one of the major research fields [1]. The liberalization was intended to encourage competition among companies to decrease the cost of electricity. Unfortunately, occurrences that were of no concern in the regulated market, such as outages, possible blackouts and price peaks are now subjects of increasing concern. Moreover, deregulation increases electricity price uncertainty, giving an increased role to forecasting.

Precision in forecasting these electricity prices is very important, since a higher accuracy reduces the risk of under- or overestimating the revenue from the production units for the companies and provides better risk management [2]. Forecast errors have significant implications for profits, market shares and ultimately shareholder value [1]. In actual electricity markets, the price curve exhibits a rich structure that has the following characteristics: high frequency, nonconstant mean and variance, multiple seasonality, calendar effect, high level of volatility and high percentage of unusual price movements. These characteristics arise from the properties of the electricity and also from the fact that the market equilibrium is influenced by both the load and the generation side uncertainties [1]. Therefore, electricity price forecasting is essential for all market participants for their survival. In the short-term, a producer needs to forecast electricity prices to be able to formulate its bidding strategy and to optimally schedule its electric energy resources [3]. In a regulated environment, traditional generation scheduling of energy resources was based on cost minimization, satisfying the electricity demand and all operating constraints [4], and therefore, the key issue was how to accurately forecast electricity demand. In a deregulated environment, since generation scheduling of energy resources, such as hydro [5] and thermal resources [6], is now based on profit maximization [7], it is accurate price forecasting that embodies crucial information for any decision making.

Over the last years, different methods and approaches have been tried for electricity price forecasts with varying degrees of success. The increasing number of studies published has been highly motivated not only by industry sector needs, but also by the challenges that the electricity markets present.

The electricity price is obtained from the demand and supply curves in a daily hourly auction, being the direct result of the supply fulfilling the demand. However, most electricity price forecasting approaches tend to overlook these underlying effects, using only market variables such as load or renewable generation forecasts. We strongly believe that the market curves give crucial information for the price formation and therefore, should be included in the forecasting models. In particular, the main contributions of this paper are:

• Introduction of a novel method for market offers forecasting, by modeling market curves, and a way of predicting electricity price from them;
• Market curve forecasting through parameter analysis, including correlation with other market variables, multiple seasonal effects derived from spectrum analysis and vector autoregression techniques;
• Analysis of the performance of the proposed model, showing that the market curves give information that is not captured through other variables, highlighting the importance of using market offers in future models.

2. The Day-Ahead Market

The liberalization and reorganization of the electric sector changed the way generation and supplier companies interact, bringing about the need for regulated markets. One of the mechanisms that was created is spot markets, know as pool markets (because it includes all the producers and suppliers of the system), with the primary objective of achieving a balance between production and demand. The day-ahead markets are the main object of the spot-markets, so named because they take place the day before the scheduled power delivery. They help optimize the system in the short term by making it easier for generation companies to correctly schedule their units, specifically their slower-start units [8,9].

The day-ahead market is based on a sealed-bid auction, typically a uniform price auction (UPA). The UPA is a kind of sealed-bid auction that does not discriminate the price over the different players and incentivizes the use of marginal price as the base for the competitive pricing. In UPA, all of the winners would be paid at a single predetermined price, namely the market clearing price (MCP), irrespective of what they had bid, and the last winning bidder, which is called the marginal unit, will get exactly the bid price. The pricing rule and the whole process of clearing the market is simple and straightforward; it is based on the equilibrium between production and demand. The independent system operator aggregates the supply offers and demand bids and arranges them in the form of ascending and descending curves. The intersection of these two curves will determine the market price and the cleared quantity, as seen in Figure 1 [10].

3. Electricity Price Forecasting

Despite the increasing interest, the large amount of electricity markets and their different complexity make it difficult to compile all the existing models in the literature and compare them with each other. Market configuration has a high influence in the final clearing price since different electricity markets around the world have different caps and floors in the electricity price. Moreover, the number of available additional markets, such as capacity, balancing or ancillary services, will change the way agents make their offers and thus, change the electricity price in the day-ahead market. In addition, geographical characteristics play a major role. The generation mix or grid design of each market is enough to completely alter the performance of the models when applied to different markets.

The MCP is obtained from all demand and supply offers in an organized auction. The main reason behind this auction is the the system operators requiring advance notice to verify that the schedule is feasible and falls within transmission constraints. This auction happens once a day and results in prices for all periods of the following day (typically 24 h).

This means that when forecasting or modeling the day-ahead market, one has to take into account that all 24 prices for the next day are determined at the same time and thus, have the same market information. There are two interesting effects that result from this: first, the dynamics of hourly price does not behave as a univariate pure time series process, instead, these prices may be seen as daily arrays of 24 h [11]. They only approximate a time series because most of the underlying dynamics in the bids are in fact univariate time series (demand, wind production, solar production, etc.). Second, the prices have no relation to the real values observed in the electrical system load or renewable energy generation but instead are highly dependent on what the market foresaw as expected for the next day. This means that forecasts of these variables will work better as exogenous variables for the models than the actual observed values.

Regarding predictive models, most of the literature refers to point forecasts, where the target is the actual day-ahead price. The point forecast can be obtained using different techniques by modeling the hourly price in the past and forecasting or by simulating the supply and demand curve for the next hours and obtaining the price by intercepting both curves. Nevertheless, models where the demand and supply curves are the object of modeling are still to be found among the main EPF authors and journals.

Electricity price forecasts models can be separated into four distinct families: Statistical, Fundamental, Machine Learning and Hybrid. Statistical models are quite common in the literature and come from the direct application of statistical techniques. They include regression type models, autoregressive type time series models and reduced-form models (Figure 2).

Fundamental models try to describe the price dynamics by modeling important physical and economic factors and simulating the operation of a system of heterogeneous agents interacting with each other. Although favored in the industry sector, these models are often used for mid- or long-term forecasts, as they perform poorly in the short-term.

Machine learning are models which combine elements of learning, evolution and fuzziness to create approaches that are capable of capturing complex dynamic systems. There has been an increased interest in machine learning models because of their capability to capture nonlinear behaviour.

Finally, hybrid models try to combine two or more techniques from the other families for EPF. They have increased in popularity because of their ability to capture the pros of different models, minimizing the cons. Since their classification is almost impossible, they have a family of their own.

3.1. Statistical Models

Historically, the first statistical EPF techniques consisted of replicating statistical methods of load forecasting. By a simple substitution of loads or temperatures for electricity prices, the researchers were able to obtain EPF models. As time passed, more and more contemporary statistical, econometric or signal processing techniques were introduced to this area.

Statistical methods forecast the current price by using a mathematical combination of the previous prices and exogenous factors, typically consumption and production figures, or weather variables.

Most models in this family rely on linear regression and represent the target variable, i.e., the price $p_h$ for time (hour) h, by a linear combination of independent variables (the predictors):

$$p_h={\mathbf{\theta }}_h{\mathbf{X}}_h+{ϵ}_h$$

(1)

where ${\mathbf{\theta }}_h=[{\theta }_{h,0},{\theta }_{h,1},\cdots ,{\theta }_{h,n}]$ is a vector of coefficients specific to hour h, ${\mathbf{X}}_h=[1,X_{h,1},X_{h,2},$$\cdots ,X_{h,n}]$ is a vector of inputs and ${ϵ}_h$ is an error term.

In most EPF papers, the author refers to a benchmark model. It is based on searching historical data for a day with similar characteristics to the predicted day and using this historical value as a forecast for the future. The most common application was introduced by [12] and is called the naïve method. The procedure is so basic that it has been extensively used in the literature for comparison purposes. If your method cannot outperform the naïve, then it is simply not good enough. The procedure consists of taking the day of the week from the previous week and applying it to the next one. This way a Monday will be equal to last Monday, a Tuesday to last Tuesday, and so on.

In addition to this approach, most statistical models consist of multiple regressions, aiming to learn more about the relationships between several independent or predictor variables and a dependent or criterion variable.

Regression models are so common and popular in EPF that it is almost impossible to make a full review on all of them. Moreover, most papers use regression models combined with some other methods, normally more sophisticated. It is also hard to separate regression and autoregressive models since most of them use some kind of lagged electricity price as regressors.

In the last few years, the most relevant contribution to EPF has been the appearance of linear regression with a large number of input features that utilize some kind of regularization technique. If the number of regressors is too large, using least absolute shrinkage and selection operator (LASSO) or the elastic net limits them. By using a penalty factor jointly with minimizing the residual sum of squares (RSS), one can turn some of the coefficients to zero and effectively eliminate redundant regressors. These regularized regression models exhibit superior performance [12,13,14,15,16,17,18].

Another innovation in this family is the use of different calibration windows on the same model and combining them afterwards. It was shown that this approach outperforms the predictions obtained for the best ex post selected calibration window [19,20].

Autoregressive models take into account the time correlations of the data we are studying. The $X_t$ is expressed linearly in terms of its p past values (autoregressive parameter), and in terms of q values of the noise (moving average part):

$$\varphi \left(B\right)X_t=\theta \left(B\right){ϵ}_t$$

(2)

where B is the backward shift operator $B^h{X}_t=X_{t-h}$, $\varphi \left(B\right)$ is the notation for $\varphi \left(B\right)=1-{\varphi }_{1}B-\cdots -{\varphi }_p{B}^p$, and $\theta \left(B\right)$ is a notation for $\theta \left(B\right)=1+{\theta }_{1}B+\cdots +{\theta }_1{B}^q$, where ${\varphi }_1,\cdots ,{\varphi }_p$ and ${\theta }_1,\cdots ,{\theta }_q$ are the coefficients of the model. This is called the AutoRegressive Moving Average model and it is stylized ARMA($p,q$).

This model composes a family of its own because many variations can be built: ARIMA, SARIMA or ARMAX. The first includes a differentiating term to ensure stationarity; the second adds a seasonal component of period s, and the latter includes exogenous variables in the model.

Autoregressive-type models represent the backbone of all time series models of electricity prices and have been widely used. The authors of [21] successfully use an AR model, with lags of 24, 48 and 168 h, where each hour of the day is modeled separately, to account for residual autocorrelation and seasonal dynamics, with the inclusion of load forecasts, and use it for short-term EPF.

Similar to this approach, the authors of [22] propose a set of 24 h ARIMA models for weekdays (which are calibrated only to weekday prices) and a set of 24 h ARIMA models for weekends (which are calibrated to weekday and weekend prices).

Wavelet-ARIMA techniques that consist of decomposing the price series using a discrete wavelet transform, modeling the resulting detail and approximation series using ARIMA processes to obtain 24 hourly predicted values, and then applying the inverse wavelet transform to yield the predicted prices for the next 24 hours have been proposed by [23,24]. The performance of the wavelet-ARIMA technique is generally better than that of a standard ARIMA process.

An interesting methodology which combines elements of time series and multi-agent modeling has been described in [25]. They forecast 24 h prices for the next day using an ARIMA model applied to the conjectural variations of the firms participating in the Spanish power market. They find that this model performs slightly better than a pure ARIMA model.

Most recent constructions of ARIMA models in the literature have been used as benchmarks for more complex machine learning models.

Statistical models are attractive because some physical interpretation may be taken from their results, thus allowing the user to understand their behavior. They are often criticized for their limited ability to model the nonlinear behavior of electricity prices and related fundamental variables; nonetheless, in practical applications, their performances are comparable to those of their nonlinear alternatives.

3.2. Fundamental Models

The next class of models, known as fundamental or structural models, tries to capture the basic physical and economic relationships which are presented in the day-ahead market. The functional associations between fundamental drivers (loads, weather conditions, system and power plant parameters, etc.) are theorized, and the fundamental inputs are modeled and predicted independently, often via statistical or machine learning techniques. Many of the EPF approaches considered in the literature are hybrid solutions with time series, regression and neural network models using fundamental factors, such as loads, fuel prices, wind power or temperature, as input variables.

Many of these models are often developed as proprietary; therefore, their details are not disclosed publicly. In fact, one can find several providers of software for long-term EPF with parameter-rich fundamental models. At the academic level, most of the results published relate to hydro-dominant power markets, since the functional behavior of hydropower plants bidding is easier to emulate in these models than with statistical ones. In particular, the authors of [26] present a supply–demand model for the Norwegian power market from a time before the common Nordic market (Nordpool) had started. He uses hydro-inflow, snow and temperature conditions to explain spot price formation.

The authors of [27] use a fundamental model that aims to simulate the market-clearing process by minimizing the total system costs, which are constrained by generation unit technical features, regulation limits, transmission limits and the demand vs. generation balance. Thus, in this model, the estimated electricity market price can be obtained as the dual variable of the demand balance constraint. The forecast clearing price is then used as input to a neural network model to obtain the final electricity price forecasting.

Under the fundamental models, there is a subclass of simpler structural models that can be built from empirical analysis of market supply and demand curves. The first approach made consists of building the day-ahead spot price process by applying the inverse of the Box–Cox transformation to an Ornstein–Uhlenbeck process [28].

A similar approach can be seen in [29], where the authors define a hockey-stick shaped supply curve that matches the empirically observed curves better than the inverse of the Box–Cox transformation. They combine this with an inelastic vertical demand curve with horizontal stochastic variations driven from and Ornstein–Uhlbeck process.

3.3. Machine Learning Models

Machine learning is the study of computer algorithms and techniques to make decisions or predictions by the use of data. Is a very diverse group that has been developed to solve problems which traditional methods (e.g., statistical) cannot handle efficiently. It combines elements of learning, evolution and fuzziness to create approaches that are capable of capturing complex nonlinear dynamic systems and because of that are regarded as ‘intelligent’.

From all the machine learning, artificial neural networks (ANN), fuzzy systems, support vector machines (SVM) and evolutionary computation, are definitely the main classes. These models are sophisticated and flexible enough to handle complex systems and nonlinearity. This makes them very promising for short-term EPF, especially in complex markets with effects from varied sources. Although artificial neural networks have probably received the most attention, other non-parametric techniques, such as fuzzy logic, genetic algorithms, evolutionary programming and swarm intelligence, have also been applied, but typically in hybrid constructions.

With the increase of computational power available to the common user, in the last six years there has been a growing interest in deep neural networks, a neural network with a more complex structure with several hidden layers. However, despite this trend, most of the published studies are quite limited, with poor benchmarks, and they usually avoid state-of-the-art statistical methods.

ANN models have been widely employed in EPF. One output node has been used to forecast the next hour’s price [30,31], the price h hours ahead [32,33,34], the next day’s peak price [35], the next day’s average on-peak price [36] or the next day’s average baseload price [37]. Using models with several output nodes, it is possible to forecast a vector of prices, typically 24 nodes for forecasting the next day’s complete price profile [38].

Multi-layer perceptron (MLP) has also been used in EPF, as it proved to be good at capturing global data trends [34,39,40,41]. A model with a hybrid system has been proposed in [42], in which a real-coded genetic algorithm (RCGA) with an enhanced stochastic search capability is used to train an MLP. The authors show that this method can provide more accurate results for the Spanish market than a standard ARIMA model, a wavelet-ARIMA model or a fuzzy ANN.

More recently, recurrent neural networks (RNN) have been employed for electricity price. The authors of [43] employ two recurrent networks, one for time-positive forward data processing, and another for negative time-backward real prices processing. In another study, an RNN is used, in particular an LSTM, for calculating electricity prices for each hour [44].

In electricity price forecasting, SVMs are normally seen in hybrid models. However, in one of the first papers on this topic, an MLP and an SVM are compared with exactly the same inputs and conclude that the SVM has a more consistent behavior while requiring less time for optimal training [45]. A hybrid model called SVRARIMA that combines support vector regression to capture the nonlinear patterns and ARIMA models outperforms some of the existing ANN approaches and traditional ARIMA models [46].

Fuzzy logic has also been applied to EPF. One of the first applications utilizes fuzzy-c-means for classifying historical data into three distinct clusters representing peak, medium and off-peak and then employs an ANN for forecasting [47,48]. An adaptive-network-based fuzzy inference system (ANFIS) can be seen in [34], which combines an adaptive mechanism with Sugenotype rules and uses a combination of the least squares method and back-propagation for training the membership function and the linear combination parameters. They show that the ANFIS performs better than an MLP.

However, the most recent evolution on EPF has been in deep learning. Deep learning is a family of machine learning methods closely related to artificial neural networks, in which several layers of neurons are used, hence the name “deep”. The existence of many layers allows for the search of more features in the input data since each layer can identify different signals.

The first published deep learning paper [49] proposes a deep learning network using stacked denoising autoencoders. The new method is compared against machine learning techniques but also against two statistical methods. Four DL models (a DNN, two recurrent neural networks (RNNs), and a convolutional network (CNN)) were developed, and the results between them were compared using a whole year of data against a benchmark of 23 different models, including 7 machine learning models, 15 statistical methods and a commercial software [15]. Moreover, among the statistical methods, the comparison includes the fARX-Lasso and fARX-EN which are among the state-of-the-art statistical methods. The study showed that the deep learning algorithms perform slightly better than the others.

Deep learning became popular buzzwords, being the techniques seen as the state of the art. However, they impose a severe computational burden, their relative performance is not well tested and the training process is described by many hyper-parameters in which the optimal values are unknown, thus requiring a lot of effort to tune and apply.

3.4. Hybrid

Within the field of EPF, the research area that had the biggest contribution has been hybrid forecasting methods. They also have the most recent contributions in the literature. In fact, most of the approaches already discussed here are actually hybrid in some sense.

Hybrid models are very complex forecasting frameworks that are composed of several algorithms and techniques. Usually, they comprise different models for different components or modules of a forecasting technique, which can be data decomposition [50,51,52,53,54,55], feature selection [52,56,57,58,59,60], data clustering [59,61,62,63], some heuristic optimization of hyperparameters [50,55,57,58,64,65,66,67] or prediction ensemble [68,69,70,71].

3.5. Industry Sector Applications

There is a significant discrepancy between the recent advances in EPF at an academic level and at an industrial level. In fact, despite all the models that have been developed and published by the academy, most models used for industry application are much simpler. This is understandable as the industry sector has different objectives from the academy.

Most of the models discussed are complex, hard to optimize and computation heavy. If the companies require fast inputs for decision making, it becomes hard to use these models on a daily basis. It also requires a specific level of expertise to be able to correctly tune and maintain most models, especially artificial neural networks and deep learning models, which may be hard (and expensive) to find. Moreover, because of the lack of a common framework for studying and benchmarking EPF models, it becomes unclear for decision makers if the time and money needed will have the expected return. Nonetheless, EPF is of major importance in the industry sector and requires constant investment.

The most-used models in the industry sector are statistical models, in particular autoregressive dynamic models, which are an extension of the ARIMA models where the exogenous variable is also a time series forecast. In this case, the use of lagged values for the exogenous variable improves the performance of the overall model. Using the notation introduced in Section 3 we can write the model as:

$$\varphi \left(B\right)X_t=\varphi \left(B\right)V_t+{ϵ}_t$$

(3)

A curious fact about this model (and most statistical ones) is that the number of exogenous or explanatory variables V used can affect the model performance. A model with fewer exogenous variables performs better than one that uses all possible explanatory variables, mostly because of high colinearity between the variables. Because of that, there are a significant number of different models with different exogenous variables. These have been the motivation for the development of new hybrid methods that focus on regressor selection already discussed and the need to develop techniques for forecast selection or ensemble.

The difficulty found in choosing the right model for the right situation raised the need for methods that combined forecasts with the most-used approach being the ensemble averaging. Although ensemble averaging had the advantages of minimizing the risk of having a poor forecast (it is never as bad as the worst available single forecast), it also makes it impossible to have the best forecast possible (it is always worse than the best single forecast available). The pursuit of a method that could lead to the best average possible while minimizing the risk of making a poor forecast led to new techniques, in particular the use of random forests and XGBOOST algorithms to “correct” the forecast errors obtained from statistical methods. In these approaches, the statistical models predictions are used as inputs for the random forests or gradient boosting models which in turn outputs new forecasts that are slightly better than the simple average.

In addition to these methods, it is also common to see simple artificial neural network machine learning models employed in the companies, such as single-layer perceptrons or support vector machines. However, the performance of these models when compared to statistical ones is not clear. Statistical models are usually preferred over machine learning because they are easier to interpret and help in fundamental market analysis.

There is still a relevant gap in the companies needs that today’s models still fail to meet. To optimize the bidding strategy for some production units, such as hydro-reservoirs, the price forecast might not be enough. We also need to know the price sensitivity with over- or undersupply. Despite its importance to the companies, this approach is disregarded by academia, mainly because of the lack of visibility over company assets and its valuation. The method and methodology described in this paper aims to fulfill that gap by introducing a new market curve forecast model.

4. Methodology

Apart from a small family of models (fundamental/structural), most of the EPF approaches focus on studying the electricity price as a vector of single hourly observations. These observations can then be studied as a time series, and all the known techniques are available for analysis and forecasting. The prediction methods tend to disregard that there is an actual physical structure behind the price, i.e., the price is a consequent construction of a series of offers that the agents make in an organized market. This means that the electricity price is not the unknown; the real unknowns are the bids/asks (or the market curves) by themselves. If you knew all the bids for the future, we could accurately infer the electricity prices.

From an industry sector point of view, the forecast of the supply and demand curves brings many advantages when compared to simple price forecasts. You can derive different measures by looking at the curves and obtaining information impossible to get otherwise, such as: (i) price sensitivity—how will the price change when small changes occur in the curves; (ii) asset optimization—with the market curves predictions we can optimize our bidding; (iii) risk management—how risky it is to operate my power plant in a given time.

For the first point, price sensitivity is important to understand how easily a price can drop (or rise) if small changes happen in the curve, e.g., the uncertainty of the demand or renewable energy sources production. For the second point, knowing the curve for the next hours or even days lets generation companies optimize their assets in term of start-up costs, available power and correct bidding. If we combine the first two points, one can even correctly manage technologies that suffer from cannibalization, such as a hydropower plant, by knowing exactly how to bid and the impact that new generation might have on the price, helping in the computation of the water value. For the third and last point, you can easily derive possible losses from the operation and potential gains from day-ahead trading by comparing the curves to future prices.

In this work, we formulate a new method to model the demand and supply curves and try to forecast these curves for the future. We then calculate the prices from these curves and compare this method to others used in the industry sector for EPF.

4.1. Market Curve Parametrization

To use any mathematical model, we face the first obstacle: the definition of the object “market curves”. The easiest way to approach this problem is by imagining that the market curve is in fact a function defined in the space (energy, price), and although we do not know this function explicitly, we can approximate it to a simpler known function or family of functions. The goal of this approach is to define:

$$s_h\left(x\right)=f\left({\mathbf{\theta }}_h,x\right)$$

(4)

where f is a known function and ${\mathbf{\theta }}_h$ a vector of parameters of f for time period h. What defines the curves for each time period is the parameter $\mathbf{\theta }$ making it the main target to predict.

Since the market curves are offers that consist of pairings (energy, price), the actual spatial representation of them does not define a continuous function, but a discrete function defined by different blocks as seen in Figure 3. This representation make it hard for curve fitting; the existence of very small (low amount of energy offered) with very big (high amount of energy offered) blocks makes the space non-uniform, and the points might be irregularly spaced. Moreover, the blocks form a monotonic function but are not entirely increasing/decreasing, having many uniform levels over the curve.

To address this issue, the curve fitting will be made over the midpoint of each block:

Consider any offer block i in the market curve as:

$$b_{i,h}=\left(V_{i,h},p_{i,h}\right)$$

(5)

where V is the volume offered and p its price at time period h. The coordinates of the block in the curve are:

$$b_{i,h}=\left(\left(v_{1,i,h},p_{i,h}\right),\left(v_{2,i,h},p_{i,h}\right)\right)$$

(6)

where:

$$v_{1,i,h}=\sum _{j=1}^{i-1}{V}_{j,h}\mathrm{with}{p}_{j+1,h}>=p_{j,h}{v}_{2,i,h}=v_{1,i,h}+V_{i,h}$$

(7)

The interpolation points are then

$$x_i=\left\{\begin{array}{c}max\left(v_{2,i,h}\right)if{p}_{i_h}<=0\hfill \\ v_{1,i,h}+\frac{v_{2,i,h}-v_{1,i,h}}{2}otherwise\hfill \end{array}\right.y_i=p_{i,h}$$

(8)

Another point to bear in mind is that these points are not evenly spaced, and since curve fitting is performed with the least squares method, the concentration of points in a given part of the curve can affect the results. A simple way to avoid the problem is to linearly interpolate the points in (8) and then take a new set of evenly spaced points $(x_{i,h},y_{i,h})$ from the interpolation results.

Since there are two distinct market curves, supply and demand, we have in fact two sets of coordinate pairs $(x,y)$ and will need two functions $f({\mathbf{\theta }}_s,x)$ and $g({\mathbf{\theta }}_d,x)$ to correctly simulate the market, which implies two distinct parameter vector ${\mathbf{\theta }}_s$ and ${\mathbf{\theta }}_d$, bringing some complexity to the system under study. However, as a turnaround to this complexity and recalling the advantages of the curves—price sensitivity to demand volume changes, confidence intervals over price forecasts and portfolio optimization—we can achieve the same results looking at the difference:

$$\eta \left(x\right)=d\left(x\right)-s\left(x\right)$$

(9)

where $d\left(x\right)$ is the demand curve, and $s\left(x\right)$ is the supply curve.

This leads to only one single function and only one set of parameters. Let us call this function the “price sensitivity curve”, since it is a direct measure of the price changes over volume. The electricity price can be found in the root of this function, more precisely:

$$p_h=\left\{s\left(x\right):\eta =0\iff d\left(x\right)=s\left(x\right)\right\}$$

(10)

On a practical point of view, since the price is the image of the supply (or demand) curve on the root $\eta$, we can simplify the electricity price calculation by taking its inverse:

$$\zeta ={\eta }^{-1}\left(x\right)$$

(11)

This way the market clearing price will be the root of $\zeta$.

The curve fitting was evaluated using different functions: polynomials of different orders, sigmoid functions and stepwise linear functions. The purpose of the method is to have the curve shape around the MCP, meaning that the curve fitting has to be weighted to points near the root of $\zeta$. To understand how the curve behaves near its root, we calculated the MCP as the root of the fitted function and calculated the error to the real price observed. The best fit was found using a sixth degree polynomial $\zeta \left(x\right)=a{x}^6+b{x}^5+c{x}^4+d{x}^3+e{x}^2+fx+g$, where the parameters $\mathbf{\theta }=(a,b,c,d,e,f,g)$ are estimated using the weighted least squares method.

The parameters $\mathbf{\theta }$ can be seen as the representation of the market curve $\zeta$. Our main objective is to find patterns and relationships between these variables so that we can, with a given set of exogenous variables, find the respective market curve.

4.2. Parameter Analysis

By observing Figure 4, where every parameter is plotted, it is possible to perceive their time series behaviour as well as some level of seasonality.

To include seasonality in the model, we first studied the periodicity of the data series by computing its power spectrum and analyzing the periodogram (Figure 5). For low frequencies, the spectrum shows a peak at frequency $f=0.005952$, representing a cycle of 168 h, i.e., weekly seasonality. For low frequencies, the highest peak is shown at frequency $f=0.04167$, representing a cycle of 24 h, i.e., daily seasonality. In both cases, the spectrum shows lower peaks at frequencies that are multiples of the ones already identified.

4.3. Curve Forecast Model

The model proposed consists of a combination of a vector autoregressive (VAR) model with a seasonal harmonic component of periods 24 and 168 and dynamic exogenous variables. This model captures the autoregressive behavior of the time series as well as the autocorrelations between the parameters. The harmonic component helps to correctly model the seasonality of the parameters, and since electricity prices are highly dependent on external variables such as electricity demand or wind and solar production, which are time series per se, the dynamic external variables are essential to capture the dynamics of the market curves.

The autoregressive models were lightly described in Section 3. Recalling the notation, we can write our model:

$$\begin{array}{c}\hfill \mathbf{\varphi }{\left(B\right)}_p{\mathbf{Y}}_t=\sum _{i=0}^l{\mathbf{\beta }}_i{\mathbf{X}}_{t-i}+\sum _{k=0}^{K_1}\left({\mathbf{a}}_{\mathbf{1}}cos\left(\frac{2\pi }{P_1}kt\right)+{\mathbf{a}}_{\mathbf{2}}sin\left(\frac{2\pi }{P_1}kt\right)\right)+\\ \hfill \sum _{k=0}^{K_2}\left({\mathbf{a}}_{\mathbf{1}}cos\left(\frac{2\pi }{P_2}kt\right)+{\mathbf{a}}_{\mathbf{2}}sin\left(\frac{2\pi }{P_2}kt\right)\right)+\mathbf{c}+{ϵ}_t\end{array}$$

(12)

where ${\mathbf{Y}}_t=(y_{1,t},y_{2,t},\cdots ,y_{m,t})$ are the m endogenous market curve parameters, B is the backward shift operator $B^h{X}_t=X_{t-h}$, $\varphi \left(B\right)$ is the notation for $\mathbf{\varphi }\left(B\right)=1-{\mathbf{\varphi }}_{1}B-\cdots -{\mathbf{\varphi }}_p{B}^p$, $\mathbf{\varphi }$ are coefficient $m\times m$ matrixes for the autoregressive terms, ${\mathbf{\beta }}_i$ are l coefficient $(m\times n)$ matrixes for the n dynamic exogenous variables for each lag l, ${\mathbf{a}}_{\mathbf{1}}$ and ${\mathbf{a}}_{\mathbf{2}}$ the coefficient vectors $(m\times 1)$ for the harmonics, $(K_1,K_2)$ are the number of harmonic terms for each period $(P_1=24,P_2=168)$ and ${ϵ}_t$ a white noise process with mean 0 and covariance matrix $\mathsf{\Sigma }$.

The parameters can be estimated with an ordinary least squares method and the hyperparameters p, l and K can be obtained by performing a search on the hyperparameter space.

Summarizing, all the process consists of: (1) supply and demand curve interpolation; (2) price sensitivity estimation through the difference of the demand and the supply interpolated curves and its inversion to obtain the market curve in the study; (3) market-curve fitting to known parametric function; (4) parameter forecast with the model in (12); and (5) electricity price forecasting by finding the root of the forecast curve (Figure 6).

5. Empirical Results

5.1. Market Data

For the construction and evaluation of this model, we used market curves data from 2017 to 2020 from the Iberian Market OMIE. These data include all the offers (energy, price) from market agents for each hour of the period.

We also include in the analysis different market data available, such as wind production forecasts, solar production forecasts, load forecasts, availability forecasts for different technologies, interconnection capacity and forecasts and French and German market data. These data will be used as exogenous variables in the model and will help explain the curve shape for the future.

5.2. Forecasting Evaluation

Most literature agrees on the forecasting evaluation methods and metrics and sometimes even provide different metrics for their forecasts. Measuring point forecasting accuracy is based on absolute errors $A{E}_h=\left|P_h-{\widehat{P}}_h\right|$, where $P_h$ is the actual and ${\widehat{P}}_h$ is the predicted price for the period h. Obviously, the accuracy will increase with lower $A{E}_h$. For hourly forecasts, it is quite commons to see a Mean Average Error (MAE) as the mean of the absolute errors over the forecast period T:

$$\begin{array}{c}\hfill MAE=\frac{1}{T}\sum _{h=1}^T\left|P_h-{\widehat{P}}_h\right|\end{array}$$

(13)

Since electricity prices can change significantly from market to market, absolute errors are quite hard to compare between different datasets. Many authors use measures based on absolute percentage errors: $APEh=AEh/Ph$. By far the most popular is the Mean Average Percentage Error (MAPE), which is computed as the mean of absolute percentage errors over T:

$$\begin{array}{c}\hfill MAPE=\frac{1}{T}\sum _{h=1}^T\left|\frac{P_h-{\widehat{P}}_h}{P_h}\right|\end{array}$$

(14)

The MAPE measure is suitable for markets where prices are significantly higher than zero, but it can be misleading when applied to most electricity prices. In particular, when electricity prices are close to zero, MAPE values become very large, regardless of the actual absolute errors. On the other hand, when electricity prices spike, the resulting MAPE values are small, irrespective of the absolute differences. In a balance sheet impact point of view, this is inaccurate since a badly forecast price spike might have a much bigger financial impact than a poorly forecast price close to zero.

The most common approach to tackle the comparison issues between different models and different price scales is to normalize the absolute error by the average price obtained in the evaluation interval T. This yields the Weighted Mean Average Error (WMAE):

$$\begin{array}{c}\hfill WMAE=\frac{1}{T}\sum _{h=1}^T\frac{\left|P_h-{\widehat{P}}_h\right|}{{\overline{P}}_T}\end{array}$$

(15)

where ${\overline{P}}_T=\frac{1}{T}{\sum }_{h=1}^T{P}_h$ is the mean price in the time interval T.

For this reason, the metric used for the model evaluation are the MAE and WMAE.

5.3. Model Hyperparameter Selection

Since the basis of the model proposed is a VAR model for forecasting the market curves parameters, it is important to guarantee that all of the variables are stationary. To check the stationarity of the time series, we used the Augmented Dickey–Fuller (ADF) test, which tests the null hypothesis that a unit root is present in the sample. The p-value obtained for all the parameters was $p<0.005$, rejecting the null hypothesis and demonstrating that the time series are indeed stationary.

The hyperparameters of the model, the number of lags p for the VAR model, the number of lags l for the exogenous variables and the max iteration K for Fourier Series were estimated by performing a grid search. We ran a range of values for each parameter: p between 1 and 168, l between 1 and 24 and K between 1 and 12 and evaluated the model performance in three distinct ways: computing information criterion AIC, BIC e HQIC, calculating the MAE of each parameter and calculating the MAE of the resulting electricity price. Since the main target is to predict the market curves, the combination of parameters with a lower combined MAE for the seven parameters was chosen.

The criteria were evaluated for one year of in-sample data, and the errors for the parameters and price were calculated for 1 month of out-sample predictions. The results can be summarized in:

• The AIC shows a descending trend until $p=72$ and an ascending trend for higher values. This shows that for this criterion the model should be complex enough, having a high number of lags in the autoregressive term;
• On the other hand, the BIC and HQIC seem to be lower with lower parameters (low p, l and K) and start rising rapidly, suggesting a more simple model. However, the out-sample residuals of parameters and price for the models with lower BIC/HQIC, are higher than what is expected in the industry sector;
• For the parameters error, the higher order parameters MAE was lower with $p=24$, but for the lower order parameters, the lowest MAE was found for $p=72$. Since we are in the presence of seven distinct parameters, it is not possible to find a scenario where all the parameters MAE are lower. However, when comparing the two hypothesis just mentioned, the average of the MAE is lower for $p=72$;
• The price MAE is lower when $p=48$ or $p=72$, showing some relationship with the parameters MAE;
• For every measure, the best models for the remaining hyperparameter were found for $l=24$ and $K=(K_1,K_2)=(10,1)$.

After observing these results, the model chosen consisted of the hyperparameters $p=72$, $l=24$ and $K=(10,1)$. We ran the model for 2019 and 2020 and calculated the out-sample predictions for each hour and then compared to some of the best models used in the industry, both statistical and machine learning.

5.4. Model Benchmark

The model was benchmarked against other models used in the sector, both statistical and machine learning, and the results obtained can be seen in

Table 1. Model results and benchmark against industry sector models.

Model	MAE	WMAE
Market Curves	$3.0$	$8.8\%$
Statistical1	$3.6$	$10.6\%$
Statistical2	$3.7$	$10.9\%$
Statistical3	$3.4$	$10.0\%$
Statistical3	$3.7$	$10.8\%$
ELM1	$3.5$	$10.2\%$
ELM2	$3.5$	$10.3\%$
XGBOOST1	$2.8$	$8.3\%$
XGBOOST2	$2.7$	$7.9\%$

.

The statistical models are dynamic regression models with different combinations of the exogenous variables already discussed in this work and used in the market curves VAR model. The ELM models have a 500 sample random runs, and their average is considered as the final forecast. They use different exogenous variables between them but the same hyperparameters. The XGBOOST have a significant difference to the other models; they use the results from the other models, statistical and machine learning, as inputs, working as an ensemble algorithm for the other forecasts, being able to fine tune them.

These models forecast the electricity hourly price itself; so, to benchmark the model proposed in this thesis, the metric used for comparison is the electricity price MAE and WMAE although the model actually forecast the market curves for each hour of the day ahead.

It is shown that the market curve model outperforms the non-ensemble models, i.e., the different statistical models already employed in the sector and the ELM models, falling just short of the XGBOOST models. However, as already stated, the XGBOOST models work as an ensemble for the forecasts obtained with the other methods, using them as an input. All the known market variables exogenous to the other models work are also used in XGBOOST as dependent variables.

Since the proposed model outperforms the pure electricity price forecasting ones—statistical and ELM—it is reasonable to conclude that it captures some information that the others cannot find, leading to lower errors. On the other hand, the comparison with XGBOOST is proof of what is seen in the literature: ensemble models have better performance than individual forecasting model, being able to capture the advantages of each model while evading the disadvantages.

Since this is an ensemble model, it is possible to include the new market curve results in the pool of forecasts already used. By doing this, the MAE of the XGBOOST can decrease to €2.2/MWh as seen in

Table 2. Using the market curve forecasts in the XGBOOST clearly improves the forecasts.

Model	MAE	WMAE
Market Curves	$3.0$	$8.8\%$
XGBOOST2	$2.7$	$7.9\%$
XGBOOST with Market Curves	$2.2$	$6.6\%$

.

This means that, not only does the market curve model outperform all the other pure price forecasting models, but it also adds relevant information to the ensemble, significantly improving its results. This shows the the proposed model is useful for capturing market dynamics that the other models cannot, dynamics that are only obtainable with the ask/bid offers information.

Moreover, the model has shown a good adaptability when there are significant price changes, being able to capture regime changes better than any other model. By calculating the MAE of the models in the days, where the average daily price changes by more than three standard deviations, we can see a lower error than in the XGBOOST that can be improved even further by including the market curve price forecasts, as shown in

Table 3. MAE of the models when evaluated on the days where the average daily price changes by more than 3 standard deviations.

Model	MAE	WMAE
Market Curves	$3.5$	$9.8\%$
XGBOOST2	$4.1$	$11.5\%$
XGBOOST with Market Curves	$3.1$	$8.7\%$

. The market curves can actually capture the nonlinearity of the electricity prices, indicating that most of it might come from the bid and ask offer in the market.

6. Discussion and Conclusions

The challenge of predicting electricity prices in a complex market with high volatility makes EPF a very interesting field. In the industry sector point of view, EPF is a must-have for energy management, since it is impossible to correctly dispatch energy or manage risk without accurate electricity price predictions.

In the actual context, accurate predictions become more than just a management tool, they are a differentiating factor from the competition. Researching and investing in new EPF models is needed to remain competitive. This thesis presents not only a new method for EPF, but a new framework for researchers to work with. Looking at the market curves instead of the electricity price has proven more efficient than the traditional point forecasts for electricity prices.

A new market curve model was developed for forecasting electricity prices. This new approach is not only useful for electricity price forecasting, but also brings clear advantages for risk management in the industry sector. There is no other model that uses this approach to electricity price forecasting, as models and methods proposed often disregard the market curves structures that are behind the market clearing price.

The electricity price forecasts made by the market curve model outperformed traditional statistical models and some machine learning models and also showed a capacity to adapt to extreme price changes. Regime changes in time series are always hard to forecast without a complete knowledge of the underlying processes. In the electricity price forecast, the underlying process are the market curves. Incorporating them into the forecasts increases the adaptability of the model in scenarios where there are abrupt changes to the price. In addition, it gives an actual forecast for the market curves, in particular to the price sensitivity curve, given by the difference between the demand and the supply. This is an extra measure of this model and thus is not comparable to the other models in the industry sector.

Although the model proposed in this work is classic in the sense of the techniques used—vector autoregression with exogenous variables and an harmonic factor for seasonality—it is able to obtain better results than the models that are actually used by the industry sector in its everyday operations. This shows that the method of using and forecasting the market curves/offers, which is a novel method built for this model, is essential for EPF and must be studied.

This approach is innovative in the sense that the models described in the literature do not incorporate the information about the bids/asks in their price forecasts, and this is, most probably, the first model with a practical application to be designed with the market curves.

There are two important points to take from this work: First, modelling, identifying and forecasting market curves is complex, and there is still much room to improve. The second point is that incorporating this knowledge in electricity price forecasting gives better results than traditional models and gives additional information that is not usually captured by the other models.

This work suggested a parameterization of the curves by fitting them to a known easier to work function and then analyzing these parameters. However, more complex methods with a higher number of parameters can be used and will probably lead to better overall forecasts. Comparison and similarity analysis between curves can also be performed to add extra information to a market curve forecast model. Other forecasting techniques can also be applied, such as applying feature selection techniques to the exogenous variables, the state-of-art statistical models, or machine learning techniques for curve forecasting.