# Assessing and Comparing Short Term Load Forecasting Performance

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

- Mean absolute error (MAE),
- Mean absolute percentage error (MAPE), and
- Root mean squared error (RMSE).

## 2. Needs to Develop Short Term Load Forecasting Criteria

- Assessing the economic costs of forecasting errors on the electricity markets is more complex than often assumed.
- The cost impacts tend to be asymmetric. Under-forecasting the load peaks typically causes higher costs than a similar amount of over-forecasting.
- The economic costs of forecasting errors tend to concentrate to the load peaks at the network bottleneck and system levels and to the rarely occurring high price peaks on the markets for electricity and the ancillary services of the power system and grids.
- Some popular and useful methods tend to predict higher and shorter load peaks than what actually occur.

## 3. Cases Studied

#### 3.1. Electricity Market Costs Due to Forecasting Errors

_{1}to the total expected forecast error e is as follows.

_{0}is the expected total error of all the other forecast components. Figure 1a shows how the total error e behaves as a function of e

_{1}when e

_{0}is set to 1. For small e

_{1}, the increase in e is quadratic.

#### 3.2. Distribution Network Costs Due to Forecasting Errors

^{2.98}. The sample size is small, so the accuracy of this dependency is questionable. Nevertheless, the dependency is clearly different from f(P) = P

^{2}that results from assuming constant voltage at the purely active power load and transmission losses relative to the square of the current. Underestimating the peak loads causes much higher losses and related costs than other load forecasting errors. Thus, the impact of forecasting errors to the energy losses is very nonlinear and depends on the direction of the error and size of the load.

_{a}are almost independent from the temperature T. In the steady state, the difference between the component or insulator temperature and the ambient temperature is linearly proportional to the losses. Components are normally operated much below their nominal or design capacity and the impact of forecasting errors on the losses and aging is small. When the component during peak load is operated at or above its nominal load and is subject to high ambient temperature and poor power quality high losses, fast component aging, or expensive operational measures result from under-forecasting the load.

#### 3.3. A Consumer Cost Case Study: Load Forecasting Based Control of Domestic Energy Storage System

_{max}) was calculated. The monetary value of the cost savings depends on the customer’s load profile, so the results are given as percentage values of cost savings. The results of the simulations are shown in Figure 2. From the result points, we calculated least-squares fitted line (R1) and least-squares fitted second-order curve (R2).

_{max}describe the effect of forecasting error for the savings almost as well as traditional forecast error criteria. This is logical when most of the savings come from the decrease in monthly peak powers. When the other forecast error criteria take into account all hours during the year (8760 h), this MAE

_{max}is calculated only from one hour per month (12 h).

#### 3.4. Comparison of Methods across Different Forecasting Cases

#### 3.5. Load Peak Sensitive Validation

- The fraction of correct prediction (true positive rate, TPR)
- the false positive rate, FPR
- the Success Index (SI): SI = TPR − FPR

- Probability of detection (POD): POD = TP/TP + FN
- Critical success index (CSI): CSI = TP/(TP + FN + FP)
- False alarms ratio (FAR): FAR = FP/(FP + TP).

## 4. Criteria for other Relevant Aspects than Forecasting Performance

#### 4.1. Estimation of Confidence Intervals for Short Term Load Forecasting

#### 4.2. Computational Time

## 5. Discussion

## 6. Conclusions

## Supplementary Materials

Supplementary File 1## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## References

- Bessa, R.J.; Miranda, V.; Botterud, A.; Wang, J. ‘Good’ or ‘bad’ wind power forecasts: A relative concept. Wind Energy
**2011**, 14, 625–636. [Google Scholar] [CrossRef] - Frances, P.H. A note on the Mean Absolute Scaled Error. Int. J. Forecast.
**2016**, 32, 20–22. Available online: https://doi.org/10.1016/J.IJFORECAST.2015.03.008 (accessed on 5 February 2020). [CrossRef] [Green Version] - Schreck, S.; de la Comble, I.P.; Thiem, S.; Niessen, S. A methodological framework to support load forecast error assessment in Local energy markets. IEEE Trans. Smart Grid
**2020**. [Google Scholar] [CrossRef] - Lusis, P.; Khalipour, K.R.; Andrew, L.; Liebman, A. Short–term residential load forecasting: Impact of calendar effects and forecast granularity. Appl. Energy
**2017**, 205, 654–669. [Google Scholar] [CrossRef] - Hodge, B.; Lew, D.; Milligan, M. Short-term load forecast error distributions and implications for renewable integration studies. In Proceedings of the 2013 IEEE Green Technologies Conference (Green Tech), Denver, CO, USA, 4–5 April2013; pp. 435–442. Available online: https://doi.org/10.1109/GreenTech.2013.73 (accessed on 20 February 2020).
- Abdulla, K.; Steer, K.; Wirth, A.; Halgamuge, S. Improving the On–line Control of Energy Storage via Forecast Error Metric Customization. J. Energy Storage
**2016**, 10, 51–59. [Google Scholar] - Haben, S.; Ward, J.; Greetham, D.V.; Singleton, C.; Grindrod, P. A new error measure for forecasts of household-level, high resolution electrical energy consumption. Int. J. Forecast
**2014**, 30, 246–256. [Google Scholar] [CrossRef] [Green Version] - Hyndman, R.J.; Koehler, A.B. Another look for measures of forecast accuracy. Int. J. Forecast
**2006**, 22, 679–688. [Google Scholar] [CrossRef] [Green Version] - Jakob, M.; Neves, C.; Vukanovic’Greetham, D. Short Term Load Forecasting. In Forecasting and Assessing Risks of Individual Electricity Peaks; Mathematics of Planet Earth; Springer: Cham, Switzerland, 2020; pp. 33–34. [Google Scholar]
- Lococ, J.; Beeks, C.; Seidl, T.; Skopal, T. Paramterized Eart Movers Distance for Efficient Metric Space Indexing. In Proceedings of the SISAP ’11, Lipari, Italy, 30 June–1 July 2011; p. 2. [Google Scholar]
- Koponen, P.; Niska, H.; Mutanen, A. Mitigating the Weaknesses of Machine Learning in Short–Term Forecasting of Aggregated Power System Active Loads. In Proceedings of the IEEE INDIN19, Helsinki–Espoo, Finland, 22–25 July 2019; p. 8. [Google Scholar]
- Ding, D.; Zhang, M.; Pan, X.; Yang, M.; He, X. Modelling Extreme Events in Time Series Prediction, KDD’19. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, AK, USA, 4–8 August 2019; pp. 1114–1122. Available online: https://doi.org/10.1145/3292500.3330896 (accessed on 11 February 2020).
- Two-Price and One-Price System, Fingrid. Available online: https://www.fingrid.fi/en/services/balance-service/description-of-balance-model/two-price-and-one-price-system/ (accessed on 11 February 2020).
- Nord Pool. Elspot Day-a-Head Electricity Prices. Available online: https://www.nordpoolgroup.com/Market-data1/Dayahead/Area-Prices/ALL1/Hourly/?view=table (accessed on 11 February 2020).
- Fees, Fingrid. Available online: https://www.fingrid.fi/en/services/balance-service/fees/ (accessed on 15 April 2020).
- Koponen, P.; Salmi, T.; Evens, C.; Takala, S.; Hyttinen, A.; Brester, C.; Niska, H. Aggregated forecasting of the load control responses using a hybrid model that combines a physically based model with machine learning. In Proceedings of the CIRED 2020 Workshop, Berlin, Germany, 22–23 September 2020; p. 4. [Google Scholar]
- SmartNet. Available online: http://smartnet-project.eu/ (accessed on 15 April 2020).
- Strbac, G.; Djapic, P.; Pudjianto, D.; Konstantelos, I.; Moreira, R. Strategies for Reducing Losses in Distribution Networks; Imperial College: London, UK, 2018; p. 87. [Google Scholar]
- Husnayain, F.; Latif, M.; Garniwa, I. Tranformer Oil Lifetime Prediction Using the Arrhenius Law Based on Physical and Electrical Characteristics. In Proceedings of the IEEE 2015 International Conference on Quality in Research, Lombok, Indonesia, 10–13 August 2015; pp. 115–120. [Google Scholar]
- Bowler, N.; Liu, S. Aging mechanisms and monitoring of cable polymers. Int. J. Progn. Health Manag.
**2015**, 6, 12. [Google Scholar] - Koskela, J.; Rautiainen, A.; Järventausta, P. Utilization possibilities of electrical energy storages in households’ energy management in Finland. Int. Rev. Electr. Eng.
**2016**, 11, 607–617. [Google Scholar] [CrossRef] - Koskela, J.; Lummi, K.; Mutanen, A.; Rautiainen, A.; Järventausta, P. Utilization of electrical energy storage with power-based distribution tariffs in households. IEEE Trans. Power Syst.
**2019**, 34, 1693–1702. [Google Scholar] [CrossRef] - Lummi, K.; Rautiainen, A.; Järventausta, P.; Heine, P.; Lehtinen, J.; Hyvärinen, M.; Salo, J. Alternative power-based pricing schemes for distribution network tariff of small customers. In Proceedings of the IEEE Innovative Smart Grid Technologies-Asia (ISGT Asia), Singapore, 22–25 May 2018; p. 6. [Google Scholar]
- Mutanen, A. Improving Electricity Distribution System State Estimation with AMR-Based Load Profiles; Tampere University of Technology: Tampere, Finland, 2018; p. 91. [Google Scholar]
- Humeasu, S.; Wijaya, T.K.; Vasirani, M.; Aberer, K. Electricity load forecasting for residential customers: Exploiting aggregation and correlation between households. In Proceedings of the 2013 Sustainable internet and ICT for sustainability, SustainIT, Palermo, Italy, 30–31 October 2013; Available online: http://dx.doi.org/10.1109/SustainIT.2013.6685208 (accessed on 18 April 2020).
- Koponen, P.; Mutanen, A.; Niska, H. Assessment of some methods for short–term load forecasting. In Proceedings of the IEEE PES ISGT Europe 2014, Istanbul, Turkey, 12–15 October 2014; p. 6. [Google Scholar]
- Brester, C.; Niska, H.; Ciszek, R.; Kolehmainen, M. Weather-based fault prediction in electricity networks with artificial neural networks. In Proceedings of the IEEE World Congress on Computational Intelligence (WCCI) 2020, Glasgow, UK, 19–24 July 2020. [Google Scholar]
- Willmott, C.J.; Ackleson, S.G.; Davis, R.E.; Feddema, J.J.; Klink, K.M.; Legates, D.R.; O’Donnell, J.; Rowe, C.M. Statistics for the evaluation and comparison of models. J. Geophys. Res.
**1985**, 900, 8995–9005. [Google Scholar] [CrossRef] [Green Version] - Sarduy, J.R.G.; Di Santo, K.G.; Saidel, M.A. Linear and non-linear methods for prediction of peak load at University of São Paulo. Measurement
**2016**, 78, 187–201. Available online: https://doi.org/10.1016/J.MEASUREMENT.2015.09.053 (accessed on 18 April 2020). [CrossRef] - Grant, J.; Eltoukhy, M.; Asfour, S. Short-Term Electrical Peak Demand Forecasting in a Large Government Building Using Artificial Neural Networks. Energies
**2014**, 7, 1935–1953. [Google Scholar] [CrossRef] [Green Version] - Youden, W.J. Index for rating diagnostic tests. Cancer
**1950**, 50, 32–35. [Google Scholar] [CrossRef] - Van Aalst, R.M.; De Leeuw, F.A. European Topic Centre on Air Quality (RIVM, NILU, NOA, DNMI). In National Ozone Forecasting Systems and International Data Exchange in Northwest Europe; Report of the Technical Working Group on Data Exchange and Forecasting for Ozone Episodes in Northwest Europe (TWG-DFO)); European Environment Agency: København, Danmark, 1997. [Google Scholar]
- Gajowniczek, K.; Ząbkowski, T. Short Term Electricity Forecasting Using Individual Smart Meter Data. Procedia Comput. Sci.
**2014**, 35, 589–597. [Google Scholar] [CrossRef] [Green Version] - Ferreira, C.; Gama, J.; Matias, L.; Botterud, A.; Wang, J. A Survey on Wind Power Ramp Forecasting; Technical Report No. ANL/DIS-10-13; Argonne National Laboratory: Lemont, IL, USA, 2010. [Google Scholar]
- Zhang, J.; Florita, A.; Hodge, B.-M.; Freedman, J. Ramp forecasting performance from improved short-term wind power forecasting. In Proceedings of the ASME 2014 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference IDETC/CIE 2014, Buffalo, NY, USA, 17–20 August 2014; p. 12. [Google Scholar]
- Petiau, R.B. Confidence interval estimation for short-term load forecasting. In Proceedings of the 2009 IEEE Bucharest PowerTech, Bucharest, Romania, 28 June–2 July 2009; pp. 1–6. [Google Scholar]
- Efron, B.; Tibshirani, R.J. An Introduction to the Bootstrap; Chapman and Hall: New York, NY, USA, 1993. [Google Scholar]
- Hong, T.; Fan, S. Probabilistic electric load forecasting: A tutorial review. Int. J. Forecast.
**2016**, 32, 914–938. Available online: https://doi.org/10.1016/J.IJFORECAST.2015.11.011 (accessed on 18 April 2020). [CrossRef] - Ljung, L. System Identification, Theory for the User, 2nd ed.; Prentice Hall: Upper Saddle River, NJ, USA, 1999; p. 640. [Google Scholar]
- Madridakis, S.; Spillotis, E.; Assimakopoulos, V. Statistical and Machine Learning forecasting methods: Concerns and ways forward. PLoS ONE
**2018**, 13, 26. [Google Scholar]

**Figure 1.**(

**a**) Impact of an additional error std. component to the total error std. when assuming normally distributed independent error sequences, (

**b**) the behavior of the mean of simulated balancing error costs, (

**c**) the range of balancing error cost variation in the simulations.

**Figure 2.**Percentage yearly savings of customers when using energy storage to decrease monthly maximum peak powers and the costs of market priced energy, shown as a function of different forecast error criteria. Color of points shows the used error level (0–200%).

**Figure 3.**Load forecasting accuracy as a function of the number of aggregated customers as collected from our own published results. The results support the assumption of independent errors except for the largest group size and the other differences reduce the comparability among the cases much more.

**Figure 4.**Comparison of normalized root mean squared error (NRMSE) and mean absolute percentage error (MAPE) in 38 forecasts in 6 different short term load forecasting cases comprising together 12 different forecasting methods.

**Figure 5.**Weather-based fault prediction in the electricity network: comparison of NN models in two experiments with and without resampling. (

**a**) All faults, (

**b**) Faults caused by wind.

Criteria | Data Points | Logical Order | Mean of Residuals R1 | Max Error R1 | Median of Residual R1 | Mean of Residuals R2 | Max Error R2 | Median of Residuals R2 |
---|---|---|---|---|---|---|---|---|

SSE | 5500 | Yes | 1.70 | 26.39 | 1.32 | 1.69 | 26.14 | 1.31 |

RMSE | 5500 | Yes | 1.63 | 25.41 | 1.25 | 1.59 | 24.81 | 1.23 |

NRMSE | 5500 | No | 1.61 | 24.96 | 1.23 | 1.57 | 24.36 | 1.18 |

MAE | 5500 | Yes | 1.62 | 25.43 | 1.23 | 1.58 | 24.83 | 1.22 |

MAPE | 1310 | Yes | 1.25 | 13.20 | 1.03 | 1.25 | 13.17 | 1.04 |

MAE_{max} | 5500 | Yes | 1.70 | 25.62 | 1.32 | 1.65 | 24.89 | 1.26 |

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Koponen, P.; Ikäheimo, J.; Koskela, J.; Brester, C.; Niska, H.
Assessing and Comparing Short Term Load Forecasting Performance. *Energies* **2020**, *13*, 2054.
https://doi.org/10.3390/en13082054

**AMA Style**

Koponen P, Ikäheimo J, Koskela J, Brester C, Niska H.
Assessing and Comparing Short Term Load Forecasting Performance. *Energies*. 2020; 13(8):2054.
https://doi.org/10.3390/en13082054

**Chicago/Turabian Style**

Koponen, Pekka, Jussi Ikäheimo, Juha Koskela, Christina Brester, and Harri Niska.
2020. "Assessing and Comparing Short Term Load Forecasting Performance" *Energies* 13, no. 8: 2054.
https://doi.org/10.3390/en13082054