# Advances in Time Series Forecasting Development for Power Systems’ Operation with MLOps

^{1}

^{2}

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

- i.
- We implement a probabilistic short-term forecasting model based on previous work, which we extend by new scoring and tuning methods,
- ii.
- We compare established forecasting models from research and the TSO’s published forecast from production by means of the sum of time-varying cost for all forecast deviations,
- iii
- We integrate the resulting forecast time series into a day-ahead operations framework and thoroughly discuss the quality of grid state forecasts in the chosen operational environment, and
- iv.
- Finally, we formulate a list of considerations to contrast the model’s scalability, maintainability, and trust, which were in focus during the open-source development of ProLoaF.

## 2. Methodology

#### 2.1. Statistical Forecast Evaluation Metrics

- -
- MSE: Quantifies the squared distance of the expected value ${\widehat{y}}_{t}$ from the target variable ${y}_{t}$, averaged over the prediction horizon from $t=1$ to $t=T$. This indicator reflects both the bias and variance of the predicted time-series with respect to the observed one. As such, forecasters seek to minimize it. The MSE is strictly positive and takes the squared scale of the observed data.
- -
- RMSE: The positive square root of the MSE describes, in essence, the same information but is more sensitive to outliers, as larger errors have a disproportionately large effect on it. Forecasters seek to minimize it. The RMSE is strictly positive and takes the scale of the observed data.
- -
- MASE: Averages positive and negative forecast errors equally, irrespective of the scale of the observed data. Forecasters seek to minimize it, as values smaller than one indicate that the given method outperforms an in-sample naïve persistence forecast.
- -
- QS: Reflects a balanced calibration of quantile estimates ${q}_{q}={F}^{-1}\left(q\right)$, penalizing deviations proportionally to their amplitude. Forecasters seek to minimize it. Here, q describes the probability that the observation lies within the quantile and F is the cumulative distribution function (CDF) of the observed data [24].
- -
- PICP: Quantifies the unconditional prevalence of observed data that lie within the considered PI. The PI, in turn, indicates a lower and upper limit within which we expect the target variable to be. The $P{I}_{95\%}$ is a commonly used significance level. As a high PICP implies a higher reliability, forecasters seek to maximize it. The metric is given in percent.
- -
- MIS: Evaluates the quality of produced prediction intervals on a predefined significance level. The average interval score [23] grows marginally with the sharpness [23]. By adding penalties on any positive or negative observation outside of the PI, the MIS gives a balanced calibration measure over the prediction horizon from $t=1$ to $t=T$. Thus, it reflects how tightly the predicted distribution covers the actual observation.

#### 2.2. Forecast Implementation

#### 2.2.1. Data Pre-Processing

#### 2.2.2. Auto-ML Model: auto.arima

#### 2.2.3. Auto-ML Model: Facebook Prophet

#### 2.2.4. Encoder-Decoder RNN

## 3. Numerical Results

#### 3.1. Operational Planning Context

#### 3.2. Case Study and Data

#### 3.3. Correlation Effect

#### 3.4. Data Pre-Processing

- -
- Actual load,
- -
- Actual and forecasted PV and wind energy production,
- -
- Actual local energy generation (conventional Fossil and Other RES),
- -
- Actual temperature,
- -
- Dummy variable for day of the week,
- -
- Dummy variable for hour of day and month of the year (cyclic coding as sine and cosine).

#### 3.5. Model Parameterization

#### 3.6. Computational Resources

#### 3.7. Statistical Evaluation

## 4. Discussion

#### 4.1. Making a Black-Box Sequence-to-Sequence Model Intepretable

#### 4.2. Ablation Study

#### 4.3. Designing MLOps

## 5. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## Abbreviations

ADAM | Adaptive Moment Estimation |

AIC | Akaike’s Information Criterion |

ANN | Artificial Neural Network |

ARIMA | Autoregressive Integrated Moving Average |

ASM | Active Power System Management |

CDF | Cumulative Distribution Function |

CPU | Central Processing Unit |

CUDA | Compute Unified Device Architecture |

DSO | Distribution System Operator |

DWD | Deutscher Wetterdienst |

EDA | Explorative Data Analysis |

GAMs | General Additive Methods |

GPU | Graphics Processing Unit |

GRU | Gated Recurrent Unit |

LSTM | Long-Short-Term Memory Units |

MASE | Mean Absolute Scaled Error |

MIS | Mean Interval Score |

ML | Machine Learning |

MLOps | Machine Learning Operations |

MSE | Mean Squared Error |

PI | Prediction Interval |

PICP | Prediction Interval Coverage Probability |

PLF | Probabilistic Load Forecasting |

PV | Photovoltaics |

QRF | Quantile Regression Forests |

QS | Quantile Score |

RES | Renewable Energy Source |

RMSE | Root Mean Squared Error |

RNN | Recurrent Neural Network |

Seq2Seq | Sequence to Sequence Learning Neural Networks |

SOGNO | Service-based Open-source Grid Automation Platform for Network Operation of the Future |

STLF | Short Term Load Forecasting |

TPE | Tree-Structured Parzen Estimator |

TSO | Transmission System Operator |

XGBoost | Extreme Gradient Boosting |

## References

- CEDEC; E.DSO; ENTSO-E; EURELECTRIC; GEODE (Eds.) TSO–DSO Report: An Integrated Approach to Active System Management; CEDEC; E.DSO; ENTSO-E; EURELECTRIC; GEODE: Brussels, Belgium, April 2019. [Google Scholar]
- Kloubert, M.L.; Schwippe, J.; Muller, S.C.; Rehtanz, C. Analyzing the impact of forecasting errors on redispatch and control reserve activation in congested transmission networks. In Proceedings of the 2015 IEEE Eindhoven PowerTech, Eindhoven, The Netherlands, 29 June–2 July 2015; pp. 1–6. [Google Scholar]
- Gürses-Tran, G. ProLoaF: V0.2.2. 2021. Available online: https://zenodo.org/record/6408478#.Yo8y4VRBxPY (accessed on 25 May 2022).
- Messner, J.W.; Pinson, P.; Browell, J.; Bjerregård, M.B.; Schicker, I. Evaluation of wind power forecasts—An up-to-date view. Wind. Energy
**2020**, 23, 1461–1481. [Google Scholar] [CrossRef] [Green Version] - Hyndman, R.J.; Athanasopoulos, G. (Eds.) Forecasting: Principles and Practice; Otexts: Lexington, Kentucky, May 2018. [Google Scholar]
- Dobschinski, J.; Bessa, R.; Du, P.; Geisler, K.; Haupt, S.E.; Lange, M.; Möhrlen, C.; Nakafuji, D.; de la Torre Rodriguez, M. Uncertainty Forecasting in a Nutshell: Prediction Models Designed to Prevent Significant Errors. IEEE Power Energy Mag.
**2017**, 15, 40–49. [Google Scholar] [CrossRef] [Green Version] - Hong, T.; Pinson, P.; Wang, Y.; Weron, R.; Yang, D.; Zareipour, H. Energy Forecasting: A Review and Outlook. IEEE Open Access J. Power Energy
**2020**, 7, 376–388. [Google Scholar] [CrossRef] - Murphy, A.H. What Is a Good Forecast? An Essay on the Nature of Goodness in Weather Forecasting. Weather. Forecast.
**1993**, 8, 281–293. [Google Scholar] [CrossRef] [Green Version] - Bessa, R.J.; Miranda, V.; Botterud, A.; Wang, J. ‘Good’ or ‘bad’ wind power forecasts: A relative concept. Wind. Energy
**2011**, 14, 625–636. [Google Scholar] [CrossRef] - Haupt, S.E.; Casado, M.G.; Davidson, M.; Dobschinski, J.; Du, P.; Lange, M.; Miller, T.; Mohrlen, C.; Motley, A.; Pestana, R.; et al. The Use of Probabilistic Forecasts: Applying Them in Theory and Practice. IEEE Power Energy Mag.
**2019**, 17, 46–57. [Google Scholar] [CrossRef] - Bracale, A.; Caramia, P.; Falco, P.D.; Hong, T. Multivariate Quantile Regression for Short-Term Probabilistic Load Forecasting. IEEE Trans. Power Syst.
**2020**, 35, 628–638. [Google Scholar] [CrossRef] - Toubeau, J.F.; Bottieau, J.; Vallee, F.; de Greve, Z. Deep Learning-Based Multivariate Probabilistic Forecasting for Short-Term Scheduling in Power Markets. IEEE Trans. Power Syst.
**2019**, 34, 1203–1215. [Google Scholar] [CrossRef] - Makridakis, S.; Spiliotis, E.; Assimakopoulos, V. The M4 Competition: 100,000 time series and 61 forecasting methods. Int. J. Forecast.
**2020**, 36, 54–74. [Google Scholar] [CrossRef] - Makridakis, S.; Spiliotis, E.; Assimakopoulos, V. M5 accuracy competition: Results, findings, and conclusions. Int. J. Forecast.
**2022**, 36, 224–227. [Google Scholar] [CrossRef] - Farrokhabadi, M.; Browell, J.; Wang, Y.; Makonin, S.; Su, W.; Zareipour, H. Day-Ahead Electricity Demand Forecasting Competition: Post-COVID Paradigm. IEEE Open Access J. Power Energy
**2022**, 1. [Google Scholar] [CrossRef] - Gürses-Tran, G.; Flamme, H.; Monti, A. Probabilistic Load Forecasting for Day-Ahead Congestion Mitigation. In Proceedings of the 16th International Conference on Probabilistic Methods Applied to Power Systems, PMAPS 2020, Liege, Belgium, 18–21 August 2020; pp. 1–6. [Google Scholar]
- Sutskever, I.; Vinyals, O.; Le, V.Q. Sequence to Sequence Learning with Neural Networks. Adv. Neural Inf. Process. Syst.
**2014**, 27, 5–8. [Google Scholar] - Bottieau, J.; Hubert, L.; De Greve, Z.; Vallee, F.; Toubeau, J.F. Very-Short-Term Probabilistic Forecasting for a Risk-Aware Participation in the Single Price Imbalance Settlement. IEEE Trans. Power Syst.
**2020**, 35, 1218–1230. [Google Scholar] [CrossRef] - Friedman, J.H. Greedy function approximation: A gradient boosting machine. Ann. Stat.
**2001**, 29, 1189–1232. [Google Scholar] [CrossRef] - Seabold, S.; Perktold, J. statsmodels: Econometric and statistical modeling with python. In Proceedings of the 9th Python in Science Conference, Austin, TX, USA, 28 June–3 July 2010. [Google Scholar]
- Taylor, S.J.; Letham, B. Forecasting at Scale. Am. Stat.
**2018**, 72, 37–45. [Google Scholar] [CrossRef] - Gneiting, T.; Balabdaoui, F.; Raftery, A.E. Probabilistic forecasts, calibration and sharpness. J. R. Stat. Soc. Ser.
**2007**, 69, 243–268. [Google Scholar] [CrossRef] [Green Version] - Gneiting, T.; Raftery, A.E. Strictly Proper Scoring Rules, Prediction, and Estimation. J. Am. Stat. Assoc.
**2007**, 102, 359–378. [Google Scholar] [CrossRef] - Hyndman, R.J.; Koehler, A.B. Another look at measures of forecast accuracy. Int. J. Forecast.
**2006**, 22, 679–688. [Google Scholar] [CrossRef] [Green Version] - Petropoulos, F.; Apiletti, D.; Assimakopoulos, V.; Babai, M.Z.; Barrow, D.K.; Taieb, S.B.; Bergmeir, C.; Bessa, R.J.; Bijak, J.; Boylan, J.E.; et al. Forecasting: Theory and practice. arXiv
**2020**, arXiv:2012.03854. [Google Scholar] [CrossRef] - Barandas, M.; Folgado, D.; Fernandes, L.; Santos, S.; Abreu, M.; Bota, P.; Liu, H.; Schultz, T.; Gamboa, H. TSFEL: Time Series Feature Extraction Library. SoftwareX
**2020**, 11, 100456. [Google Scholar] [CrossRef] - Taylor, G.S. pmdarima: ARIMA Estimators for Python. 2017. Available online: http://www.alkaline-ml.com/pmdarima (accessed on 25 May 2022).
- Rimi, N.F.; Bast, H.; Wittwer, C. Comparative Study of Forecasting Algorithms for Energy Data. 2019. Available online: https://ad-publications.informatik.uni-freiburg.de/theses/Master_Nishat_Fariha_Rimi_2019.pdf (accessed on 25 May 2022).
- Akiba, T.; Sano, S.; Yanase, T.; Ohta, T.; Koyama, M. Optuna: A Next-generation Hyperparameter Optimization Framework. In Proceedings of the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Anchorage, AK, USA, 4–8 August 2019. [Google Scholar]
- Ashour Novirdoust, A.; Bichler, M.; Bojung, C.; Buhl, H.; Fridgen, G.; Gretschko, V.; Hanny, L.; Knörr, J.; Maldonado, F.; Neuhoff, K.; et al. Electricity Sport Market Design 2030–2050. In Report, Fraunhofer-Publica; The Institute of Energy Economics at the University of Cologne (EWI): Cologne, Germany, 2021. [Google Scholar] [CrossRef]
- Val Leeuwen, T.; Meinerzhagen, A.K.; Raths, S.; Roether, A. Integration kurativer Maßnahmen in das Engpassmanagement im deutschen Übertragungsnetz. In Symposium Energieinnovation; Amprion: Graz, Austria, 2020. [Google Scholar]
- 50Hertz Transmission GmbH. Daten und Fakten 2019; 50Hertz Transmission GmbH: Berlin, Germany, 2020; Available online: https://www.50hertz.com (accessed on 25 May 2022).
- Böttcher, F.; Deutschländer, T.; Friedrich, A.; Friedrich, K.; Fröhlich, K.; Früh, B.; Ganske, A.; Heinrich, H.; Kreienkamp, F.; Malitz, G.; et al. National Climate Report. In Report, Deutscher Wetterdienst; Deutscher Wetterdienst: Offenbach am Main, Germany, 2017. [Google Scholar]
- Kingma, D.P.; Ba, J. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015. [Google Scholar]
- NVIDIA. NVIDIA Data Center Inference Products: Performance Specs. 2022. Available online: https://www.nvidia.com/en-us/deep-learning-ai/solutions/inference-platform/hpc/ (accessed on 25 May 2022).
- Tashman, L.J. Out-of-sample tests of forecasting accuracy: An analysis and review. Int. J. Forecast.
**2000**, 16, 437–450. [Google Scholar] [CrossRef] - Rajapaksha, D.; Bergmeir, C.; Hyndman, R.J. LoMEF: A Framework to Produce Local Explanations for Global Model Time Series Forecasts. arXiv
**2021**, arXiv:2111.07001. [Google Scholar] - Gürses-Tran, G.; Körner, T.A.; Monti, A. Introducing Explainability in Sequence-to-Sequence Learning for Short-term Load Forecasting. Electr. Power Syst. Res.
**2022**, in press. [Google Scholar] - Alliander N.V. OpenSTEF–Automated Machine Learning Pipelines. In Builds the Open Short Term Forecasting Package; Alliander N.V.: Arnhem, The Netherlands, 2022. [Google Scholar]
- Makridakis, S.; Hyndman, R.J.; Petropoulos, F. Forecasting in social settings: The state of the art. Int. J. Forecast.
**2020**, 36, 15–28. [Google Scholar] [CrossRef] - Lim, B.; Arık, S.O.; Loeff, N.; Pfister, T. Temporal Fusion Transformers for interpretable multi-horizon time series forecasting. Int. J. Forecast.
**2021**, 37, 1748–1764. [Google Scholar] [CrossRef]

**Figure 2.**Exemplary unfolded encoder-decoder architecture of sequence-to-sequence RNN. We pass inputs on both the encoder and decoder sides. The number of the layer stack is subject to parameterization. The gray boxes represent RNN cells.

**Figure 3.**Timeline of the consecutive operational planning processes of the German transmission system operators. Graphical representation adapted from [31].

**Figure 4.**Figures of generation, load, and import-exports in 50 HzT Zone (Year 2019). Data source: Entsoe Transparency Platform [32].

**Figure 5.**Average net load profiles per season from October 2018 to December 2021. See: Entsoe Transparency Platform [32].

**Figure 6.**Cost for redispatch measures in the 50 HzT grid since 2017, varying over the time of the year.

**Figure 10.**Comparison of forecast profiles with actual load in winter week 2021 with Christmas holidays as of 24 December.

PV | Offshore | Onshore | Other RES | Fossil | Load | |
---|---|---|---|---|---|---|

Mean | 1331 | 451 | 3909 | 893 | 8474 | 11,904 |

Std. Dev. | 2143 | 352 | 3397 | 641 | 2296 | 1977 |

q(25%) | 0 | 120 | 1258 | 511 | 6878 | 10,973 |

q(75%) | 1972 | 787 | 5624 | 1056 | 10,136 | 13,359 |

PV | Offshore | Onshore | Load | |
---|---|---|---|---|

Mean | 1327 | 467 | 3930 | 10,397 |

Std. Dev. | 2110 | 345 | 3322 | 1816 |

q(25%) | 0 | 142 | 1357 | 8992 |

q(75%) | 1990 | 794 | 5541 | 12,008 |

ARIMA | ARIMAX | SARIMA | SARIMAX | |
---|---|---|---|---|

Season | - | - | 24 | 24 |

Exog. | None | Temperature Wind Onshore Wind Offshore PV Conventional Other RES | None | Temperature Wind Onshore Wind Offshore PV Conventional Other RES |

Lag | 168 | 168 | 168 | 168 |

Model | (2,1,1)(0,0,0)[0] | (2,1,1)(0,0,0)[0] | (1,1,0)(1,0,2)[24] | (4,1,1)(2,0,2)[24] |

AIC | −9150.47 | −9152.59 | −10,411.57 | −10,570.10 |

Parameter | Min | Max | Best ProLoaF RNN |
---|---|---|---|

Learning Rate | 1 × 10${}^{-6}$ | 1 × 10${}^{-4}$ | 7.47 × 10${}^{-5}$ |

Batch Size | 32 | 128 | 64 |

Linear Layers | 1 | 4 | 3 |

RNN Layers | 1 | 4 | 4 |

**Table 5.**GPU performance data (see [35]).

Data Points | Tesla P40 |
---|---|

Single-Precision Performance (FP32) | 12 TFLOPS |

Integer Operations (INT8) | 47 TOPS |

GPU Memory | 24GB |

Memory Bandwidth | 346 GB/s |

Power | 250 W |

**Table 6.**Average probabilistic forecast performance of the auto.arima, Prophet, and ProLoaF, based on scaled values.

Metric | auto.arima | Prophet | ProLoaF |
---|---|---|---|

RMSE | 0.1047 | 0.0696 | 0.0514 |

Sharpness | 0.4730 | 0.2156 | 0.1748 |

PICP | 95.88% | 87.19% | 92.17% |

MIS | 0.5370 | 0.3299 | 0.075 |

Prophet | ProLoaF | |
---|---|---|

Average ${\Delta}_{t}$ in MWh | 907.18 | 666.26 |

Summed ${\Delta}_{t}$ in GWh | 3049.95 | 2437.19 |

Total Cost C in Mio.€ | 580.96 | 352.72 |

**Table 8.**Ablation results of presented ProLoaF RNN configuration. The underlined values indicate the best model in the analyzed metric category.

Metric | ProLoaF RNN | -No Temp. | -No PV | -No Fossil | Enc. Size (24 h × 6 Days = 144 Time Steps Embedded as Encoder Input Size during Training.) |
---|---|---|---|---|---|

MSE | 0.002 | 0.002 | 0.003 | 0.002 | 0.002 |

RMSE | 0.050 | 0.049 | 0.058 | 0.049 | 0.050 |

Sharpness | 0.197 | 0.220 | 0.185 | 0.219 | 0.202 |

PICP | 94.63% | 96.92% | 90.19% | 96.86% | 95.36% |

RAE | 0.235 | 0.228 | 0.272 | 0.231 | 0.237 |

MAE | 0.038 | 0.037 | 0.044 | 0.037 | 0.038 |

MIS | 0.075 | 0.073 | 0.087 | 0.073 | 0.075 |

MASE | 0.960 | 0.933 | 1.116 | 0.944 | 0.973 |

QS | 0.025 | 0.025 | 0.030 | 0.024 | 0.025 |

Residuals | 0.002 | 0.005 | 0.019 | 0.010 | 0.009 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Gürses-Tran, G.; Monti, A.
Advances in Time Series Forecasting Development for Power Systems’ Operation with MLOps. *Forecasting* **2022**, *4*, 501-524.
https://doi.org/10.3390/forecast4020028

**AMA Style**

Gürses-Tran G, Monti A.
Advances in Time Series Forecasting Development for Power Systems’ Operation with MLOps. *Forecasting*. 2022; 4(2):501-524.
https://doi.org/10.3390/forecast4020028

**Chicago/Turabian Style**

Gürses-Tran, Gonca, and Antonello Monti.
2022. "Advances in Time Series Forecasting Development for Power Systems’ Operation with MLOps" *Forecasting* 4, no. 2: 501-524.
https://doi.org/10.3390/forecast4020028