A Scenario-Based Model Comparison for Short-Term Day-Ahead Electricity Prices in Times of Economic and Political Tension

Round 1

Reviewer 1 Report

In the current period marked by the COVID-19 crisis and the Russian-Ukrainian war, energy prices have become increasingly volatile, and making an efficient forecast of them is a real challenge. Thus, I consider the topic addressed to be interesting Interesting article and well-written from the methodological side. The graphs greatly help to understand and follow the research process that has been carried out.

But, the paper must be improved in some sections:

- In the Introduction, the authors must highlight the novelty of the study and specify the objectives pursued, the methodology used, the results obtained, and the structure of the work;

- For section 1.2. Techniques for Energy Market Prediction, a more detailed literature review needs to be done (has only one reference)

- I recommend that the Discussion and Conclusion sections be restructured. In the Discussion section, authors should draw conclusions based on the results obtained for the three scenarios that be supported by evidence from other research. The paper's final section could discuss the most important theoretical and practical implications emerging from the study, the possibility of generalizing the study results, and present possible limits of the current study. At the same time, the authors can point out future research directions.

It is necessary to include the practical implications of your results that can help academic and policymaker communities. This will help you to underline the importance of your study compared to similar ones.

Author Response

Reviewer 1 

In the current period marked by the COVID-19 crisis and the Russian-Ukrainian war, energy prices have become increasingly volatile, and making an efficient forecast of them is a real challenge. Thus, I consider the topic addressed to be interesting Interesting article and well-written from the methodological side. The graphs greatly help to understand and follow the research process that has been carried out. 

But, the paper must be improved in some sections: 

- In the Introduction, the authors must highlight the novelty of the study and specify the objectives pursued, the methodology used, the results obtained, and the structure of the work;

Novelty of the study

: day ahead prices have changed tremendously compared to pre-2020 and we have used very recent data for our experiments, combined with the required forecasting horizon of 168h to accommodate power plant control. ‘The aim of this study was to find a robust standard AI model for forecasting day-ahead prices in a highly volatile and changing market environment.’  

Two amendments have been made to the introduction. Firstly, we clarify that the time-horizon and multistep predictions are rarely present in the work on electricity price prediction. We point to the review paper of Lu et al. 2021 to support this point. Secondly, we stress that our comparison is based on the fact that only limited historical data can be used for model building and that model robustness is of particular relevance for the power plant control use case. Both points are considered in the scenario based evaluation approach. 

For section 1.2. Techniques for Energy Market Prediction, a more detailed literature review needs to be done (has only one reference)  

 The literature review has been expanded. Examples and its applications for each of the five respective approaches were added, the correspondingly provided references can be used to further read uponthe details. All new additions are marked in red.

I recommend that the Discussion and Conclusion sections be restructured. In the Discussion section, authors should draw conclusions based on the results obtained for the three scenarios that be supported by evidence from other research. The paper's final section could discuss the most important theoretical and practical implications emerging from the study, the possibility of generalizing the study results, and present possible limits of the current study. At the same time, the authors can point out future research directions.  

We will investigate how volatility in the data leads to forecast uncertainty with the usage of a dropout layer in a neural network as well as with conformal predictions. You can find more details in the paper at the end of the discussion section (line 580+). 

It is necessary to include the practical implications of your results that can help academic and policymaker communities. This will help you to underline the importance of your study compared to similar ones. 

.

 The practical results of this study are currently being tested by dispatchers in the Power Plant Dispatching Department and have a weighted influence on decision-making in power plant operation. The forecasts are used in addition to forecasts from third parties, thus increasing the data basis for decisions. Future developments should focus on explainability to answer questions like why fly ups/downs occur?, how much is each predictor going to price? or are my predictors plausible? 

Reviewer 2 Report

The paper investigates the application of machine and deep learning approaches to predict day-ahead electricity prices for a 7-day horizon on the German spot market in order to give power plants enough time to ramp up or down. The paper is well written, the methodology is sound, and the results are generally presented in a clear way, but I think it needs some revisions and amelioration. The main recommendation is to elaborate on the discussion:

1.       What motivated you to conduct this study, and why did you choose the German spot market as the focus of your research?

2.       How did you select the three test scenarios used to evaluate the performance of the models, and what criteria did you use to define them?

3.       Can you explain how you conducted hyperparameter optimization for the different machine and deep learning models and how this affected their performance?

4.       Were there any challenges or limitations you encountered while working with the data set, and if so, how did you address them?

5.       How do your findings contribute to the existing body of research on energy market prediction, and how might they be used by market participants such as power plant dispatchers?

6.       Can you comment on the computational resources required to train and evaluate the different models, and how this might impact their practical usability?

7.       How did you choose the performance metrics used to evaluate the models, and do you think there are other metrics that could provide valuable insights?

8.       What are some potential applications of your methodology beyond the day-ahead electricity price prediction task explored in this study?

9.       Can you discuss any ethical considerations that arose during the course of this research, particularly with respect to the potential impact of accurate price predictions on market dynamics?

10.   In your discussion, you suggested that the results support a particular theoretical framework. Could you explain more about how the results align with this framework and how they contribute to our understanding of it?

11.   Toward the end of the discussion, you briefly mentioned some limitations of the study. Could you expand on these limitations and discuss how they might impact the generalizability or applicability of your findings?

12.   What are some areas for future research that you believe could build upon the findings of this study?

13.   I noticed a few minor errors in spelling and grammar throughout the text. To ensure that your manuscript is presented in the best possible light, I would suggest having it thoroughly proofread by a professional editor or native English speaker. This would help to eliminate any remaining errors and ensure that the manuscript is clear, concise, and easy to read.

Author Response

Reviewer 2 

The paper investigates the application of machine and deep learning approaches to predict day-ahead electricity prices for a 7-day horizon on the German spot market in order to give power plants enough time to ramp up or down. The paper is well written, the methodology is sound, and the results are generally presented in a clear way, but I think it needs some revisions and amelioration. The main recommendation is to elaborate on the discussion:

1.       What motivated you to conduct this study, and why did you choose the German spot market as the focus of your research?  

We have made the following additions in the introduction: 

The German energy market is particularly interesting for this study as it has a unique, constantly changing market environment due to the predetermined exit path of conventional power plants and a very regionally specific increase in the number of renewable power plants. In addition, there are well-documented and publicly available data sets for this market that can be used. 

2.       How did you select the three test scenarios used to evaluate the performance of the models, and what criteria did you use to define them?  

The test scenarios were determined by the mean and standard deviation of the day-ahead electricity prices. The assumption is that the model makes higher errors if these statistics are higher than usual. This holds true as the 3^rd scenario has the lowest mean as well as lowest standard deviation. The absolute error of the best model is about 50% lower than in test scenario 1 and 2. More details can be found in section 3.2. 

3.       Can you explain how you conducted hyperparameter optimization for the different machine and deep learning models and how this affected their performance?  

We have added a detailed description of the hyperparameter optimization in section 3.3. The deep learning models do not have a default architecture or default hyperparameters so that a statement on the improvement cannot be made. However, one can see that the median RMSE is about 20 Euro/MWh higher than the 0.25 quantile. 

4.       Were there any challenges or limitations you encountered while working with the data set, and if so, how did you address them?  

In fact, there were some challenges and limitations while working with the dataset. IBy looking at the historical prices in figure 1, we can state: 1.) Older data is not relevant anymore as it has a lower volatility 2.) energy market regulations and the expansion of renewable energy as well as the reduction of conventional power plant capacities (nuclear, hard coal, …) were completely different in earlier years and change every week/month. As challenge or limitation, we can conclude, that we must use sufficienct, but as little data as needed, especially for the training set. 

Another limitation is the resolution of the neutral gas price,published only daily. A finer resolution could theoretically lead to slight improvements. Lastly, missing values in the weather data set required an imputation which is described in section 3.1. 

5.       How do your findings contribute to the existing body of research on energy market prediction, 

Two contributions are stressed in the introduction. First, we clarify that the time-horizon and multistep predictions are rarely present in the work on electricity price prediction. We point to the review paper of Lu et al. 2021 to support this point. Secondly, we stress that our comparison is based on the fact that only limited historical data can be used for model building and that model robustness is of particular relevance for the use case. Both points are considered in the scenario based evaluation approach. 

 and how might they be used by market participants such as power plant dispatchers?  

The practical results of this study are currently being tested by dispatchers in the Power Plant Dispatching Department and have a weighted influence on decision-making in power plant operation. The forecasts are used in addition to forecasts from third parties, thus increasing the data basis for decisions. Future developments should focus on explainability to answer questions like why fly ups/downs occur?, how much is each predictor going to price? or are my predictors plausible? 

6.       Can you comment on the computational resources required to train and evaluate the different models, and how this might impact their practical usability?  

The whole experiment can be conducted on a decent laptop with a Nvidia GPU within a few days. However, parallel computing could significantly reduce the training time. Inferencing predictions  can be run just on the CPU. More details in section 3.3. 

7.       How did you choose the performance metrics used to evaluate the models, and do you think there are other metrics that could provide valuable insights?  

The performance metrics were chosen by the following reasons, see Chapter discussion: “There are other possible metrics that give an indication of robustness (e.g., MAPE, DAE, and normalized variants of commonly used metrics \cite{Tschora2022}). However, in our view, RMSE is the most suitable metric, as it penalizes outliers quadratically. Furthermore, RMSE is valuable for its ease of interpretation due to unit conservation, here EUR/MWh.” 

At this point we do not think, that further error metrics do not bring any additional benefit. 

8.       What are some potential applications of your methodology beyond the day-ahead electricity price prediction task explored in this study?  

From our daily work in the field of energy generating capacity dispatching as well as energy marketing, we can derive various other products. We ask for your understanding that we do not want to present all our overlays in one paper. However, we have tried to enhance the text with general considerations. 

The basic findings and model analysis can, in principle, be applied to any form of forecasting of time series data. Given the specific context of the study, it is quite conceivable to predict other parameters in the energy market, such as the hourly forward curve Merit order. 

9.       Can you discuss any ethical considerations that arose during the course of this research, particularly with respect to the potential impact of accurate price predictions on market dynamics?  

Ethical aspect of market forecasts is an interesting point. The following is our contribution to the discussion:  In the German and European energy markets, it has become common in recent years for market participants to base their decisions on forecasts. Due to the ratio of market participants and volumes traded, it is very unlikely that a forecast will give any single market participant a decisive advantage. This applies to suppliers as well as buyers and network operators. However, it will lead to a better balance between supply and demand and a reduction in dispatching costs. 

10.   In your discussion, you suggested that the results support a particular theoretical framework. Could you explain more about how the results align with this framework and how they contribute to our understanding of it?  

We are not quite sure what is meant by this question. We have not established or described a theoretical framework. Can you please identify the section or passage you are referring to? 

11.   Toward the end of the discussion, you briefly mentioned some limitations of the study. Could you expand on these limitations and discuss how they might impact the generalizability or applicability of your findings?  

We have mentioned limitations regarding the error metrics we used. In our view, we chose an appropriate metric and we justified that in the paper. One could think of using other metrics, e.g., other types or metrics over a certain time span, but the generalizability or applicability will suffer if you use highly specific metric. Because of the nature of our used metrics, generalizability or applicability is implied and is persistent at any time. In addition to that, using custom metrics makes comparisons with other papers impractical. 

Furthermore, a shorter forecast horizon decreases the complexity of the prediction task and could lead to better forecasts. Also, no regularization methods were incorporated. 

12.   What are some areas for future research that you believe could build upon the findings of this study?  

We will investigate how volatility in the data leads to forecast uncertainty with the usage of a dropout layer in a neural network as well as with conformal predictions. You can find more details in the paper at the end of the discussion section (line 580+). 

13.   I noticed a few minor errors in spelling and grammar throughout the text. To ensure that your manuscript is presented in the best possible light, I would suggest having it thoroughly proofread by a professional editor or native English speaker. This would help to eliminate any remaining errors and ensure that the manuscript is clear, concise, and easy to read.  

Thank you for the note, we have checked the text with dedicated language software.   

Reviewer 3 Report

Dear Authors!

The work is certainly interesting and useful, given the importance of price forecasting in an unstable world economy.

As a specialist in the field of machine learning and artificial intelligence, your logic is clear.

However, I would like to draw your attention to some points that may help increase the value of your research, namely:

1. I recommend describing the machine learning model in more detail (instructions, patterns, and inferences).

2. Give a clearer description of the hyperparameter optimization process.

3. Taking into account the name of the journal "Algorithms", the reduction of the algorithm in any classical form of representation suggests itself (this is an important point).

4. Many places in the work are about optimization (and this is true), but this is not supported by any formula (please take into account the audience of the readers of the magazine).

5. Is it possible to consider the algorithm proposed by you: a. sustainable and b. additive? If yes, what is the rate of its adaptation?

6. I recommend to improve the quality of graphic materials.

7. And in conclusion, you conducted a good review of the literature, made the appropriate conclusion, but I still did not find out for myself how accurate your forecast is, especially in the current conditions, when the domestic market is unpredictable, and the possibilities of energy enterprises are very limited, taking into account the influence of political factors and influences (external / internal)

P.S.: The question does not require a mandatory answer, but still it is quite difficult to predict anything under such conditions.

Author Response

Reviewer 3 

Dear Authors! 

The work is certainly interesting and useful, given the importance of price forecasting in an unstable world economy. 

As a specialist in the field of machine learning and artificial intelligence, your logic is clear. 

However, I would like to draw your attention to some points that may help increase the value of your research, namely: 

1. I recommend describing the machine learning model in more detail (instructions, patterns, and inferences). 

We agree on that this information is important for reproducibility. For this reason, we chose where possible implementation available as open source, in particular the scikit learn and sktime libraries. The paper references both libraries. We believe, that this information is sufficient for the reader to find the relevant information about the models and their implementation. For the two models that are not publicly available (namely the LSTM and the hybrid CNN-LSTM model) we have described the models in more detail.  

2. Give a clearer description of the hyperparameter optimization process.

We have added a detailed description of the hyperparameter optimization in section 3.3. The deep learning models do not have a default architecture or default hyperparameters so that a statement on the improvement cannot be made. However, one can see that the median RMSE is about 20 Euro/MWh higher than the 0.25 quantile. 

3. Taking into account the name of the journal "Algorithms", the reduction of the algorithm in any classical form of representation suggests itself (this is an important point).

               The focus of this paper is a comparison of existing ML model algorithms for a particular application. No model inventions or variants were made.  

4. Many places in the work are about optimization (and this is true), but this is not supported by any formula (please take into account the audience of the readers of the magazine). 

Since we have only used standard functions, we do not consider it necessary to mathematically redefine them again here. this has already been done sufficiently in other papers cited here. We have included statistical formulas for explanatory purposes. 

5. Is it possible to consider the algorithm proposed by you: a. sustainable and b. additive? If yes, what is the rate of its adaptation?

Obviously, the deep learning methods are suitable for an incremental or warm start training. But these aspects are not the focus of the presented work. The objective was not to provide an online or incremental learning approach, which would be another interesting research topic. It was therefore also not part of the evaluation, and no specific rate of adaption was used in the model training. 

If additive is to be understood in the sense of “Generalized Additive Models”, this also was not the focus of our work. There are certainly interactions between features, and the marginal effects of each variable are not directly visible. This could be an important contribution towards model interpretability, which we discuss in the future work section. 

In theory, every of our investigated models is suitable to build an ensemble model where predictions are combined. However, this is not the focus of this work. 

6. 6. I recommend to improve the quality of graphic materials.

The diagrams now use the entire column width which looks more appropriate and makes the texts more readable.  

7. And in conclusion, you conducted a good review of the literature, made the appropriate conclusion, but I still did not find out for myself how accurate your forecast is, especially in the current conditions, when the domestic market is unpredictable, and the possibilities of energy enterprises are very limited, taking into account the influence of political factors and influences (external / internal)

P.S.: The question does not require a mandatory answer, but still it is quite difficult to predict anything under such conditions.

 Comparisons with other paper might not be applicable as they do not have the same evaluation period. This would lead to an unfair comparison.

Round 2

Reviewer 1 Report

In this version of the updated paper, the authors took into account a part of my observations, and thus I consider their study improved to have.

However, the authors must highlight the results obtained and the structure of the paper in the Introduction section.

Reviewer 2 Report

Thank you for your hard work and dedication in addressing all of the comments that were provided during the review process. Your efforts in incorporating the suggested changes have significantly improved the quality of the paper, and it will be a valuable contribution to the scientific literature.