Dynamic Modeling of Power Outages Caused by Thunderstorms

Round 1

Reviewer 1 Report

Interesting work. More detail on genralizability to other regions of the country would help. Also, particularly in the results where the plots are illustrated, there are significant deviations at the peaks and data concerns are cited in the manuscript. At an r2 of 0.42, it is quite hard to appreciate the improvement over the state of the art. It would help to bolster the rationale and the arguments around the thesis.

Author Response

More detail on generalizability to other regions of the country would help.

One of the benefits of the dynamic modeling approach of this paper is that a model is fit for each particular region or area, without the need for detailed information about the population or infrastructure within.  This approach however does require actual outage data from each area being modeled, so the model can implicitly learn the vulnerability each area has to thunderstorms.  This data can be typically provided by a power distribution utility.  The utilities are generally very security conscious, treat this data as sensitive, and do not share it freely even with regulators.  We currently only have relationships with two utilities that operate in the American Northeast, and do not have access to data from other regions.  So it would not be possible for us to test our modeling approach in different regions at this time.

However, some of the co-authors currently have an accepted paper in The Journal of Engineering that describes the generalizability of outage prediction across a region.  Those findings show that data from different service territories and power utilities can be mixed, which suggests that OPMs are broadly generalizable. Because this kind of analysis has not yet been performed for dynamic OPMs, we have inserted in the Conclusion of our manuscript a call for future work to this effect.

 

Also, particularly in the results where the plots are illustrated, there are significant deviations at the peaks and data concerns are cited in the manuscript. At an r2 of 0.42, it is quite hard to appreciate the improvement over the state of the art.

We have included in our results a Poisson regression model that is built on the dynamic framework we developed in our initial manuscript. This model is able to accurately capture peaks for several low-, medium-, and high-impact storms (Figure 4). However, even this model that is particularly skilled in predicting large outage peaks is prone to sources of error which we analyze in the Discussion and illustrate in Figure 5 and Supplementary Figures 4-5 of our revised manuscript. In particular, storms that are especially transient and do not have a clear and strong front appear to be more difficult to predict.

In our manuscript, we now compare several dynamic OPMs (including the LSTM and Poisson regression models) against a baseline, state-of-the-art eventwise OPM that was modified for the thunderstorm outage prediction problem with the HRRR dataset. These eventwise OPMs by definition do not provide temporally granular outage predictions. The baseline eventwise OPM we consider can predict only that all hours of a given storm will have severe outage impacts, or that none will. We have compiled results for each model in Table 1, and find that the baseline OPM has an r^2 of 0.142 with respect to SOD, whereas the best-performing dynamic OPM with respect to this metric (random forest) demonstrates an r^2 of 0.521. In addition to the SOD metric, our dynamic OPMs show substantial improvements over the baseline with respect to several new metrics related to multiple aspects of outage prediction accuracy.

It would help to bolster the rationale and the arguments around the thesis.

We very much appreciate the reviewer’s comment, and we believe we have greatly strengthened our thesis that a dynamic approach to modeling thunderstorm-caused outages can yield accurate outage predictions and valuable insights into timing the response of crews to repair the power grid.

We have introduced several new models that, like the LSTM we presented in our original manuscript, is built on a dynamic, hourly approach to predicting outages. We also added a baseline eventwise model and several new error metrics to our comparisons of performance so that we may better judge the improvements made by our approach. Our results show that the dynamic modeling framework shows substantial improvements with regard to the baseline across distributional, hourly, and storm total perspectives of outage prediction accuracy.

Reviewer 2 Report

Overall the paper is well written and shows a good applicaiton of recurrent net with LSTM on a metrological problem. 

The only point which is missing is the comprasion with a baseline method. Can you please add a comparsion with baseline approach and demonstrate how the proposed method is better ?

Author Response

The only point which is missing is the comprasion with a baseline method. Can you please add a comparsion with baseline approach and demonstrate how the proposed method is better?

We thank the reviewer for this extremely valuable comment. Based on this advice, we have added several new models that operate on the same dynamic framework we described in our original draft, as well as a baseline eventwise OPM. This baseline eventwise OPM (which we call the Event-OPM) is a version of an existing OPM used by the Eversource utility company, modified to the thunderstorm prediction problem with the HRRR dataset. In fact, this is the same model which we coupled with the LSTM to improve its prediction of outage quantities in our original manuscript.

In our revised manuscript, we compare the performance of all of these models with regard to six new metrics describing errors in the distribution of outages over the period of a storm, marginal hourly outages, and total storm outages (Table 1). These results show that dynamic OPMs can make substantial improvements over the baseline method in several respects.

Reviewer 3 Report

This paper proposed a method using LSTM and RNN to dynamically predict outages in thunderstorm weather. The study is useful for utility companies to develop plans to restore power supply. Detailed comments are as follows.

1. The paper provided the outage prediction results of the proposed method. However, the improvement was not demonstrated by comparing with existing outage prediction approaches used by utility companies.
2. For most thunderstorms shown in Fig 4, the predicted cumulative outages are smaller than the actual values. Please explore if there is any explanation or ways to improve its accuracy.

Author Response

The paper provided the outage prediction results of the proposed method. However, the improvement was not demonstrated by comparing with existing outage prediction approaches used by utility companies.

Thank you for this extremely valuable comment. Based on this advice, we have added several new models that operate on the same dynamic framework we described in our original draft, as well as a baseline eventwise OPM. This baseline eventwise OPM (which we call the Event-OPM) is a version of an existing OPM used by the Eversource utility company, modified to the thunderstorm prediction problem with the HRRR dataset. In fact, this is the same model which we coupled with the LSTM to improve its prediction of outage quantities in our original manuscript.

In our revised manuscript, we compare the performance of all of these models with regard to six new metrics describing errors in the distribution of outages over the period of a storm, marginal hourly outages, and total storm outages (Table 1). These results show that dynamic OPMs can make substantial improvements over the baseline method in several respects.

 

For most thunderstorms shown in Fig 4, the predicted cumulative outages are smaller than the actual values. Please explore if there is any explanation or ways to improve its accuracy.

We greatly appreciate this comment as it was a critical motivating factor in our exploration of new dynamic OPMs. The new dynamic OPMs which we have introduced include two models, random forest and Poisson regression, which demonstrate greatly improved squared correlations (r^2=0.573 and r^2=0.600, respectively) between actual and predicted cumulative storm outages. In contrast, the baseline eventwise OPM we consider demonstrates an r^2=0.262. In Table 1, we show that each of our dynamic OPMs make improvements over the eventwise OPM in at least one respect (r^2, Nash-Sutcliffe efficiency, or MAPE). Random forest and Poisson regression dynamic OPMs make improvements over the baseline OPM with respect to every metric of total outage prediction.

In addition to improving our accuracy with respect to total outage prediction, we have included an analysis in the Discussion of meteorological and forecasting effects on our dynamic OPMs’ accuracy, as well as several related illustrations (Figure 5, Supplementary Figures 4-5). In short, storms that are especially transient and do not have a clear and strong front appear to be more difficult to predict.

Reviewer 4 Report

In the following, there is a list of questions that the authors should answer:

- Can you calculate statistical metrics such as Nash-Sutcliffe efficiency or Correlation Coefficient to evaluate your system performance?

-Is it possible to derive an analytical expression of the outage probability from the proposed model?

- Did you examine your dynamic framework for another region except for US northeastern states? If so, how does this model work?

- How the hourly error in quantitatively evaluating can affect the performance of the LSTM?

-The error metrics used in the paper for evaluating model skills can be described in one subsection or the Appendix.

-I highly recommend arranging future works and conclusions as a separate conclusion section.

Author Response

Can you calculate statistical metrics such as Nash-Sutcliffe efficiency or Correlation Coefficient to evaluate your system performance?

We thank the reviewer for this suggestion. To our manuscript, we have added six new metrics describing prediction errors in the distribution of the outage time series, marginal hourly outages, and total storm outages. Among these metrics, we compute the Nash-Sutcliffe efficiency and squared correlation coefficient for hourly outage predictions. The comparative performance of the presented OPMs across these metrics is compiled in Table 1.

 

Is it possible to derive an analytical expression of the outage probability from the proposed model?

Unfortunately, we are not able to get an analytical expression for the distribution of hourly outages conditional on weather features.  It is possible to extend the machine learning models to empirically predict the probability distribution of outages through ensemble representation of model simulations, either through an error model derived based on validation data, or ensemble input forcing data. Empirical estimates of probabilities are also possible by model-specific methods — by, for example, dropout analysis with LSTMs.

We certainly understand the reviewer’s sentiment with regard to developing rigorous, analytic expression of uncertainty in weather forecast and outage reporting datasets, which are prone to errors. Although this is beyond the scope of our study, we have suggested future work in the direction of probabilistic dynamic OPMs in our Conclusion.

 

Did you examine your dynamic framework for another region except for US northeastern states? If so, how does this model work?

We thank the reviewer for this comment, and recommend this analysis of scalability as well. Accordingly, we have inserted in our Conclusion a suggestion for future work to this effect. However, we are constrained by the sensitivity with which utilities regard outage data.

One of the benefits of the dynamic modeling approach of this paper is that a model is fit for each particular region or area, without the need for detailed information about the population or infrastructure within. This approach however does require actual outage data from each area being modeled, so the model can implicitly learn the vulnerability each area has to thunderstorms.  This data can be typically provided by a power distribution utility. The utilities are generally very security conscious, treat this data as sensitive, and do not share it freely even with regulators. We currently only have relationships with two utilities that operate in the American Northeast, and do not have access to data from other regions. So it would not be possible for us to test our modeling approach in different regions at this time.

Some of the co-authors currently have an accepted paper in The Journal of Engineering that describes the generalizability of outage prediction across a region. Those findings show that data from different service territories and power utilities can be mixed, which suggests that OPMs are broadly generalizable.

 

How the hourly error in quantitatively evaluating can affect the performance of the LSTM?

Based on this excellent suggestion, we added a simulation analysis of the effects of errors in the features to our LSTM model, as well as to the three new dynamic OPMs which we have introduced in our revised manuscript, during prediction. Specifically, we added zero-centered Gaussian noise, at a variety of settings of the standard deviation, to the PCA-scored weather forecasts of a held out storm that was in aggregate the best-predicted by the dynamic OPMs. We calculated the mean absolute error at each setting of the standard deviation, repeated three hundred times in order to compute standard errors. The results of this experiment are shown in Supplementary Figure 3, and interpretation of these results has been added to the Discussion (Section 5). 

In summary, we found that large errors in PCA-scored weather forecasts induce large errors in outage predictions in the Poisson regression dynamic OPM, but that the predictions of the remaining models (including the LSTM) were relatively insensitive to these feature errors.

 

The error metrics used in the paper for evaluating model skills can be described in one subsection or the Appendix.

The metrics we have introduced — namely the expected hour of peak outages, outage time series variance, and severe outage duration — are now described in the Appendix.

 

I highly recommend arranging future works and conclusions as a separate conclusion section.

We thank the reviewer for this suggestion, we believe it has improved the readability of our paper. We now produce interpretations of our results in the Discussion (Section 5), and outline the contributions of our study and future work in the Conclusion (Section 6).

 

Round 2

Reviewer 4 Report

Thank you for your revision.