# Towards Improved Understanding of the Applicability of Uncertainty Forecasts in the Electric Power Industry

^{1}

^{2}

^{3}

^{4}

^{5}

^{6}

^{7}

^{8}

^{9}

^{*}

## Abstract

**:**

## 1. Introduction

- Atmospheric unpredictability,
- Uncertainty of (observational) data interpretation,
- Uncertainty when composing the forecast, and
- Forecast interpretation.

## 2. Definition of Forecast Uncertainty

- Uncertainty estimation should be conditional to a set of explanatory variables, like forecasted wind speed, wind direction, expected value of the generation level, etc. Meteorologists call this flow dependent uncertainty (e.g., [21,22]). The collection of historical forecast errors and construction of the empirical distribution should not be named the forecast intervals or identified as the forecast uncertainty.

## 3. Review of Weather Uncertainty Forecasting Approaches and Methods

- Initial condition uncertainties.
- Physical approximation uncertainties.
- Boundary condition uncertainties (at the surface and, for limited area models, at the lateral boundaries from the driving models).

#### 3.1. NWP Ensemble Generation

- Quantification of initial conditions uncertainties:
- Singular vector methodsSingular Vectors (SV) identify the directions of initial uncertainty that are responsible for the largest forecast uncertainty of a model state at a given time in the future. In their review, Diaconescu and Laprise [37] describe SVs to provide optimal information about the probability density function of model states at a future time. The fact that SVs capture the dynamically most unstable perturbations means that they identify the directions of initial uncertainty that are responsible for the largest forecast uncertainty. This property made the SV a good candidate in producing uncertainty from the initial conditions with reasonable dispersion and very suitable for medium range ensemble forecasting. In recent developments, SV are used to detect regions of large sensitivity to small perturbations in order to generate adaptive observations. Such adaptive observations have a large influence on enhancing weather forecasts and producing a better representation of uncertainty [38].
- Breeding methodsThe bred-vectors can be understood as difference fields between two non-linear fields. In order to obtain a realistic development of growing errors in the analysis cycle, the difference between the current analysis fields and the previous ensemble member fields are applied. The bred vectors are constructed as superpositions of leading local time-dependent Lyapunov vectors (named after Aleksandr Lyapunov, describe characteristic expanding and contracting directions of a dynamical system, and are being used in predictability analysis and as initial perturbations for ensemble forecasting in NWP.) of the atmosphere. When implemented back in 1994, it was important that the structure of the leading Lyapunov vectors assumed a transient period, which for large-scale atmospheric processes are 3 days, fitting the target time frame for the uncertainties in the medium range forecasting [39]. Bred vectors grow slower than singular vectors and therefore have been shown in comparisons to the singular vectors to lack spread in the first 3–6 days , but then approach the SV results [40].
- Ensemble Kalman Filter techniquesEnKF techniques are today mostly used to provide initial conditions for Ensemble prediction systems (EPS). In that sense, the EnKF is a variant of the initial conditions perturbations. The EnKF was first introduced in 1994 by [29] and has, according to Evensen gained popularity, because of its conceptual formulation and relative ease of implementation. The work in [41] states that it requires no derivation of a tangent linear operator or adjoint equations, and no integrations backward in time, such that the computational requirements are affordable and comparable with other popular sophisticated assimilation methods. The most known methods applied in the generation of initial conditions perturbations are:
- –
- Stochastic and Deterministic Ensemble Kalman Filter (EnKF)The assumption in the stochastic EnKF is that in order to generate a consistent analysis field, the observations need to be treated as random variables with stochastic features. To keep the filter computationally reasonable, a Gaussian error distribution of the observation matrix is assumed [42].The deterministic EnKF does not treat the observations as random, because it is assumed that the small, but spurious correlations between the ensembles of backgrounds and observations could lead to a degradation of analysis quality. The disadvantage of the deterministic EnKF is that the ensemble often becomes under-dispersive and needs covariance inflation or relaxation procedures to compensate for the missing spread [42]. Variants of the deterministic EnKF are:
- (a)
- Ensemble square root filter (EnSRF)
- (b)
- Ensemble adjustment Kalman filter (EAKF)
- (c)
- Ensemble transform Kalman filter (ETKF)

All these variants follow in large the same principles. - –
- Sequential and Local Ensemble Kalman Transform Filter (EnKTF)The sequential EnKF is a data assimilation method that applies an ensemble of model states to represent the error statistics of the model estimate, it applies ensemble integrations to predict the error statistics forward in time, and it uses an analysis scheme which operates directly on the ensemble of model states when observations are assimilated (EnKF: The ensemble Kalman filter home page: enkf.nersc.no).In the Local EnKF (LETKF) the computational heavy part is moved to a smaller, local scale. This practice has the advantage that more observations can be handled with parallel computing techniques and thereby exponential increase in required computational resources is avoided. The EnKF takes the analysis resulting from the data assimilation to lie in the same subspace as the expected forecast error. The LETKF makes use of the hypothesis that the dimension of the subspace corresponding to local regions is low and therefore allows operations only on relatively low-dimensional matrices in a local space. In this way all local matrices are used to compute the global states for advancement to the next forecast time [43]. The main difference between the algorithms are that the sequential algorithm assimilates observations in a sequence of small batches, while in the local algorithm the spatial domain is split into a number of local areas, where the analysis is solved independently ([42]).

Newest developments also use the EnKF method for data assimilation and perturbations in order to overcome the spacial scales issues in the global models [44]. Hybrids of EnKF, LETKF and Physical parameterization schemes, multi-models or other physics perturbations are applied in a number of meteorological centers in order to capture both initial condition and model uncertainties, even though many issues remain and no optimal solution has been found as yet [42].

- Quantification of uncertainties due to physical approximations:
- Stochastic physics (perturbations of tendencies in the physics)The principle of stochastic physics is to represent uncertainty in the model physics, or in other words the uncertainty arising from assumptions made to solve the physical equations of motion is used to perturb physical tendencies such as wind components, temperature and humidity. The first stochastic physics perturbation scheme was introduced by [45] in order to make ensemble forecasts using only perturbations in the initial conditions more reliable. In fact, it had been found that uncertainties in the model and the model’s parameterization schemes to solve sub-grid scale problems have an equally large effect on the uncertainty of weather predictions as uncertainties in the initial conditions. The first versions of the stochastic physics contained multiplicative noise perturbations on the net parametrized physics tendencies of the wind components, temperature and humidity.Since then, other, more refined or complementary methods have been developed. One of them is the so-called Stochastically Perturbed Parameterization Tendencies (SPPT) method that uses perturbations collinear to the unperturbed tendencies [46]. The main difference is that the multivariate distribution of the original scheme is replaced by an univariate distribution in order to achieve perturbations that are more consistent with the model physics. The so-called Backscatter scheme is a complementary scheme and describes aspects of structural uncertainty in the dynamic parametrization that is missing in conventional parameterization schemes [47].In summary, these types of perturbations try to simulate the uncertainty related to the assumptions taken in the NWP models physical and dynamical equations due to non-solvable sub-grid scale processes.
- Multi-Scheme approach (application of different physics schemes)The multi-scheme approach is characterized by using one NWP model, where entire parameterization schemes are exchanged in order to form new members. In that way, perturbations can be added effectively in the dynamics and physics of the model, e.g., convection, cloud and micro physics, horizontal and vertical diffusion, radiation and surface roughness. This approach, even though computationally relatively expensive, has the advantage over a multi-model approach that it is computationally easier to handle, due to the common model kernel. Additionally, the differences of the individual model results are well-defined. The main advantage of the multi-scheme approach is that the uncertainty has no time-dependence and is valid in every time step of a forecast. This means, if the amount of members is large enough and the chosen parameterization schemes reflect the uncertainty of the variables of interest, there is no post-processing or calibration required. The use of multiple physical parameterizations also permits the sampling of different possible closure assumptions in deep convection and in boundary layer processes [42], which are the main driver of uncertainty in the context of renewable energy forecasting. Here, the ideal multi-scheme ensemble is one, where the schemes are targeted to the fast physical processes in the boundary layer since these processes at the surface are the driver for uncertainty. If the uncertainties are covered by the EPS member’s parameterization schemes, the uncertainty will be well described by the ensemble in every time step of the forecast. An example of such a dedicated system for the power industry is the Multi-Scheme Ensemble Prediction System (MSEPS) run by weather ensemble service provider WEPROG (Weather & Energy PROGnoses) [48,49].

- Quantification of surface boundary condition uncertainties:
- Perturbation of surface parametersAlthough given a lower weight compared to the impact of initial conditions perturbations and model perturbations on the forecast uncertainty in the literature so far ([50]), several centers apply perturbations on surface boundary conditions. The main objective here is to account for the effects of uncertainties in the surface energy budget and the surface roughness which significantly determine latent and sensible heat fluxes and turbulences in the planetary boundary layer. Parameters that are most commonly varied include roughness length, soil moisture, snow cover, surface albedo, vegetation properties, sea surface temperature. The variations are introduced e.g., by time constant perturbation fields [50], variations relative to the climatological analysis fields ([51]), or additive, multiplicative and correlated stochastic perturbations [52]. Their impact have been shown to be significant on the quality of forecasts in particular near the surface [52,53].

- Methods that implicitly combine several sources of uncertainties:
- Multi-model approachProbably the easiest way to create an ensemble is simply to combine forecasts from several NWP models [42,54]. The underlying assumption is that the combination of different model systems is a good representation of model error. Depending on the nature of the combined models (deterministic and/or including initial condition perturbations), the resulting ensemble may also include effects of uncertain initial conditions. The interpretation of the outcome again depends on the characteristics of the ensemble. The members can be taken as random, equally likely draws from a probability distribution or as scenarios with assigned probability of occurrence [55]. Based on these assumptions, probabilistic products can be derived. This approach is particularly attractive for (energy-) meteorological services without running an own operational NWP model system. However, the sources of the resulting variability in the forecast are not necessarily transparent and therefore the statistical assumptions on the resulting distributions are error-prone. In addition, optimization of the forecast distribution can only be done statistically as a post-processing step (see later) rather than by optimizing the ensemble properties themselves. Forming a multi-model EPS with deterministic NWP models are often prone to being under-dispersive, because deterministic models usually suppress extremes, which are desired in an EPS for reliability purposes.

#### 3.2. Reliable Weather Forecast Products through Statistical Post-Processing

#### 3.2.1. Deterministic or Probabilistic Weather Input

#### 3.2.2. Application-Specific Weather Forecast Products

#### 3.3. Overview of NWP Ensemble Methods and Their Applicability in the Power Industry

## 4. From Weather to Wind Power Uncertainty

- Statistical methods based on deterministic NWP forecastsUncertainty is determined by a statistical approach using deterministic NWP input
- Methods based on NWP ensemble forecastsUncertainty is derived by applying a weather-to-power conversion method on each NWP ensemble member or by considering input derived by a reduction method applied on the ensembles

#### 4.1. Statistical Methods Based on Deterministic NWP Forecasts

#### 4.1.1. Parametric and Non-Parametric Probabilistic Forecasts

#### 4.1.2. Statistically-Based Ensembles

#### 4.2. Methods Based on NWP Ensembles

#### 4.2.1. Physical Methods

#### 4.2.2. Statistical Methods

#### 4.3. Regional Aggregation

- The capacity of the aggregation may change as new wind farms can be installed or old ones decommissioned.
- Historical data of the new wind farms are not necessarily available to retrain the forecast models on the new configuration.
- Maintenance operations and down-regulation of the wind farms production due to grid issues corrupt the measured production time series and impact the forecast performance.

## 5. Communication of Uncertainty

#### 5.1. Communication of Weather Uncertainty

- Promote enhanced decision making;
- Manage user expectations;
- Promote user confidence;
- Reflect the state of the science.

#### 5.2. Communication of Wind Power Uncertainty

- Probabilistic forecasting;
- Scenario or ensemble forecasting;
- Skill forecast;
- Ramp forecasting.

- Magnitude $\left(\Delta P\right)$: wind power variation observed during the event;
- Duration $\left(\Delta t\right)$: time period associated to the large variation event;
- Ramp rate: ramp intensity, which is the ratio between magnitude and duration $\left(\Delta P/\Delta P\right)$;
- Timing $\left({t}_{0}\right)$: time instant related with the ramp occurrence, which can be the start or central time instant of the event;
- Direction: increase or decrease in wind power.

## 6. Mapping Uncertainty Representations and End-User Requirements

#### 6.1. Reserve Requirements and Unit Commitment

#### 6.2. Participation in Electricity Markets

- Uses wind speed measurements, power curves of the wind turbines and/or physical models of the turbines to estimate in real-time the reference power (i.e., called AAP);
- Delivered amount of reserve power is calculated in real-time as the difference between the reference and the actual feed-in of the wind farm.

#### 6.3. Predictive Grid Management

#### 6.4. Maintenance Scheduling of Wind Power Plants

#### 6.5. Long-Term Portfolio Planning

- Policy makers and regulatory authorities: Annual and decadal forecasts can be utilized to investigate the associated future risks of wind resource volatility for a specific region and country. Annual to decadal forecasts help the policy makers to understand the changes in energy mix.
- Energy investors and wind power operators: Seasonal forecasts can be used to plan operation and trading of portfolios with mixed wind power and other generation capacity.
- Power transmission and distribution system operators: Weekly, monthly, seasonal, and annual wind power forecasts can be used to plan energy balance, UC/ED, and future power grid capacity expansion / reinforcement purposes for the target power system. Generation cost models simulate the UC and ED operations of the specific power system in the short to long term forecast horizon (one week to a year) with 5 min to hourly resolution.

#### 6.6. Summary of Links between Uncertainty Representations and End-User Requirements

## 7. Pitfalls in Decision-Aid Methods

#### 7.1. Uncertainty Modeling and Integration in the Decision-Making Phase

#### 7.2. Decision Quality Evaluation

#### 7.3. The Role of Short-Term Forecasting Uncertainty in Longer Term Decisions

## 8. Recommendations and Best Practices

- ERCOT added a “Reliability Risk Desk” in January 2017 into its control room for RES forecasting and extreme weather monitoring. This desk makes forecast adjustments during icing and other extreme weather events and monitors the adequacy of scheduled resources to cover forecast errors and net-load ramps [170].
- The Compagnie Nationale du Rhône (CNR) included meteorological feedback in the wind power forecasting system by exploring expert charts (maps) from different NWPs available over France. This information is then used to examine spatial-temporal trajectories when forecast uncertainty is high, and to use expert knowledge to provide different weightings to different NWP models [171].
- The Hawaiian Electric Company (HECO), system operator in Hawaii, is in the process of implementing uncertainty forecasts, a ramp warning system, Automatic Generation Control, reserve monitoring and unit commitment into the control room. The implementation is part of the Distributed Resource Energy Analysis and Management System) project (DREAMS)[172] that supports the development of the next generation, integrated energy management infrastructure able to incorporate advance visualization of behind-the-meter distributed technology performance information and probabilistic renewable energy generation forecasts to inform operational decisions [9].

## Acknowledgments

## Author Contributions

## Conflicts of Interest

## Abbreviations

AAP | Available Active Power |

aFRR | Automatic Frequency Restoration Reserve |

AnEn | Analog Ensemble |

BMA | Bayesian Model Averaging |

CIM | Common Information Model |

CLT | Central Limit Theorem |

CMC | Canadian Meteorological Center |

CRPS | Continuous Ranked Probability Score |

EAKF | Ensemble Adjustment Kalman Filter |

ECC | Ensemble Copula Coupling |

ECMWF | European Centre for Medium-Range Weather Forecasts |

ED | Economic Dispatch |

EMOS | Ensemble Model Output Statistics |

EnKF | Ensemble Kalman Filter |

EnSRF | Ensemble Square Root Filter |

EPS | Ensemble Prediction Systems |

ERCOT | Electric Reliability Council of Texas |

ETKF | Ensemble Transform Kalman Filter |

IEA | International Energy Agency |

IEC | International Electrotechnical Commission |

LETKF | Local Ensemble Kalman Filter |

MAE | Mean Absolute Error |

MOS | Model Output Statistics |

MTD | Mass Transportation Distance |

MSEPS | Multi-Scheme Ensemble Prediction System |

NCEP | National Centers for Environmental Prediction |

NWP | Numerical Weather Predictions |

Probability Density Function | |

RMSE | Root Mean Square Error |

RTO | Regional Transmission Operators |

SD | Stochastic–Dynamic |

SPPT | Stochastically Perturbed Parameterization Tendencies |

SV | Singular Vector |

UC | Unit Commitment |

VPP | Virtual Power Plant |

## References

- Pinson, P.; Chevallier, C.; Kariniotakis, G. Trading wind generation from short-term probabilistic forecasts of wind power. IEEE Trans. Power Syst.
**2007**, 22, 1148–1156. [Google Scholar] [CrossRef] [Green Version] - Matos, M.A.; Bessa, R. Setting the operating reserve using probabilistic wind power forecasts. IEEE Trans. Power Syst.
**2011**, 26, 594–603. [Google Scholar] [CrossRef] - Bessa, R.; Moreira, C.; Silva, B.; Matos, M. Handling renewable energy variability and uncertainty in power systems operation. Wiley Interdiscip. Rev. Energy Environ.
**2014**, 3, 156–178. [Google Scholar] [CrossRef] - Möhrlen, C.; Bessa, R.J.; Barthod, M.; Goretti, G.; Siefert, M. Use of forecast uncertainties in the power sector: State-of-the-art of business practices. In Proceedings of the 15th International Workshop on Large-Scale Integration of Wind Power into Power Systems as well as on Transmission Networks for Offshore Wind Power Plants, Energynautics, Vienna, Austria, 15–17 November 2016. [Google Scholar]
- Sioshansi, R.; Hurlbut, D. Market protocols in ERCOT and their effect on wind generation. Energy Policy
**2010**, 38, 3192–3197. [Google Scholar] [CrossRef] - Holttinen, H.; Milligan, M.; Ela, E.; Menemenlis, N.; Dobschinski, J.; Rawn, B.; Bessa, R.; Flynn, D.; Lazaro, E.; Detlefsen, N. Methodologies to determine operating reserves due to increased wind power. IEEE Trans. Sustain. Energy
**2012**, 3, 713–723. [Google Scholar] [CrossRef] - Steffan, N.; Du, P.; Mago, N.; Sharma, S. Integrating probabilistic forecasts into the energy management system (EMS) and market management system (MMS) software. In Proceedings of the UVIG Forecasting Workshop, Atlanta, GA, USA, 20–22 June 2017. [Google Scholar]
- Matos, M.; Bessa, R.J.; Gonçalves, C.; Cavalcante, L.; Miranda, V.; Machado, N.; Marques, P.; Matos, F. Setting the maximum import net transfer capacity under extreme RES integration scenarios. In Proceedings of the 2016 International Conference on Probabilistic Methods Applied to Power Systems (PMAPS), Beijing, China, 16–20 October 2016. [Google Scholar]
- Nakafuji, D.; Gouveia, L.; Oshiro, S. Distributed Resource Energy Analysis and Management System (DREAMS) Development for Real-Time Grid Operations; Technical Report DOE-EE0006331-FTR1; Hawaiian Electric Company: Honolulu, HI, USA, 2016. [Google Scholar]
- Miettinen, J.; Holttinen, H.; Ammälä, J.; Piironen, M. Wind power forecasting at Transmission System Operator’s control room. In Proceedings of the 2015 IEEE Power & Energy Society General Meeting, Denver, CO, USA, 26–30 July 2015. [Google Scholar]
- Storck, P.; Grimit, E.; Holger, T. Determining the value of more accurate wind power forecasting in global electricity markets. In Proceedings of the EWEA Technical Workshop on Wind Power Forecasting, Leuven, Belgium, 1–2 October 2015. [Google Scholar]
- Koblitz, T. What’s next for wind energy forecasting systems? In Proceedings of the EWEA Technical Workshop on Wind Power Forecasting, Leuven, Belgium, 1–2 October 2015. [Google Scholar]
- Grimit, E. Providing uncertainty information to end users in the electric sector. In Proceedings of the UVIG Forecasting Workshop, Tutorial on Integration of Uncertainty Forecasts into Power System Operations, Atlanta, GA, USA, 20–22 June 2017. [Google Scholar]
- Gill, J.; Rubiera, J.; Martin, C.; Cacic, I.; Mylne, K.; Dehui, C.; Jiafeng, G.; Xu, T.; Yamaguchi, M.; Foamouhoue, A.K.; et al. Guidelines on Communicating Forecast Uncertainty; Technical Report WMO/TD No. 4122; World Meteorological Organization: Geneve, Switzerland, 2008. [Google Scholar]
- IEA Wind Task 36 Wind Energy Forecasting Project (2016–2018). Available online: http://www.ieawindforecasting.dk (accessed on 17 August 2017).
- Stine, R.A. Bootstrap prediction intervals for regression. J. Am. Stat. Assoc.
**1985**, 80, 1026–1031. [Google Scholar] [CrossRef] - Smithson, M. Confidence Intervals, 1st ed.; Quantitative Applications in the Social Sciences; SAGE Publications Inc.: Thousand Oaks, CA, USA, 2002. [Google Scholar]
- Pinson, P. Estimation of the Uncertainty in Wind Power Forecasting. Ph.D. Thesis, PSL Research University, Ecole des Mines de Paris, Center for Energy and Processes, Sophia Antipolis, France, 2006. [Google Scholar]
- Lewis, J.M. Roots of ensemble forecasting. Mon. Weather Rev.
**2005**, 133, 1865–1885. [Google Scholar] [CrossRef] - Palmer, T.N. The economic value of ensemble forecasts as a tool for risk assessment: From days to decades. Q. J. R. Meteorol. Soc.
**2002**, 128, 747–774. [Google Scholar] [CrossRef] - Frame, T.H.A.; Methven, J.; Gray, S.L.; Ambaum, M.H.P. Flow-dependent predictability of the North Atlantic jet. Geophys. Res. Lett.
**2013**, 40, 2411–2416. [Google Scholar] [CrossRef] - Engström, A.; Magnusson, L. Estimating trajectory uncertainties due to flow dependent errors in the atmospheric analysis. Atmos. Chem. Phys.
**2009**, 9, 8857–8867. [Google Scholar] [CrossRef] - Bludszuweit, H.; Dominguez-Navarro, J.; Llombart, A. Statistical analysis of wind power forecast error. IEEE Trans. Power Syst.
**2008**, 23, 983–991. [Google Scholar] [CrossRef] - Hodge, B.M.; Ela, E.; Milligan, M. The distribution of wind power forecasting errors from operational systems. In Proceedings of the 10th International Workshop on Large-Scale Integration of Wind Power into Power Systems, Energynautics, Aarhus, Denmark, 25–26 October 2011. [Google Scholar]
- Jordá, O.; Marcellino, M. Path forecast evaluation. J. Appl. Econom.
**2010**, 25, 635–662. [Google Scholar] [CrossRef] - Chew, V. Simultaneous prediction intervals. Technometrics
**1968**, 10, 323–330. [Google Scholar] [CrossRef] - Eady, E. Long waves and cyclone waves. Tellus
**1949**, 1, 33–52. [Google Scholar] [CrossRef] - Epstein, E.S. Stochastic dynamic prediction. Tellus
**1969**, 21, 739–759. [Google Scholar] [CrossRef] - Evensen, G. Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res. Oceans
**1994**, 99, 10143–10162. [Google Scholar] [CrossRef] - Thompson, P.; Lorenz, E. Dialogue between Phil Thompson and Ed Lorenz on 31 July 1986. Am. Meteor Soc. Tape Rec. Interview
**1986**, 1, 19–53. [Google Scholar] - Palmer, T.N. Extended-range atmospheric prediction and the Lorenz model. Bull. Am. Meteorol. Soc.
**1993**, 74, 49–65. [Google Scholar] [CrossRef] - Molteni, F.; Buizza, R.; Palmer, T.N.; Petroliagis, T. The ECMWF Ensemble Prediction System: Methodology and validation. Q. J. R. Meteorol. Soc.
**1996**, 122, 73–119. [Google Scholar] [CrossRef] - Toth, Z.; Kalnay, E. Ensemble forecasting at NMC: The generation of perturbations. Bull. Am. Meteorol. Soc.
**1993**, 74, 2317–2330. [Google Scholar] [CrossRef] - Tracton, M.S.; Kalnay, E. Operational ensemble prediction at the National Meteorological Center: Practical aspects. Weather Forecast.
**1993**, 8, 379–398. [Google Scholar] [CrossRef] - Mitchell, H.L.; Houtekamer, P.L. An adaptive ensemble Kalman filter. Mon. Weather Rev.
**2000**, 128, 416–433. [Google Scholar] [CrossRef] - Houtekamer, P.L.; Mitchell, H.L. A sequential ensemble Kalman filter for atmospheric data assimilation. Mon. Weather Rev.
**2001**, 129, 123–137. [Google Scholar] [CrossRef] - Diaconescu, E.P.; Laprise, R. Singular vectors in atmospheric sciences: A review. Earth-Sci. Rev.
**2012**, 113, 161–175. [Google Scholar] [CrossRef] - Kim, H.M.; Jung, B.J. Singular vector structure and evolution of a recurving tropical cyclone. Mon. Weather Rev.
**2009**, 137, 505–524. [Google Scholar] [CrossRef] - Toth, Z.; Kalnay, E. Ensemble forecasting at NCEP and the breeding method. Mon. Weather Rev.
**1997**, 125, 3297–3319. [Google Scholar] [CrossRef] - Magnusson, L.; Leutbecher, M.; Källén, E. Comparison between singular vectors and breeding vectors as initial perturbations for the ECMWF ensemble prediction system. Mon. Weather Rev.
**2008**, 136, 4092–4104. [Google Scholar] [CrossRef] - Evensen, G. The ensemble Kalman filter: Theoretical formulation and practical implementation. Ocean Dyn.
**2003**, 53, 343–367. [Google Scholar] [CrossRef] - Houtekamer, P.L.; Zhang, F. Review of the ensemble Kalman filter for atmospheric data assimilation. Mon. Weather Rev.
**2016**, 144, 4489–4532. [Google Scholar] [CrossRef] - Ott, E.; Hunt, B.; Szunyogh, I.; Zimin, A.; Kostelich, E.; Corazza, M.; Kalnay, E.; Patil, D.; Yorke, J. A local ensemble Kalman filter for atmospheric data assimilation. Tellus A
**2004**, 56, 415–428. [Google Scholar] [CrossRef] - Meng, Z.; Zhang, F. Limited-area ensemble-based data assimilation. Mon. Weather Rev.
**2011**, 139, 2025–2045. [Google Scholar] [CrossRef] - Buizza, R.; Miller, M.; Palmer, T.N. Stochastic representation of model uncertainties in the ECMWF ensemble prediction system. Q. J. R. Meteorol. Soc.
**1999**, 125, 2887–2908. [Google Scholar] [CrossRef] - Palmer, T.; Buizza, R.; Doblas-Reyes, F.; Jung, T.; Leutbecher, M.; Shutts, G.; Steinheimer, M.; Weisheimer, A. Stochastic Parametrization and Model Uncertainty; Technical Report 598, ECMWF Technical Memoranda; ECMWF: Reading, UK, 2009. [Google Scholar]
- Shutts, G. A kinetic energy backscatter algorithm for use in ensemble prediction systems. Q. J. R. Meteorol. Soc.
**2005**, 131, 3079–3102. [Google Scholar] [CrossRef] - Möhrlen, C. Uncertainty in Wind Energy Forecasting. Ph.D. Thesis, National University of Ireland, Cork, Ireland, 2004. [Google Scholar]
- Möhrlen, C.; Jørgensen, J. chapter The role of ensemble forecasting in integrating renewables into power systems: From theory to real-time applications. In Integration of Large-Scale Renewable Energy into Bulk Power Systems. From Planning to Operation; Springer Nature; Springer International Publishing AG: Cham, Switzerland, 2008; pp. 79–134. [Google Scholar]
- Buizza, R.; Houtekamer, P.L.; Pellerin, G.; Toth, Z.; Zhu, Y.; Wei, M. A comparison of the ECMWF, MSC, and NCEP Global Ensemble Prediction Systems. Mon. Weather Rev.
**2005**, 133, 1076–1097. [Google Scholar] [CrossRef] - Stensrud, D.J.; Bao, J.W.; Warner, T.T. Using initial condition and model physics perturbations in short-range ensemble simulations of mesoscale convective systems. Mon. Weather Rev.
**2000**, 128, 2077–2107. [Google Scholar] [CrossRef] - Lavaysse, C.; Carrera, M.; Bélair, S.; Gagnon, N.; Frenette, R.; Charron, M.; Yau, M.K. Impact of surface parameter uncertainties within the Canadian regional ensemble prediction system. Mon. Weather Rev.
**2013**, 141, 1506–1526. [Google Scholar] [CrossRef] - Deng, G.; Zhu, Y.; Gong, J.; Chen, D.; Wobus, R.; Zhang, Z. The effects of land surface process perturbations in a global ensemble forecast system. Adv. Atmos. Sci.
**2016**, 33, 1199–1208. [Google Scholar] [CrossRef] - Emmrich, P.B.K. A gratis two-model-ensemble versus EPS. In Fifth Workshop on Meteorological Operational Systems; ECMWF: Reading, UK, 1995. [Google Scholar]
- Krishnamurti, T.; Kishtawal, C.; Zhan, Z.; Larow, T.; Bachiochi, D.; Williford, E.; Gadgill, S.; Surendran, S. Multimodel ensemble forecasts for weather and seasonal climate. J. Clim.
**2000**, 13, 4196–4216. [Google Scholar] [CrossRef] - Hamill, T.M.; Whitaker, J.S.; Wei, X. Ensemble reforecasting: Improving medium-range forecast skill using retrospective forecasts. Mon. Weather Rev.
**2004**, 132, 1434–1447. [Google Scholar] [CrossRef] - Wilks, D.S.; Hamill, T.M. Comparison of ensemble-MOS methods using GFS reforecasts. Mon. Weather Rev.
**2007**, 135, 2379–2390. [Google Scholar] [CrossRef] - Hagedorn, R.; Hamill, T.M.; Whitaker, J.S. Probabilistic forecast calibration using ECMWF and GFS ensemble reforecasts. Part I: Two-meter temperatures. Mon. Weather Rev.
**2008**, 136, 2608–2619. [Google Scholar] [CrossRef] - Hamill, T.M.; Bates, G.T.; Whitaker, J.S.; Murray, D.R.; Fiorino, M.; Galarneau, T.J., Jr.; Zhu, Y.; Lapenta, W. NOAA’s second-generation global medium-range ensemble reforecast dataset. Bull. Am. Meteorol. Soc.
**2013**, 94, 1553–1565. [Google Scholar] [CrossRef] - Glahn, H.R.; Lowry, D.A. The use of model output statistics (MOS) in objective weather forecasting. J. Appl. Meteorol.
**1972**, 11, 1203–1211. [Google Scholar] [CrossRef] - Wilks, D.S. Statistical Methods in the Atmospheric Sciences: An Introduction, 1st ed.; International Geophysics; Academic Press: Cambridge, MA, USA, 1995; Volume 59. [Google Scholar]
- Wilks, D.S. Extending logistic regression to provide full-probability-distribution MOS forecasts. Meteorol. Appl.
**2009**, 16, 361–368. [Google Scholar] [CrossRef] - Monache, L.D.; Nipen, T.; Liu, Y.; Roux, G.; Stull, R. Kalman Filter and Analog Schemes to Postprocess Numerical Weather Predictions. Mon. Weather Rev.
**2011**, 139, 3554–3570. [Google Scholar] [CrossRef] - Eckel, F.A.; Monache, L.D. A hybrid NWP-analog ensemble. Mon. Weather Rev.
**2016**, 144, 897–911. [Google Scholar] [CrossRef] - Gneiting, T.; Raftery, A.E.; Westveld, A.H., III; Goldman, T. Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation. Mon. Weather Rev.
**2005**, 133, 1098–1118. [Google Scholar] [CrossRef] - Raftery, A.E.; Gneiting, T.; Balabdaoui, F.; Polakowski, M. Using bayesian model averaging to calibrate forecast ensembles. Mon. Weather Rev.
**2005**, 133, 1155–1174. [Google Scholar] [CrossRef] - Schuhen, N.; Thorarinsdottir, T.L.; Gneiting, T. Ensemble model output statistics for wind vectors. Mon. Weather Rev.
**2012**, 140, 3204–3219. [Google Scholar] [CrossRef] - Schmeits, M.J.; Kok, K.J. A Comparison between Raw Ensemble Output, (Modified) Bayesian Model Averaging, and Extended Logistic Regression Using ECMWF Ensemble Precipitation Reforecasts. Mon. Weather Rev.
**2010**, 138, 4199–4211. [Google Scholar] [CrossRef] - Scheuerer, M.; MMöller, D. Probabilistic wind speed forecasting on a grid based on ensemble model output statistics. Ann. Appl. Stat.
**2015**, 9, 1328–1349. [Google Scholar] [CrossRef] - Schefzik, R.; Thorarinsdottir, T.L.; Gneiting, T. Uncertainty quantification in complex simulation models using ensemble copula coupling. Stat. Sci.
**2013**, 28, 616–640. [Google Scholar] [CrossRef] - Wilks, D.S. Multivariate ensemble Model Output Statistics using empirical copulas. Q. J. R. Meteorol. Soc.
**2015**, 141, 945–952. [Google Scholar] [CrossRef] - Ben Bouallègue, Z.; Heppelmann, T.; Theis, S.E.; Pinson, P. Generation of scenarios from calibrated ensemble forecasts with a dual-ensemble copula-coupling approach. Mon. Weather Rev.
**2016**, 144, 4737–4750. [Google Scholar] [CrossRef] - Messner, J.W.; Zeileis, A.; Broecker, J.; Mayr, G.J. Probabilistic wind power forecasts with an inverse power curve transformation and censored regression. Wind Energy
**2013**, 17, 1753–1766. [Google Scholar] [CrossRef] - Pinson, P. Very short-term probabilistic forecasting of wind power with generalized logit-Normal distributions. J. R. Stat. Soc. Ser. C
**2012**, 61, 555–576. [Google Scholar] [CrossRef] - Koenker, R.; Leorato, S.; Peracchi, F. Distributional vs. Quantile Regression; CEIS Working Paper No. 300; CEIS: Glasgow, UK, 2013. [Google Scholar]
- Foresi, S.; Peracchi, F. The conditional distribution of excess returns: An empirical analysis. J. Am. Stat. Assoc.
**1995**, 90, 451–466. [Google Scholar] [CrossRef] - Ziel, F.; Croonenbroeck, C.; Ambach, D. Forecasting wind power—Modeling periodic and non-linear effects under conditional heteroscedasticity. Appl. Energy
**2016**, 177, 285–297. [Google Scholar] [CrossRef] - Dowell, J.; Dinwoodie, I.; McMillan, D. Forecasting for offshore maintenance scheduling under uncertainty. In Proceedings of the European Safety and Reliability Conference, ESRA, Glasgow, UK, 25–29 September 2016. [Google Scholar]
- Koenker, R. Encyclopedia of Environmetrics; John Wiley & Sons, Ltd.: Hoboken, NJ, USA, 2014; chapter Quantile regression. [Google Scholar]
- Landry, M.; Erlinger, T.P.; Patschke, D.; Varrichio, C. Probabilistic gradient boosting machines for GEFCom2014 wind forecasting. Int. J. Forecast.
**2016**, 32, 1061–1066. [Google Scholar] [CrossRef] - Hong, T.; Pinson, P.; Fan, S.; Zareipour, H.; Troccoli, A.; Hyndman, R.J. Probabilistic energy forecasting: Global Energy Forecasting Competition 2014 and beyond. Int. J. Forecast.
**2016**, in press. [Google Scholar] [CrossRef] [Green Version] - Andrade, J.; Bessa, R. Improving renewable energy forecasting with a grid of numerical weather predictions. IEEE Trans. Sustain. Energy
**2017**, in press. [Google Scholar] [CrossRef] - Anastasiades, G.; McSharry, P. Quantile forecasting of wind power using variability indices. Energies
**2013**, 6, 662–695. [Google Scholar] [CrossRef] - Cavalcante, L.; Bessa, R.J.; Reis, M.; Dowell, J. LASSO vector autoregression structures for very short-term wind power forecasting. Wind Energy
**2017**, 20, 657–675. [Google Scholar] [CrossRef] - Dowell, J.; Pinson, P. Very-short-term probabilistic wind power forecasts by sparse vector autoregression. IEEE Trans. Smart Grid
**2016**, 7, 763–770. [Google Scholar] [CrossRef] - Sperati, S.; Alessandrini, S.; Monache, L.D. Gridded probabilistic weather forecasts with an analog ensemble. Q. J. R. Meteorol. Soc.
**2017**, in press. [Google Scholar] [CrossRef] - Pinson, P.; Madsen, H. Ensemble-based probabilistic forecasting at Horns Rev. Wind Energy
**2009**, 12, 137–155. [Google Scholar] [CrossRef] [Green Version] - Papaefthymiou, G.; Pinson, P. Modeling of spatial dependence in wind power forecast uncertainty. In Proceedings of the 10th International Conference on Probabilistic Methods Applied to Power Systems (PMAPS 2008), Rincón, Puerto Rico, 25–29 May 2008; pp. 1–9. [Google Scholar]
- Møller, J.K.; Zugno, M.; Madsen, H. Probabilistic forecasts of wind power generation by stochastic differential equation models. J. Forecast.
**2016**, 35, 189–205. [Google Scholar] [CrossRef] [Green Version] - Staid, A.; Watson, J.P.; Wets, R.J.B.; Woodruff, D.L. Generating short-term probabilistic wind power scenarios via nonparametric forecast error density estimators. Wind Energy
**2017**, in press. [Google Scholar] [CrossRef] - Clark, M.; Gangopadhyay, S.; Hay, L.; Rajagopalan, B.; Wilby, R. The Schaake shuffle: A method for reconstructing space-time variability in forecasted precipitation and temperature fields. J. Hydrometeorol.
**2004**, 5, 243–262. [Google Scholar] [CrossRef] - Tastu, J.; Pinson, P.; Madsen, H. Modeling and Stochastic Learning for Forecasting in High Dimensions. In Lecture Notes in Statistics; Chapter Space-Time Trajectories of Wind Power Generation: Parameterized Precision Matrices Under a Gaussian Copula Approach; Springer: Berlin, Germany, 2015; Volume 217, pp. 267–296. [Google Scholar]
- McLean, J. Equivalent Wind Power Curves; Technical Report, Deliverable 2.4 of the TradeWind Project; Garrad Hassan and Partners Ltd.: Glasgow, UK, 2008. [Google Scholar]
- Nielsen, H.A.; Madsen, H.; Nielsen, T.S.; Badger, J.; Giebel, G.; Landberg, L.; Sattler, K.; Feddersen, H. Wind power ensemble forecasting. In Proceedings of the 2004 Global Windpower Conference and Exhibition, London, UK, 22–25 November 2004. [Google Scholar]
- Junk, C.; von Bremen, L.; Heinemann, D. Assessment of probabilistic wind power forecasts for wind farms in Northern Ireland. In Proceedings of the EWEA 2012 Annual Event, Copenhagen, Denmark, 16–19 April 2012. [Google Scholar]
- Pinson, P.; Kariniotakis, G. On-line assessment of prediction risk for wind power production forecasts. Wind Energy
**2004**, 7, 119–132. [Google Scholar] [CrossRef] - Giebel, G.; Badger, J.; Landberg, L.; Nielsen, H.A.; Nielsen, T.S.; Madsen, H.; Sattler, K.; Feddersen, H.; Vedel, H.; Tøfting, J.; et al. Wind Power Prediction Using Ensembles; Technical Report RisÃ¸-R-1527, RisÃ¸; RisÃ: Foothill Ranch, CA, USA, 2005. [Google Scholar]
- Bremnes, J.B. Probabilistic wind power forecasts using local quantile regression. Wind Energy
**2004**, 7, 47–54. [Google Scholar] [CrossRef] - Juban, J.; Fugon, L.; Kariniotakis, G. Uncertainty estimation of wind power forecasts. In Proceedings of the European Wind Energy Conference (EWEC 2008), Brussels, Belgium, 31 March–3 April 2008. [Google Scholar]
- Nielsen, T.; Landberg, L.; Giebel, G. Prediction of regional wind power. In Proceedings of the 2002 Global Wind Power Conference, EWEA, Paris, France, 10–12 March 2002. [Google Scholar]
- Siebert, N. Development of Methods for Regional Wind Power Forecasting. Ph.D. Thesis, PSL Research University, Centre for Process Energies and Energy Systems (PERSEE), Sophia Antipolis, France, 2008. [Google Scholar]
- Camal, S.; Michiorri, A.; Kariniotakis, G.; Liebelt, A. Short-term forecast of automatic frequency restoration reserve from a renewable energy based virtual power plant. In Proceedings of the 2017 IEEE PES ISGT Europe Conference, Torino, Italy, 26–29 September 2017. [Google Scholar]
- Pinson, P.; Kariniotakis, G. Conditional prediction intervals of wind power generation. IEEE Trans. Power Syst.
**2010**, 25, 1845–1856. [Google Scholar] [CrossRef] [Green Version] - Joslyn, S.L.; LeClerc, J.E. Uncertainty forecasts improve weather-related decisions and attenuate the effects of forecast error. J. Exp. Psychol. Appl.
**2012**, 18, 126–140. [Google Scholar] [CrossRef] [PubMed] - Council, N.R. Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts; The National Academies Press: Washington, DC, USA, 2006. [Google Scholar]
- Nadav-Greenberg, L.; Joslyn, S.L.; Taing, M.U. Uncertainty forecasts improve decision making among nonexperts. J. Cogn. Eng. Decis. Mak.
**2009**, 3, 209–227. [Google Scholar] [CrossRef] - Joslyn, S.; LeClerc, J. The cry wolf effect and weather-related decision making. Risk Anal.
**2015**, 35, 385–395. [Google Scholar] - Joslyn, S.; Savelli, S. Communicating forecast uncertainty: public perception of weather forecast uncertainty. Meteorol. Appl.
**2010**, 17, 180–195. [Google Scholar] [CrossRef] - LeClerc, J. Communicating Weather and Climate Uncertainty: Exploratory Research in Cognitive Psychology. Ph.D. Thesis, University of Washington, Seattle, WA, USA, 2014. [Google Scholar]
- Ramos, M.H.; van Andel, S.J.; Pappenberger, F. Do probabilistic forecasts lead to better decisions? Hydrol. Earth Syst. Sci.
**2013**, 17, 2219–2232. [Google Scholar] [CrossRef] - Joslyn, S.; Pak, K.; Jones, D.; Pyles, J.; Hunt, E. The effect of probabilistic information on threshold forecasts. Weather Forecast.
**2006**, 22, 804–812. [Google Scholar] [CrossRef] - Nadav-Greenberg, L.; Joslyn, S.L.; Taing, M.U. The effect of uncertainty visualizations on decision making in weather forecasting. J. Cogn. Eng. Decis. Mak.
**2008**, 2, 24–47. [Google Scholar] [CrossRef] - Joslyn, S.; Savelli, S. The advantages of predictive interval forecasts for non-expert users and the impact of visualizations. Appl. Cogn. Psychol.
**2013**, 27, 527–541. [Google Scholar] - Joslyn, S.L.; Nichols, R.M. Probability or frequency? Expressing forecast uncertainty in public weather forecasts. Meteorol. Appl.
**2009**, 16, 309–314. [Google Scholar] [CrossRef] - Morss, R.E.; Lazo, J.K.; Demuth, J.L. Examining the use of weather forecasts in decision scenarios: results from a US survey with implications for uncertainty communication. Meteorol. Appl.
**2010**, 17, 149–162. [Google Scholar] [CrossRef] - Zhang, Y.; Wang, J.; Wang, X. Review on probabilistic forecasting of wind power generation. Renew. Sustain. Energy Rev.
**2014**, 32, 255–270. [Google Scholar] [CrossRef] - Bessa, R.J. From marginal to simultaneous prediction intervals of wind power. In Proceedings of the 18th Intelligent Systems Applications to Power Systems Conference and Debate (ISAP 2015), Porto, Portugal, 14–16 September 2015. [Google Scholar]
- Rockafellar, R.T.; Wets, R.J.B. Scenarios and policy aggregation in optimization under uncertainty. Math. Oper. Res.
**1991**, 6, 119–147. [Google Scholar] [CrossRef] - Raftery, A.E. Use and communication of probabilistic forecasts. Stat. Anal. Data Min. ASA Data Sci. J.
**2016**, 9, 397–410. [Google Scholar] [CrossRef] [PubMed] - Golestaneh, F.; Pinson, P.; Azizipanah-Abarghooee, R.; Gooi, H.B. Ellipsoidal prediction regions for multivariate uncertainty characterization. arXiv
**2017**, arXiv:1708.02703. [Google Scholar] - Pinson, P.; Nielsen, H.; Madsen, H.; Kariniotakis, G. Skill forecasting from ensemble predictions of wind power. Appl. Energy
**2009**, 86, 1326–1334. [Google Scholar] [CrossRef] [Green Version] - Holton, G.A. Defining risk. Financ. Anal. J.
**2004**, 60, 6. [Google Scholar] [CrossRef] - Nielsen, H.A.; Madsen, H.; Nielsen, T.S. Using quantile regression to extend an existing wind power forecasting system with probabilistic forecasts. Wind Energy
**2006**, 9, 95–108. [Google Scholar] [CrossRef] - Gallego-Castillo, C.; Cuerva-Tejero, A.; Lopez-Garcia, O. A review on the recent history of wind power ramp forecasting. Renew. Sustain. Energy Rev.
**2015**, 52, 1148–1157. [Google Scholar] [CrossRef] - McSharry, P.; von Bremen, L.; Holmgren, E.; Siebert, N.; Kariniotakis, G.; Girard, R. Predictability Measure and Communication of Information on Uncertainty to Forecast Users; Technical Report, EU Project SafeWind, Deliverable DP-6.6; European Union: Brussels, Belgium, 2012. [Google Scholar]
- Zhao, J.; Abedi, S.; He, M.; Du, P.; Sharma, S.; Blevins, B. Quantifying risk of wind power ramps in ERCOT. IEEE Trans. Power Syst.
**2017**, in press. [Google Scholar] [CrossRef] - Wang, J.; Botterud, A.; Bessa, R.; Keko, H.; Miranda, V.; Akilimali, J.; Carvalho, L.; Issicaba, D. Wind power forecasting uncertainty and unit commitment. Appl. Energy
**2011**, 88, 4014–4023. [Google Scholar] [CrossRef] - Da Silva, A.; de Resende, L.; da Fonseca Manso, L.; Billinton, R. Well-being analysis for composite generation and transmission systems. IEEE Trans. Power Syst.
**2004**, 19, 1763–1770. [Google Scholar] [CrossRef] - Bessa, R.J.; Matos, M.A.; Costa, I.C.; Bremermann, L.; Franchin, I.G.; Pestana, R.; Machado, N.; Waldl, H.P.; Wichmann, C. Reserve setting and steady-state security assessment using wind power uncertainty forecast: A case study. IEEE Trans. Sustain. Energy
**2012**, 3, 827–836. [Google Scholar] [CrossRef] - Menemenlis, N.; Huneault, M.; Robitaille, A. Computation of Dynamic Operating Balancing Reserve for Wind Power Integration for the Time-Horizon 1–48 Hours. IEEE Trans. Sustain. Energy
**2012**, 3, 692–702. [Google Scholar] [CrossRef] - Sari, D.; Lee, Y.; Ryan, S.; Woodruff, D. Statistical metrics for assessing the quality of wind power scenarios for stochastic unit commitment. Wind Energy
**2015**, 19, 873–893. [Google Scholar] [CrossRef] - Bessa, R.; Miranda, V.; Botterud, A.; Wang, J. Good or bad wind power forecasts: a relative concept. Wind Energy
**2011**, 14, 625–636. [Google Scholar] [CrossRef] - Gneiting, T.; Stanberry, L.; Grimit, E.; Held, L.; Johnson, N. Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds. TEST
**2008**, 17, 211–235. [Google Scholar] [CrossRef] - Scheuerer, M.; Hamill, T.M. Variogram-based proper scoring rules for probabilistic forecasts of multivariate quantities. Mon. Weather Rev.
**2015**, 143, 1321–1334. [Google Scholar] [CrossRef] - Bessa, R. On the quality of the gaussian copula for multi-temporal decision-making problems. In Proceedings of the 19th Power Systems Computation Conference (PSCC 2016), Genoa, Italy, 20–24 June 2016. [Google Scholar]
- Pinson, P.; Girard, R. Evaluating the quality of scenarios of short-term wind power generation. Appl. Energy
**2012**, 96, 12–20. [Google Scholar] [CrossRef] [Green Version] - Botterud, A.; Wang, J.; Valenzuela, J.; Bessa, R.; Keko, H.; Miranda, V.; Akilimali, J. Unit commitment and operating reserves with probabilistic wind power forecasts. In Proceedings of the IEEE PowerTech Conference, Trondheim, Norway, 19–23 June 2011. [Google Scholar]
- Möhrlen, C.; Jorgensen, J.U. Reserve forecasting for enhanced renewable energy management. In Proceedings of the 12th International Workshop on Large-Scale Integration of Wind Power into Power Systems as well as on Transmission Networks for Offshore Wind Farms, Energynautics, Berlin, Germany, 11–13 November 2014. [Google Scholar]
- Botterud, A.; Wang, J.; Zhou, Z.; Bessa, R.; Keko, H.; Akilimali, J.; Miranda, V. Wind power trading under uncertainty in LMP markets. IEEE Trans. Power Syst.
**2012**, 27, 894–903. [Google Scholar] [CrossRef] - Rodriguez, A. Wind generation forecasting at REE. In Proceedings of the Workshop on Experiences in using Wind Power Predictions and Gaps in Forecasting Research, Barcelona, Spain, 6 June 2016. [Google Scholar]
- Gogolou, E. Significance of the forecast in the short term power trading. In Proceedings of the Workshop on Experiences in using Wind Power Predictions and Gaps in Forecasting Research, Barcelona, Spain, 6 June 2016. [Google Scholar]
- Pahlow, M.; Möhrlen, C.; Jørgensen, J. Chapter Application of cost functions for large-scale integration of wind power using a multi-scheme ensemble prediction technique. In Optimization Advances in Electric Power Systems; Nova Science Publishers: Hauppauge, NY, USA, 2009; pp. 151–180. [Google Scholar]
- Dent, C.J.; Bialek, J.W.; Hobbs, B.F. Opportunity cost bidding by wind generators in forward markets: Analytical results. IEEE Trans. Power Syst.
**2011**, 26, 1600–1608. [Google Scholar] [CrossRef] [Green Version] - Möhrlen, C.; Pahlow, M.; Jørgensen, J.U. Untersuchung verschiedener Handelsstrategien für Wind- und Solarenergie unter Berücksichtigung der EEG 2012 Novellierung. Z. Energiewirtschaft
**2012**, 36, 9–25. [Google Scholar] [CrossRef] - Elia, Windvision, Enercon, and Eneco Delivery of Downward aFRR by Wind Farms (2014–2015). Available online: http://www.elia.be/~/media/files/Elia/users-group/task-force-balancing/Downward_aFRR_windfarms_EN_2015.pdf (accessed 17 August 2017).
- Knorr, K.; Zimmermann, B.; Kirchner, D.; Steinke, F.; Wolfrum, P.; Leveringhaus, T.; Lager, T.; Filzek, D.; Kusserow, B.; Nicklaus, L.; Ritter, P. Kombikraftwerk 2. Final Report. Technical Report, Kombikraftwerk 2 Consortium. 2014. Available online: http://www.kombikraftwerk.de/fileadmin/Kombikraftwerk_2/English/Kombikraftwerk2_FinalReport.pdf (accessed 17 August 2017).
- Jansen, M.; Speckmann, M. Wind turbine participation on control reserve markets. In Proceedings of the European Wind Energy Conference and Exhibition (EWEC 2013), Vienna, Austria, 4–7 February 2013; pp. 1432–1439. [Google Scholar]
- Castronuovo, E.; Usaola, J.; Bessa, R.; Matos, M.; Costa, I.; Bremermann, L.; Lugaro, J.; Kariniotakis, G. An integrated approach for optimal coordination between wind power and hydro pumping storage. Wind Energy
**2014**, 17, 829–852. [Google Scholar] [CrossRef] [Green Version] - Haessig, P.; Multon, B.; Ahmed, H.B.; Lascaud, S.; Bondon, P. Energy storage sizing for wind power: Impact of the autocorrelation of day-ahead forecast errors. Wind Energy
**2015**, 18, 43–57. [Google Scholar] [CrossRef] [Green Version] - Hayes, B.P.; Prodanovic, M. State forecasting and operational planning for distribution network energy management systems. IEEE Trans. Smart Grid
**2016**, 7, 1002–1011. [Google Scholar] [CrossRef] - Capitanescu, F. Critical review of recent advances and further developments needed in AC optimal power flow. Electr. Power Syst. Res.
**2016**, 136, 57–68. [Google Scholar] [CrossRef] - Soares, T.; Bessa, R.; Pinson, P.; Morais, H. Active distribution grid management based on robust AC optimal power flow. IEEE Trans. Smart Grid
**2017**, in press. [Google Scholar] [CrossRef] - European Union. Report on Uncertainty Modeling; Technical Report, EU Project UMBRELLA, Deliverable D2.1; European Union: Brussels, Belgium, 2013. [Google Scholar]
- Antoniadou, I.; Dervilis, N.; Maguire, E.P.A.; Worden, K. Aspects of structural health and condition monitoring of offshore wind turbines. Philos. Trans. R. Soc. Lond. A Math. Phys. Eng. Sci.
**2015**, 373, 1–14. [Google Scholar] [CrossRef] [PubMed] - Catterson, V.M.; McMillan, D.; Dinwoodie, I.; Revie, M.; Dowell, J.; Quigley, J.; Wilson, K. An economic impact metric for evaluating wave height forecasters for offshore wind maintenance access. Wind Energy
**2016**, 19, 199–212. [Google Scholar] [CrossRef] [Green Version] - Besnard, F.; Patriksson, M.; Stromberg, A.; Wojciechowski, A.; Fischer, K.; Bertling, L. A stochastic model for opportunistic maintenance planning of offshore wind farms. In Proceedings of the 2011 IEEE Trondheim PowerTech, Trondheim, Norway, 19–23 June 2011. [Google Scholar]
- MacLeod, D.; Davis, M.; Doblas-Reyes, P. Modelling Wind Energy Generation Potential on Seasonal Timescales With Impact Surfaces; Technical Report, SPECS Technical Note 3; SPECS: Berlin, Germany, 2014. [Google Scholar]
- Schmidt, J.; Cancella, R.; Junior, A.O.P. Combing Windpower and Hydropower to Decrease Seasonal and Inter-Annual Availability of Renewable Energy Sources in Brazil; Universität für Bodenkultur Wien: Wien, Austria, 2014. [Google Scholar]
- Torralba, V.; Doblas-Reyes, F.J.; MacLeod, D.; Christel, I.; Davis, M. Seasonal climate prediction: A new source of information for the management of wind energy resources. J. Appl. Meteorol. Climatol.
**2017**, 56, 1231–1247. [Google Scholar] [CrossRef] - Sen, P.; Singer, J. Large Sample Methods in Statistics: An Introduction with Applications; CRC Press: Boca Raton, FL, USA, 1994. [Google Scholar]
- Pircalabu, A.; Hvolby, T.; Jung, J.; Høg, E. Joint price and volumetric risk in wind power trading: A copula approach. Energy Econ.
**2017**, 62, 139–154. [Google Scholar] [CrossRef] - Clemen, R.T.; Reilly, T. Correlations and copulas for decision and risk analysis. Manag. Sci.
**1999**, 45, 208–224. [Google Scholar] [CrossRef] - Jose, V.R.R.; Winkler, R.L. Evaluating quantile assessments. Oper. Res.
**2009**, 57, 1287–1297. [Google Scholar] [CrossRef] - Keeney, R.L. Common mistakes in making value trade-offs. Oper. Res.
**2002**, 50, 935–945. [Google Scholar] [CrossRef] - Weber, E.U.; Johnson, E.J. Chapter Decisions Under Uncertainty: Psychological, Economic, and Neuroeconomic Explanations of Risk Preference. In Neuroeconomics: Decision Making and the Brain; Academic Press: London, UK, 2008; pp. 127–144. [Google Scholar]
- Rodrigo, J.S.; Paredes, L.F.; Stoffels, N.; von Bremen, L. Wind power predictability assessment from large to local scale. In Proceedings of the European Wind Energy Conference (EWEA 2013), Wien, Austria, 4–7 February 2013. [Google Scholar]
- Kariniotakis, G.; Marti, I.; Casas, D.; Pinson, P. What performance can be expected by short-term wind power prediction models depending on site characteristics? In Proceedings of the European Wind Energy Conference (EWEC 2004) EWEA, London, UK, 22–25 November 2004. [Google Scholar]
- Laquaine, K.; Girard, R.; Kariniotakis, G. Assessment of wind power predictability as a decision factor in the investment phase of wind farms. Appl. Energy
**2012**, 101, 609–617. [Google Scholar] - Uslar, M.; Specht, M.; Dänekas, C.; Trefke, J.; Rohjans, S.; González, J.M.; Rosinger, C.; Bleiker, R. Standardization in Smart Grids. Introduction to IT-Related Methodologies, Architectures and Standards; Springer: Berlin/Heidelberg, Germany, 2013. [Google Scholar]
- Steffan, N.; Du, P.; Mago, N.; Sharma, S. Recent ERCOT developments in applications of uncertainty forecasts to system operations. In Proceedings of the 2017 UVIG Forecasting Workshop, Tutorial on Integration of Uncertainty Forecasts into Power System Operations, Atlanta, GA, USA, 20–22 June 2017. [Google Scholar]
- Siblot, S.; Bendaoud, A. Wind power forecasting in a French context. Meteorological feedback. In Proceedings of the IEA Wind Task 36 Forecasting: Workshop on Experiences and Research Gaps, Barcelona, Spain, 9 June 2016. [Google Scholar]
- DREAMS: Distributed Resource Energy Analysis and Management System Project (2014–2016). Available online: https://energy.gov/eere/sunshot/distributed-resource-energy-analysis-and-management-system-dreams-development-real-time (accessed on 17 August 2017).
- ANEMOS.plus: Advanced Tools for the Management of Electricity Grids with Large-Scale Wind Generation Project (2018–2011). Available online: https://www.anemos-plus.eu (accessed on 17 August 2017).
- EWeLiNE: Development of Innovative Weather and Power Forecast Models for the Grid Integration of Weather Dependent Energy Sources Project (2012–2017). Available online: http://www.projekt-eweline.de (accessed on 17 August 2017).
- InteGrid Project (2017–2020). Available online: http://integrid-h2020.eu (accessed on 17 August 2017).
- Wind Forecasting Improvement Project I (2011–2013). Available online: https://www.energy.gov/eere/wind/downloads/wind-forecast-improvement-project-wfip-publicprivate-partnership-improving-short (accessed on 17 August 2017).
- Wind Forecasting Improvement Project II (2015–2017). Available online: http://www.esrl.noaa.gov/gsd/renewable/wfip2.html (accessed on 17 August 2017).
- Solar Forecasting Improvement Projects (2012–2018). Available online: https://energy.gov/eere/sunshot/solar-forecast-improvement-project (accessed on 17 August 2017).
- US Department of Energy SunShot Initiative (2011–2020). Available online: https://energy.gov/eere/sunshot/about-sunshot-initiative (accessed on 17 August 2017).
- Kariniotakis, G.; Matos, M.; Miranda, V. Assessment of the benefits from advanced load & wind power forecasting in autonomous power systems. In Proceedings of the 1999 European Wind Energy Conference, Wind Energy for the Next Millennium, Nice, France, 1–5 March 1999; Petersen, E., Ed.; pp. 391–394. [Google Scholar]
- EoCoE: Energy Oriented Centre of Excellence in Computing Applications Project (2015–2018). Available online: http://www.eocoe.eu (accessed on 17 August 2017).
- SPECS: Seasonal to Decadal Climate Prediction for the Improvement of European Climate Services Project SPECS (2012–2017). Available online: http://www.specs-fp7.eu (accessed on 17 August 2017).
- EUPORIAS: European Provision of Regional Impacts Assessments on Seasonal and Decadal Timescales project EUPORIAS (2012–2017). Available online: http://www.euporias.eu (accessed on 17 August 2017).
- MacLeod, D.; Torralba, V.; Davis, M.; Doblas-Reyes, F. Transforming climate model output to forecasts of wind power production: How much resolution is enough? Meteorol. Appl.
**2017**, in press. [Google Scholar] [CrossRef] - Pantel, M.; Mancarella, P. Influence of extreme weather and climate change on the resilience of power systems: Impacts and possible mitigation strategies. Electr. Power Syst. Res.
**2015**, 127, 259–270. [Google Scholar] [CrossRef]

**Figure 1.**Example spaghetti plot of 75 normalized wind power forecasts converted from wind speed forecasts at 10 m from a multi-scheme ensemble prediction system (MSEPS) from weather ensemble service provider WEPROG at a near-coastal wind farm in the North of Spain.

**Figure 2.**Same plot as Figure 1, in this case with 300 normalized wind power from wind speeds at 10 m, 40 m, 100 m and 150 m, illustrating that an entire wind profile may be required to capture the uncertainty of the power generation.

**Figure 3.**Same as Figure 1, in this case with 75 normalized wind power from wind speeds 40 m, illustrating that in this specific case the effective wind speed at the turbines, where the MSEPS captured the entire uncertainty was at that model level.

**Figure 4.**Illustration of links between input data statistical methodologies and forecast types. Dashed lines indicate that methods relate to individual members of ensemble NWP which require calibration.

**Figure 8.**Seventy-five wind power forecasts from 75 ensemble members at 4 different model levels at 100 m for a 48 h-ahead time horizon.

Method | Type | Member Differences | Perturbation Application | Number of Members | Member Differences | Expense (Technically) | Forecast Horizon | Target Time Horizon |
---|---|---|---|---|---|---|---|---|

Monte-Carlo method | statistical | Statistical perturbations | random “white noise” perturbations from climatology | large (≥800) | random and not unique | expensive, due to large number of members | any | to be defined, e.g., 72 h |

Breeding Vector method | statistical | Statistical perturbations from non-linearized Lyapunov (Bred) vectors | differences generated from perturbation of initial conditions (analysis) | limited (typically 10–50) | can be defined statistically, but are not unique | reasonable, because only 1 NWP model is required | medium range (3-10 days) | to be defined, e.g., 96 h |

Singular vector method | statistical | Statistical perturbations from linearized equations— singular vectors | differences are generated from perturbation of the initial conditions (analysis) | limited (typically ca. 50) | can be defined statistically, but are not unique | reasonable, because only 1 NWP model is required | medium range (3–10 days) | to be defined, e.g., 96 h |

Ensemble Kalman Filter | statistical | Random statistical perturbations of the initial conditions (analysis) | differences from filtered “white noise”—basic concept from Monte Carlo Simulations | unlimited (typically 15–25) | all differences are random and not unique | low, pure statistical approach | any | to be defined, e.g., 96 h |

Stochastic Physics approach | physical | sub-grid physics parameterizations | parameterization differences | limited only to available parameters | yes, well-defined parameter changes | low, mostly used in combination with method 1–4 | short-range (0–3 days) | every time step |

Multi-Scheme approach | physical | computation of different processes inside one NWP model kernel | use of different physical and/or dynamic parameterization schemes inside NWP model | only limited to schemes & combinations (typically 25–100) | yes, well-defined by difference in parameterization schemes | manageable, maintenance is limited to 1 full NWP model | short-range and medium range (0–10 days) | every time step |

Surface perturbations | physical | well- defined | roughness length, surface albedo, snow cover, soil moisture, SST, veget. properties | limited to parameters to be perturbed | yes, well-defined parameter changes | low, mostly used in combination with method 1–4 | short-range (0–3 days) | every time step |

Multi-Model approach | physical | Every member is an individual NWP model | Models as well as initial conditions are different | unlimited | no, differences can have a technical or physical reason | large, maintenance of many models is required | short-range and medium range (0–10 days) | every time step |

Decision-Making Problem | Objectives | Uncertainty Product(s) | Performance Verification | Use of Information | End-User Requirements |
---|---|---|---|---|---|

Operating reserve (System Operator) | Setting the operating reserve requirements considering wind power variability and uncertainty. Decision strategies like setting an acceptable risk level or trade-off between economic issues and risk | (i) Conditional quantiles (ii) Conditional PDF (iii) Skill forecasts | Calibration, sharpness, quantile score and CRPS | (i) and (ii) Convolution of forecasted marginal distribution functions (iii) Mix between skill forecast and empirical rules to set reserve requirements | Sharp forecasts → low reserve requirements High calibration → Adequate estimation of tails quantiles leads to accurate risk quantification Resolution → dynamic reserve requirement according to power system operating conditions |

Unit commitment (UC) and economic dispatch (ED) (System Operator) | UC: scheduling the generation units for minimizing the cost of supplying the load with a set of operation constraints. ED: for UC result, computes the generation levels of each unit with min cost | (i) Conditional quantiles (ii) Temporal or spatial-temporal ensembles | (i) Calibration, sharpness, quantile score (ii) p-variogram score, mass transportation distance | (i) Quantile-based reserve rule; (ii) Integrate ensembles in stochastic optimization | High calibration to achieve cost savings; Too sharp ensembles do not adequately describe the uncertainty; High event-detection skill (ramp-up, ramp-down); Temporal and spatial dependence structure of forecast uncertainty |

Market bidding (Market Player) | Forecast/bid optimization to minimize imbalance costs/maximize returns; ability to participate as price maker | Conditional quantiles/PDF | Calibration, sharpness, quantile score and CRPS | (i) Maximize expected profit (optimal quantile) (ii) Risk-expected profit trade-off | Perfect calibration since optimal bid corresponds to a quantile value; Low sharpness → low uncertainty (reduced imbalance cost) |

Virtual power plant operation (Market Player) | Provision of reserve capacity; Coordination of multiple generation sources and storage units | (i) Conditional quantiles (ii) Temporal or spatial-temporal ensembles (e.g., if storage available) | (i) Calibration, sharpness, quantile score and CRPS (ii) p-variogram score, mass transportation distance, energy score | (i) Choose quantile according to system operator’s reliability requirements (ii) Stochastic multi-period optimization | (i) Perfect calibration → perfect reserve reliability estimation; high sharpness → high reserve margin (ii) temporal or spatial dependence structure of forecast uncertainty |

Predictive grid management (System Operator) | Solve and detect technical constraints violations in transmission and/or distribution grids | Spatial-temporal ensembles | p-variogram score, mass transportation distance, energy score | Stochastic optimal power flow | Spatial-temporal dependence structure; Perfect calibration → avoid false and overlooked alarms; High sharpness → low operating costs; low computational requirements |

Maintenance scheduling (Wind Farm Operator/ Market Player) | Find access windows for safe wind power plant maintenance | Simultaneous forecast intervals; ensembles | Calibration and sharpness | Cost-loss model under uncertainty | Temporal dependence structure for low wind speed periods, co-dependence with wave height in offshore environment; Perfect calibration → accurate financial risk estimation |

Long-term portfolio planning (Market Players, System Operator, Wind Farm Operator) | Portfolio management and find the optimal size of the investment; Mid-term O&M planning; Investment in storage capacity, generation and network expansions | Wind speed PDF (Gaussian), probabilistic multi-category events for 10m height | Calibration/rank and sharpness diagrams, CRPS | Business cases are still needed, but tools like stochastic UC/ED can be used for long-term analysis | Calibration is a critical requirement from the end-user perspective (trustworthiness) |

© 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Bessa, R.J.; Möhrlen, C.; Fundel, V.; Siefert, M.; Browell, J.; Haglund El Gaidi, S.; Hodge, B.-M.; Cali, U.; Kariniotakis, G.
Towards Improved Understanding of the Applicability of Uncertainty Forecasts in the Electric Power Industry. *Energies* **2017**, *10*, 1402.
https://doi.org/10.3390/en10091402

**AMA Style**

Bessa RJ, Möhrlen C, Fundel V, Siefert M, Browell J, Haglund El Gaidi S, Hodge B-M, Cali U, Kariniotakis G.
Towards Improved Understanding of the Applicability of Uncertainty Forecasts in the Electric Power Industry. *Energies*. 2017; 10(9):1402.
https://doi.org/10.3390/en10091402

**Chicago/Turabian Style**

Bessa, Ricardo J., Corinna Möhrlen, Vanessa Fundel, Malte Siefert, Jethro Browell, Sebastian Haglund El Gaidi, Bri-Mathias Hodge, Umit Cali, and George Kariniotakis.
2017. "Towards Improved Understanding of the Applicability of Uncertainty Forecasts in the Electric Power Industry" *Energies* 10, no. 9: 1402.
https://doi.org/10.3390/en10091402