# Assessing Goodness of Fit for Verifying Probabilistic Forecasts

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Goodness of Fit Metric

#### 2.1. Dissociation

#### 2.2. Goodness of Fit

#### 2.3. Skill Score

## 3. Goodness of Fit Test

## 4. Application to IRI ENSO Predictions

## 5. Discussion and Conclusions

## Author Contributions

## Funding

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## References

- Sharma, A.; Mehrotra, R.; Li, J.; Jha, S. A programming tool for nonparametric system prediction using Partial Informational Correlation and Partial Weights. Environ. Model. Software
**2016**, 83, 271–275. [Google Scholar] [CrossRef] - Sharma, A.; Luk, K.C.; Cordery, I.; Lall, U. Seasonal to interannual rainfall probabilistic forecasts for improved water supply management: Part 2—Predictor identification of quarterly rainfall using ocean-atmosphere information. J. Hydrol.
**2000**, 239, 240–248. [Google Scholar] [CrossRef] - Gneiting, T.; Raftery, A.E. Weather forecasting with ensemble methods. Science
**2005**, 310, 248–249. [Google Scholar] [CrossRef] [Green Version] - Gneiting, T.; Raftery, A.E. Strictly proper scoring rules, prediction, and estimation. J. Am. Stat. Assoc.
**2007**, 102, 359–378. [Google Scholar] [CrossRef] - Palmer, T. The economic value of ensemble forecasts as a tool for risk assessment: From days to decades. Q. J. R. Meteorol. Soc.
**2002**, 128, 747–774. [Google Scholar] [CrossRef] - Sankarasubramanian, A.; Lall, U.; Souza Filho, F.A.; Sharma, A. Improved water allocation utilizing probabilistic climate forecasts: Short-term water contracts in a risk management framework. Water Resour. Res.
**2009**, 45. [Google Scholar] [CrossRef] [Green Version] - Li, L.; Xia, J.; Xu, C.Y.; Singh, V.P. Evaluation of the subjective factors of the GLUE method and comparison with the formal Bayesian method in uncertainty assessment of hydrological models. J. Hydrol.
**2010**, 390, 210–221. [Google Scholar] [CrossRef] - Krzysztofowicz, R. Bayesian theory of probabilistic forecasting via deterministic hydrologic model. Water Resour. Res.
**1999**, 35, 2739–2750. [Google Scholar] [CrossRef] [Green Version] - Todini, E. Predictive uncertainty assessment in real time flood forecasting. In Uncertainties in Environmental Modelling and Consequences for Policy Making; Baveye, P.C., Laba, M., Mysiak, J., Eds.; Springer: Berlin/Heidelberg, Germany, 2009; pp. 205–228. [Google Scholar]
- Bradley, A.A.; Schwartz, S.S.; Hashino, T. Sampling uncertainty and confidence intervals for the brier score and brier skill score. Weather Forecast.
**2008**, 23, 992–1006. [Google Scholar] [CrossRef] - Welles, E.; Sorooshian, S.; Carter, G.; Olsen, B. Hydrologic verification: A call for action and collaboration. Bull. Am. Meteorol. Soc.
**2007**, 88, 503–511. [Google Scholar] [CrossRef] - Siegert, S. Variance estimation for Brier Score decomposition. Q. J. R. Meteorol. Soc.
**2014**, 140, 1771–1777. [Google Scholar] [CrossRef] [Green Version] - Brier, G.W. Verification of forecasts expressed in terms of probability. Mon. Weather Rev.
**1950**, 78, 1–3. [Google Scholar] [CrossRef] - Hersbach, H. Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather Forecast.
**2000**, 15, 559–570. [Google Scholar] [CrossRef] - Epstein, E.S. A scoring system for probability forecasts of ranked categories. J. Appl. Meteorol.
**1969**, 8, 985–987. [Google Scholar] [CrossRef] [Green Version] - Brown, T.A. Admissible Scoring Systems for Continuous Distributions (Report P-5235); The Rand Corporation: Santa Monica, CA, USA, 1974. [Google Scholar]
- Matheson, J.E.; Winkler, R.L. Scoring rules for continuous probability distributions. Manag. Sci.
**1976**, 22, 1087–1096. [Google Scholar] [CrossRef] - Murphy, A.H. A new vector partition of the probability score. J. Appl. Meteorol.
**1973**, 12, 595–600. [Google Scholar] [CrossRef] [Green Version] - Wilks, D.S. Statistical Methods in the Atmospheric Sciences; Academic Press: Cambridge, MA, USA, 1995; p. 262. [Google Scholar]
- Jolliffe, I.T.; Stephenson, D.B. Forecast Verification: A Practitioner’s Guide in Atmospheric Science; John Wiley & Sons: Hoboken, NJ, USA, 2012. [Google Scholar]
- Nipen, T.; Stull, R. Calibrating probabilistic forecasts from an NWP ensemble. Dyn. Meteorol. Oceanogr.
**2011**, 63, 858–875. [Google Scholar] [CrossRef] - Anderson, J.L. A method for producing and evaluating probabilistic forecasts from ensemble model integrations. J. Clim.
**1996**, 9, 1518–1530. [Google Scholar] [CrossRef] - Wilks, D.S. Indices of rank histogram flatness and their sampling properties. Mon. Weather Rev.
**2019**, 147, 763–769. [Google Scholar] [CrossRef]

**Figure 1.**Synthetic example illustrating a hypothetical probabilistic prediction system (${F}_{t}^{F}$ is a predicted probability distribution from an existing system, ${F}_{t}^{F*}$ is from a new prediction system, and ${\tilde{x}}_{t}^{O}$ is after the dissociation by Equation (2)).

**Figure 2.**ENSO forecasts from various dynamical and statistical models and corresponding observations (thick black line).

**Figure 3.**Histogram of observations ${\tilde{x}}_{t}^{O}$ according to $m$ ranges (cross marks on the bottom of each histogram are the observations ${\tilde{x}}_{t}^{O}$ and the dotted line is the expected frequency from the predicted distribution ${\tilde{F}}^{F}$).

**Table 1.**Reliability distance ${R}_{d\left(5\right)}$ and skill score ${R}_{s\left(5\right)}$ with $m=5$ for IRI ENSO forecasts.

$\mathit{i}$ | Range | Frequency | $\mathbf{Relative}\text{}\mathbf{Frequency}\text{}{\mathit{O}}_{\mathit{i}}$ | ${\left({\mathit{m}}^{-1}-{\mathit{O}}_{\mathit{i}}\right)}^{2}$ |
---|---|---|---|---|

1 | 0.0–0.2 | 5 | 0.20 | 0 |

2 | 0.2–0.4 | 4 | 0.16 | 0.0016 |

3 | 0.4–0.6 | 12 | 0.48 | 0.0784 |

4 | 0.6–0.8 | 3 | 0.12 | 0.0064 |

5 | 0.8–1.0 | 1 | 0.04 | 0.0256 |

Sum | 25 | 1.00 | 0.1120 | |

${R}_{d\left(5\right)}$ | $\sqrt{5\xb70.112}=0.75$ | |||

${R}_{s\left(5\right)}$ | $1-\sqrt{{\left(1-{5}^{-1}\right)}^{-1}\xb70.112}=0.63$ |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Kang, T.-H.; Sharma, A.; Marshall, L.
Assessing Goodness of Fit for Verifying Probabilistic Forecasts. *Forecasting* **2021**, *3*, 763-773.
https://doi.org/10.3390/forecast3040047

**AMA Style**

Kang T-H, Sharma A, Marshall L.
Assessing Goodness of Fit for Verifying Probabilistic Forecasts. *Forecasting*. 2021; 3(4):763-773.
https://doi.org/10.3390/forecast3040047

**Chicago/Turabian Style**

Kang, Tae-Ho, Ashish Sharma, and Lucy Marshall.
2021. "Assessing Goodness of Fit for Verifying Probabilistic Forecasts" *Forecasting* 3, no. 4: 763-773.
https://doi.org/10.3390/forecast3040047