# Probabilistic Forecasts: Scoring Rules and Their Decomposition and Diagrammatic Representation via Bregman Divergences

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

^{2}= 0.09 if rain is subsequently observed and (0 − p)

^{2}= 0.49 if not. The logarithmic score (an early discussion is given in [5]) is $-\mathrm{ln}\left(p\right)$ = 0.36 if rain is subsequently observed, and $-\mathrm{ln}\left(1-p\right)$ = 1.20 if not (we will use natural logarithms throughout). In practice, meteorologists are usually interested in the evaluation of a forecast scheme based on the average score for a data set comprising a sequence of forecasts and the corresponding observations. The Brier score and the logarithmic score apply different penalties; most notably, the logarithmic score attaches larger penalties than does the Brier score to forecasts for which p is close to 0 or 1 when the outcome viewed as unlikely on the basis of the forecast turns out subsequently to be the case. However, both scoring rules are “strictly proper” [6,7].

_{2}O emissions from agricultural soils at the within-season time-scale.

## 2. Methods

#### 2.1. Data, Terminology, Notation

_{t}) and the corresponding observations (o

_{t}), t = 1, …, N, with o

_{t}= 0 for observation of no-rain and o

_{t}= 1 for observation of rain.

_{k}= 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 and 1 (p

_{k}denotes the forecast probability of rain in category k, thus the forecast probability of no-rain is its complement 1 − p

_{k}). The number of observations in each category is denoted n

_{k}and the number of observations of rain in each category is denoted o

_{k}. The average frequency of rain observations in category k is ${\overline{o}}_{k}$ = o

_{k}/n

_{k}. Also ${\sum}_{k}{n}_{k}}=N$, ${\sum}_{k}{o}_{k}}=O$, and the overall average frequency of rain observations is $\overline{o}=O/N$. The components of the decomposition of the Brier score are as follows: reliability, REL

_{BS}= $\frac{1}{N}\cdot {\displaystyle {\sum}_{k}{n}_{k}\cdot {\left({\overline{o}}_{k}-{p}_{k}\right)}^{2}}$; resolution, RES

_{BS}= $\frac{1}{N}\cdot {\displaystyle {\sum}_{k}{n}_{k}\cdot {\left({\overline{o}}_{k}-\overline{o}\right)}^{2}}$; uncertainty, UNC

_{BS}= $\overline{o}\cdot \hspace{0.17em}\left(1-\overline{o}\right)$ (which is the Bernoulli variance); and then BS = REL

_{BS}– RES

_{BS}+ UNC

_{BS}. For the original data set, we calculate the Brier score: BS = 0.1445 (all calculations are shown correct to 4 d.p.). The components of the decomposition of the Brier score are: reliability, REL

_{BS}= 0.0254; resolution, RES

_{BS}= 0.0602; uncertainty UNC

_{BS}= 0.1793. As required, REL

_{BS}– RES

_{BS}+ UNC

_{BS}= BS and the summary of results provided along with the original data set [14] is thus reproduced.

#### 2.2. Probability Forecasts of Zero and One

_{k}= 0 (for category k = 1) and p

_{k}= 1 (for category k = 11); in words, respectively, “it is certain there will be no rain tomorrow” and “it is certain there will be rain tomorrow”. Such forecasts can present problems from the point of view of evaluation. Whereas probability forecasts 0 < p

_{k}< 1 explicitly leave open the chance that an erroneous forecast may be made, probability forecasts p

_{k}= 0 and p

_{k}= 1 do not. The question that then arises is how to evaluate a forecast that was made with certainty but then proves to have been erroneous. This is not a hypothetical issue, as can be seen in the original data set. For category k = 1 (p

_{k}= 0), we note that 1 out of the 46 forecasts made with certainty was erroneous, while for category k = 11 (p

_{k}= 1), we note that 2 out of 13 forecasts made with certainty were erroneous [14]. If such an outcome were to occur when the logarithmic (or divergence) score was in use, an indefinitely large penalty score would apply. In routine practice our preference is to avoid the use of probability forecasts p

_{k}= 0 and p

_{k}= 1 (as a rule of thumb: only use a probability forecast of zero or one when there is absolute certainty of the outcome). There is a price to be paid for taking this point of view, which we discuss later. Notwithstanding, for further analysis in the present article, we will replace the probability forecast for category k = 1 by p

_{k}= 0.05 (instead of zero) and the probability forecast for category k = 11 by p

_{k}= 0.95 (instead of one) (the observations remain unchanged). A summary of the data set incorporating this adjustment (to be used exclusively from this point on) is given in Table 1.

k | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 |
---|---|---|---|---|---|---|---|---|---|---|---|

p_{k} | 0.05 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 0.95 |

o_{k} | 1 | 1 | 5 | 5 | 4 | 8 | 6 | 16 | 16 | 8 | 11 |

n_{k} | 46 | 55 | 59 | 41 | 19 | 22 | 22 | 34 | 24 | 11 | 13 |

^{a}Notation: k, forecast category index; p

_{k}, probability forecast (rain) (probability of no-rain is the complement); o

_{k}, number of rain observations; n

_{k}, number of observations.

#### 2.3. The Brier Score and its Decomposition

_{k}= 0.05, 0.95 instead of 0, 1 for categories k = 1, 11 respectively) we recalculate the Brier score: BS = 0.1440. Then we recalculate the components of the decomposition of the Brier score as follows: reliability, REL

_{BS}= 0.0249; resolution, RES

_{BS}= 0.0602; uncertainty, UNC

_{BS}= 0.1793. As before, REL

_{BS}– RES

_{BS}+ UNC

_{BS}= BS (for full details see Appendix, Table 2).

#### 2.4. The Divergence Score and its Decomposition

_{c}and x

_{r}, respectively) that will be replaced by a probability or a frequency, ranged between zero and one. The distribution (x

_{c}, 1 − x

_{c}) is referred to as the comparison distribution, and the distribution (x

_{r}, 1 − x

_{r}) is referred to as the reference distribution. Note that ${D}_{KL}\left({x}_{c}\Vert {x}_{r}\right)\ge 0$ and that the divergence is not necessarily symmetric with respect to the arguments. For the purpose of numerical calculation, recall that $\underset{x\to 0}{\mathrm{lim}}\left[x\cdot \mathrm{ln}\left(x\right)\right]=0$; then we take $0\cdot \mathrm{ln}\left(0\right)=0$.

_{DS}= $\frac{1}{N}\cdot {\displaystyle {\sum}_{k}{n}_{k}\hspace{0.17em}\cdot {D}_{KL}\left({\overline{o}}_{k}\Vert {p}_{k}\right)}$ = 0.0712; resolution, RES

_{DS}= $\frac{1}{N}\cdot {\displaystyle {\sum}_{k}{n}_{k}\cdot \hspace{0.17em}{D}_{KL}\left({\overline{o}}_{k}\Vert \overline{o}\right)}$ = 0.1683; uncertainty (which in this case is characterized by the binary Shannon entropy [17]), UNC

_{DS}$=u\left(\overline{o}\right)$ $=-\left[\overline{o}\cdot \hspace{0.17em}\mathrm{ln}\left(\overline{o}\right)+\left(1-\overline{o}\right)\hspace{0.17em}\cdot \mathrm{ln}\left(1-\overline{o}\right)\right]$ = 0.5442. Then we have (for full details see Appendix, Table 2):

## 3. Forecast Evaluation via Bregman Divergences

^{2}and the Kullback-Leibler divergence (on which the divergence score is based) is the Bregman divergence associated with f(x) = x∙ln(x) + (1 − x)∙ln(1 − x) (the negative of the binary Shannon entropy function).

_{r}(the reference value). The Bregman divergence between the tangent and the curve at x

_{c}(the comparison value) is then, for scalar arguments:

_{r}. Recall that 0 ≤ x

_{c}≤ 1, 0 ≤ x

_{r}≤ 1; and note that ${D}_{B}\left({x}_{c}\Vert {x}_{r}\right)\ge 0$ and that the divergence is not necessarily symmetric with respect to the arguments. Where necessary for calculation purposes, we take $0\cdot \mathrm{ln}\left(0\right)=0$ as previously.

#### 3.1. Scoring Rules as Bregman Divergences

#### 3.1.1. Brier Score and Divergence Score Diagrams for Individual Forecast Categories

_{k}= 0.4 and an observation $o\in \left\{0,1\right\}$ (see Appendix, Table 3 and Table 4, category k = 5, for details of calculations based on Equation (3)). For individual forecasts, smaller divergences (scores) are better, and from Figure 1A (Brier score) we can see that for reference value p

_{k}= 0.4 the score for comparison value o = 0 (D

_{B}= 0.16, Table 3A, Appendix) is smaller than the score for comparison value o = 1 (D

_{B}= 0.36, Table 3B, Appendix). From Figure 1B (divergence score) we can see that for reference value p

_{k}= 0.4 the score for comparison value o = 0 (D

_{B}= 0.5108, Table 4A, Appendix) is smaller than the score for comparison value o = 1 (D

_{B}= 0.9163, Table 4B, Appendix). In each case this is as we require, because the forecast probability p

_{k}= 0.4 is closer to o = 0 than to o = 1. That is, a forecast of p

_{k}= 0.4 gets a better evaluation score if o = 0 is subsequently observed than if o = 1 is subsequently observed.

- for o = 0, ${D}_{KL}\left(0\Vert {p}_{k}\right)=0\cdot \mathrm{ln}\left(\frac{0}{0.4}\right)+1\cdot \mathrm{ln}\left(\frac{1-0}{1-0.4}\right)$ = 0.5108;
- for o = 1, ${D}_{KL}\left(1\Vert {p}_{k}\right)=1\cdot \mathrm{ln}\left(\frac{1}{0.4}\right)+0\cdot \mathrm{ln}\left(\frac{1-1}{1-0.4}\right)$ = 0.9163.

#### 3.1.2. Overall Scores

_{k}(the reference value, where the tangent is drawn) (Appendix, Table 3). For the divergence score, the Bregman divergence for each individual forecast category (as calculated via Equation (3)) is the Kullback-Leibler divergence between o (the comparison value, where the divergence is calculated) and p

_{k}(the reference value, where the tangent is drawn) (Appendix, Table 4). In each case, the overall score for a forecast-observation data set is calculated as a weighted average of the individual Bregman divergences. For the Brier score, we have$BS=\frac{1}{N}\cdot {\displaystyle {\sum}_{k}{n}_{k}\hspace{0.17em}\cdot {D}_{B}\left(o\Vert {p}_{k}\right)}$ = 49.9375/346 = 0.1440; for the divergence score we have$DS=\frac{1}{N}\cdot {\displaystyle {\sum}_{k}{n}_{k}\hspace{0.17em}\cdot {D}_{B}\left(o\Vert {p}_{k}\right)}$ = 154.6859/346 = 0.4471 (for full details see Appendix, Table 3 and Table 4).

**Figure 1.**Scoring rules as Bregman divergences. The long-dashed curve is a convex function of p, the solid line is a tangent to the convex function at the reference value of p (p

_{k}) indicated by a short-dashed line between the curve and the horizontal axis. The short-dashed lines between the curve and the tangent indicate the Bregman divergence at the comparison values of o (these lines coincide with sections of the vertical axes of the graphs, at comparison values o = 0 and o = 1). (

**A**) Brier score (for calculations see Appendix, Table 3, k = 5). For this example, a tangent to the convex function f(p) = p

^{2}is drawn at probability forecast of rain p

_{k}= 0.4. The score for this forecast depends on the subsequent observation. If no-rain is observed, the score is the Bregman divergence at o = 0, which is 0.16. If rain is observed, the score is the Bregman divergence at o = 1, which is 0.36. Bregman divergences for other forecast-observation combinations are given in the Appendix, Table 3. The overall score for a forecast-observation data set is calculated as a weighted average of the individual Bregman divergences; (

**B**) Divergence score (for calculations see Appendix, Table 4, k = 5). For this example, a tangent to the convex function f(p) = p∙ln(p) + (1 − p)∙ln(1 − p) is drawn at probability forecast of rain p

_{k}= 0.4. The score for this forecast depends on the subsequent observation. If no-rain is observed, the score is the Bregman divergence at o = 0, which is 0.5108. If rain is observed, the score is the Bregman divergence at o = 1, which is 0.9163. Bregman divergences for other forecast-observation combinations are given in the Appendix, Table 4. The overall score for a forecast-observation data set is calculated as a weighted average of the individual Bregman divergences.

#### 3.2. Reliability

#### 3.2.1. Reliability Diagrams for Individual Forecast Categories

_{k}= 0.6 and comparison value ${\overline{o}}_{k}=0.2727$ (see also Appendix, Table 5, category k = 7, for details of calculations based on Equation (3)). From Figure 2A (for the Brier score reliability component) ${D}_{B}\left({\overline{o}}_{k}\Vert {p}_{k}\right)$ = 0.1071. From Figure 2B (for the divergence score reliability component) ${D}_{B}\left({\overline{o}}_{k}\Vert {p}_{k}\right)$ = 0.2198. The corresponding calculation for this divergence score reliability component directly as a Kullback-Leibler divergence is as follows:

**Figure 2.**Reliability as a Bregman divergence. The long-dashed curve is a convex function of p, the solid line is a tangent to the convex function at the reference value of p (p

_{k}) indicated by a short-dashed line between the curve and the horizontal axis. A second short-dashed line, between the curve and the tangent, indicates the Bregman divergence at the comparison value of o (for calculations see Appendix, Table 5). Overall reliability for a forecast-observation data set is calculated as a weighted average of individual Bregman divergences. (

**A**) Brier score reliability. For this example, a tangent to the convex function f(p) = p

^{2}is drawn at probability forecast of rain p

_{k}= 0.6. The reliability component depends on the corresponding ${\overline{o}}_{k}$, the average frequency of rain observations following such forecasts, which is 0.2727 for the example data set. The reliability component is the Bregman divergence at ${\overline{o}}_{k}$= 0.2727, which is 0.1071; (

**B**) Divergence score reliability. For this example, a tangent to the convex function f(p) = p∙ln(p) + (1 − p)∙ln(1 − p) is drawn at probability forecast of rain p

_{k}= 0.6. The reliability component depends on the corresponding ${\overline{o}}_{k}$which is 0.2727 for the example data set. The reliability component is the Bregman divergence at ${\overline{o}}_{k}$ = 0.2727, which is 0.2198.

#### 3.2.2. Overall Reliability

_{k}(the reference value, where the tangent is drawn) (see Appendix, Table 5A). For the divergence score reliability, the Bregman divergence for each individual forecast category (as calculated via Equation (3)) is the Kullback-Leibler divergence between ${\overline{o}}_{k}$and p

_{k}(see Appendix, Table 5B). In each case, the overall reliability score for a forecast-observation data set is calculated as a weighted average of the individual Bregman divergences. For the Brier score, we have $RE{L}_{BS}=\frac{1}{N}\cdot {\displaystyle {\sum}_{k}{n}_{k}}\cdot \hspace{0.17em}{D}_{B}\left({\overline{o}}_{k}\Vert {p}_{k}\right)$ = 8.6204/346 = 0.0249; for the divergence score, we have $RE{L}_{DS}=\frac{1}{N}\cdot {\displaystyle {\sum}_{k}{n}_{k}}\hspace{0.17em}\cdot {D}_{B}\left({\overline{o}}_{k}\Vert {p}_{k}\right)$ = 24.6440/346 = 0.0712 (for full details see Appendix, Table 5).

#### 3.2.3. Interpreting Reliability

_{k}that contribute a small ${D}_{B}\left({\overline{o}}_{k}\Vert {p}_{k}\right)$ to the overall calculation of REL

_{BS}or REL

_{DS}.

#### 3.3. Resolution

#### 3.3.1. Resolution Diagrams for Individual Forecast Categories

#### 3.3.2. Overall Resolution

**Figure 3.**Resolution as a Bregman divergence. The long-dashed curve is a convex function of o, the solid line is a tangent to the convex function at the reference value of o $\left(\overline{o}\right)$ indicated by a short-dashed line between the curve and the horizontal axis. A second short-dashed line, between the curve and the tangent, indicates the Bregman divergence at the comparison value of o (for calculations see Appendix, Table 6). Overall resolution based on a forecast-observation data set is calculated as a weighted average of the individual Bregman divergences. (

**A**) Brier score resolution. For this example, a tangent to the convex function f(o) = o

^{2}is drawn at the overall average frequency of rain observations, $\overline{o}$ = 0.2341. The components of resolution are calculated for each particular ${\overline{o}}_{k}$, the average frequency of rain observations in each category. For k = 9, ${\overline{o}}_{k}$ = 0.6667 for the example data set. The corresponding resolution component is the Bregman divergence at ${\overline{o}}_{k}$ = 0.6667, which is 0.1871; (

**B**) Divergence score resolution. For this example, a tangent to the convex function f(o) = o∙ln(o) + (1 − o)∙ln(1 − o) is drawn at the overall average frequency of rain observations, $\overline{o}$ = 0.2341. The components of resolution are calculated for each particular ${\overline{o}}_{k}$, the average frequency of rain observations in each category. For k = 9, ${\overline{o}}_{k}$ = 0.6667 for the example data set. The corresponding resolution component is the Bregman divergence at ${\overline{o}}_{k}$ = 0.6667, which is 0.4204.

#### 3.3.3. Interpreting Resolution

_{k}= 0 and p

_{k}= 1 are allowed. Further, let us suppose that all 265 observations of no-rain followed forecasts of p

_{k}= 0 (in which case ${\overline{o}}_{k}=0$) and all 81 observations of rain followed forecasts of p

_{k}= 1 (so ${\overline{o}}_{k}=1$). Recall $\overline{o}=0.2341$. If we calculate resolution based on squared Euclidean distance, we have RES

_{BS}= $\frac{1}{N}\cdot \left[265\cdot {\left(0-\overline{o}\right)}^{2}+81\cdot {\left(1-\overline{o}\right)}^{2}\right]$ = 62.0366/346 = 0.1793 = UNC

_{BS}. Alternatively, if we calculate resolution based on the Kullback-Leibler divergence, we have RES

_{DS}= $\frac{1}{N}\cdot \left[265\cdot {D}_{KL}\left(0\Vert \overline{o}\right)+81\cdot {D}_{KL}\left(1\Vert \overline{o}\right)\right]$ = 188.2875/346 = 0.5442 = UNC

_{DS}. That is to say, if we were to allow probability forecast categories p

_{k}= 0 and p

_{k}= 1, then use them exclusively in making forecasts and do so without error, resolution would be equal to uncertainty (i.e., RES

_{BS}= UNC

_{BS}and RES

_{DS}= UNC

_{DS}).

_{k}= 0.05 and p

_{k}= 0.95. Now, the best resolution we can achieve is if all 265 observations of no-rain followed forecasts of p

_{k}= 0.05 (in which case ${\overline{o}}_{k}=0.05$) and all 81 observations of rain followed forecasts of p

_{k}= 0.95 (so ${\overline{o}}_{k}=0.95$). If we calculate resolution based on squared Euclidean distance, we have RES

_{BS}= $\frac{1}{N}\cdot \left[265\cdot {\left(0.05-\overline{o}\right)}^{2}+81\cdot {\left(0.95-\overline{o}\right)}^{2}\right]$ = 50.4960/346 = 0.1459. Alternatively, if we calculate resolution based on the Kullback-Leibler divergence, we have RES

_{DS}= $\frac{1}{N}\cdot \left[265\cdot {D}_{KL}\left(0.05\Vert \overline{o}\right)+81\cdot {D}_{KL}\left(0.95\Vert \overline{o}\right)\right]$ = 130.5177/346 = 0.3772. Thus, the price we pay for restricting the extreme allowed probabilities to p

_{k}= 0.05 and p

_{k}= 0.95 is to reduce the achievable upper limit of resolution.

^{2}(for the Brier score) we have $RES=\frac{1}{N}\cdot {\displaystyle {\sum}_{k}{n}_{k}\cdot}{\left({\overline{o}}_{k}-\overline{o}\right)}^{2}$, the sample variance (e.g., [3]). With f(x) = x∙ln(x) + (1–x)∙ln(1–x) (for the divergence score) we have $RES=\frac{1}{N}\cdot {\displaystyle {\sum}_{k}{n}_{k}}\cdot {D}_{KL}\left({\overline{o}}_{k}\Vert \overline{o}\right)$, the expected mutual information (see also [11,12]).

#### 3.4. Uncertainty

_{BS}= $u\left(\overline{o}\right)=\overline{o}\cdot \hspace{0.17em}\left(1-\overline{o}\right)=0.1793$ (Figure 4A). For the divergence score, uncertainty is calculated as the value of the uncertainty function (the binary Shannon entropy) at $\overline{o}$: UNC

_{DS}= $u\left(\overline{o}\right)$ = $-\left[\overline{o}\hspace{0.17em}\mathrm{ln}\left(\overline{o}\right)+\left(1-\overline{o}\right)\hspace{0.17em}\mathrm{ln}\left(1-\overline{o}\right)\right]$ = 0.5442 (Figure 4B). We interpret uncertainty as a quantification of our state of knowledge in the absence of a forecast, so based only on the data set from which overall average frequency of rain observations $\overline{o}$ is calculated.

**Figure 4.**Uncertainty functions. The long-dashed curves are uncertainty functions, u(o); the short dashed lines indicate $\overline{o}$ (= 0.2341 for the example data set) and the corresponding value of $u\left(\overline{o}\right)$. (

**A**) The Bernoulli variance u(o) = o∙(1 − o). For the example data set, $u\left(\overline{o}\right)$ = 0.1793; (

**B**) The Shannon entropy u(o) = −(o∙ln(o) + (1 − o)∙ln(1 − o)). For the example data set, $u\left(\overline{o}\right)$ = 0.5442.

#### 3.5. Overview

## 4. Discussion

_{k}represent the allowed probability forecasts for rain. For a perfectly reliable forecaster, the observed frequencies of rain events, ${\overline{o}}_{k}/{n}_{k}$, will be equal to p

_{k}in each category k; then REL = 0. Resolution (RES) is a measure of the extent to which the forecaster accounts for uncertainty (but not reliability), i.e., RES ≤ UNC. As mentioned above, in the case of the divergence score, resolution is characterized by expected mutual information. Then, the divergence score (DS) characterizes the uncertainty not accounted for by the forecaster (UNC – RES) together with the reliability (REL), so that DS = UNC – RES + REL.

**Figure 5.**The overall divergence score and its components. The overall divergence score is denoted DS, with components uncertainty (UNC), reliability (REL) and resolution (RES), such that DS = UNC – RES + REL, with RES ≤ UNC as indicated by the vertical dashed line.

_{2}O emissions from agricultural soils, but studies of management interventions aimed at greenhouse gas mitigation have mainly been concerned with emissions inventory, and mitigation options tend to be assessed on an integrated seasonal time-scale [27,28]. An interesting example of the potential for a probabilistic approach to describing short-term N

_{2}O flux dynamics was offered in discussion of a modelling study by Hawkins et al. [29], as follows: “The model depicts a realistic positive emissions response to soil moisture at the mean values of the other factors. This reflects the general understanding that N efficiency, in terms of lower N

_{2}O emission, may be promoted by drier conditions. The WETTEST and DRIEST scenarios were simulated to investigate the magnitude of this efficiency difference. Although these scenarios are hypothetical because in practice the wettest or driest day in a week in terms of soil moisture is not known until the end of the week, they are analogous to spreading fertiliser before or after a rainfall event.” We note here that although the wettest and driest day in a week in terms of soil moisture may only be known retrospectively, weather forecasts provide (probabilistic) advance warning of rainfall events.

## Acknowledgments

## Author Contributions

## Conflicts of Interest

## Appendix

k | p_{k} | n_{k} | o_{k} | ${\overline{o}}_{k}$ | n_{k}/N | REL_{BS,k} | RES_{BS,k} | REL_{DS,k} | RES_{DS,k} |
---|---|---|---|---|---|---|---|---|---|

1 | 0.05 | 46 | 1 | 0.0217 | 0.1329 | 0.0367 | 2.0745 | 0.4862 | 8.6362 |

2 | 0.1 | 55 | 1 | 0.0182 | 0.1590 | 0.3682 | 2.5642 | 2.9939 | 10.8561 |

3 | 0.2 | 59 | 5 | 0.0847 | 0.1705 | 0.7837 | 1.3162 | 2.9746 | 4.5399 |

4 | 0.3 | 41 | 5 | 0.1220 | 0.1185 | 1.2998 | 0.5157 | 3.6576 | 1.6589 |

5 | 0.4 | 19 | 4 | 0.2105 | 0.0549 | 0.6821 | 0.0106 | 1.5491 | 0.0302 |

6 | 0.5 | 22 | 8 | 0.3636 | 0.0636 | 0.4091 | 0.3691 | 0.8286 | 0.9292 |

7 | 0.6 | 22 | 6 | 0.2727 | 0.0636 | 2.3564 | 0.0328 | 4.8346 | 0.0883 |

8 | 0.7 | 34 | 16 | 0.4706 | 0.0983 | 1.7894 | 1.9014 | 3.8702 | 4.5244 |

9 | 0.8 | 24 | 16 | 0.6667 | 0.0694 | 0.4267 | 4.4907 | 1.1695 | 10.0892 |

10 | 0.9 | 11 | 8 | 0.7273 | 0.0318 | 0.3282 | 2.6754 | 1.3052 | 5.9706 |

11 | 0.95 | 13 | 11 | 0.8462 | 0.0376 | 0.1402 | 4.8699 | 0.9745 | 10.9241 |

Column sums^{b} | 346 | 81 | 1.0000 | 8.6204 | 20.8205 | 24.6439 | 58.2471 |

^{a}Notation: k, forecast category index; p

_{k}, probability forecast (rain) (probability forecast of no-rain is the complement); n

_{k}, number of observations; o

_{k}, number of rain observations; ${\overline{o}}_{k}$, average frequency of rain observations = o

_{k}/n

_{k}; n

_{k}/N, normalized frequency of observations; REL

_{BS},

_{k}(components of REL

_{BS}) = ${n}_{k}\cdot {\left({p}_{k}-{\overline{o}}_{k}\right)}^{2}$; RES

_{BS},

_{k}(components of RES

_{BS}) = ${n}_{k}\cdot {\left({\overline{o}}_{k}-\overline{o}\right)}^{2}$; REL

_{DS},

_{k}(components of REL

_{DS}) = ${n}_{k}\hspace{0.17em}\cdot {D}_{KL}\left({\overline{o}}_{k}\Vert {p}_{k}\right)$; RES

_{DS},

_{k}(components of RES

_{DS}) = ${n}_{k}\hspace{0.17em}\cdot {D}_{KL}\left({\overline{o}}_{k}\Vert \overline{o}\right)$; with $\overline{o}=O/N=0.2341$ (footnote b);

^{b}Column sums: ${\sum}_{k}{n}_{k}}=N=346$; ${\sum}_{k}{o}_{k}}=O=81$; ${\sum}_{k}{n}_{k}}/N=1$; $\sum}_{k}{n}_{k}\cdot {\left({p}_{k}-{\overline{o}}_{k}\right)}^{2$ = 8.6204; $\sum}_{k}{n}_{k}\cdot {\left({\overline{o}}_{k}-\overline{o}\right)}^{2$ = 20.8205; ${\sum}_{k}{n}_{k}\hspace{0.17em}\cdot {D}_{KL}\left({\overline{o}}_{k}\Vert {p}_{k}\right)$ = 24.6439; ${\sum}_{k}{n}_{k}\hspace{0.17em}\cdot {D}_{KL}\left({\overline{o}}_{k}\Vert \overline{o}\right)$ = 58.2471.

k | p_{k} | o | n_{k} | ${f}^{\prime}\left({p}_{k}\right)$ | f(o) | f(p_{k}) | $\left(o-{p}_{k}\right)\xb7{f}^{\prime}\left({p}_{k}\right)$ | D_{B}(0||p_{k}) |
---|---|---|---|---|---|---|---|---|

1 | 0.05 | 0 | 45 | 0.1 | 0 | 0.0025 | −0.0050 | 0.0025 |

2 | 0.1 | 0 | 54 | 0.2 | 0 | 0.0100 | −0.0200 | 0.0100 |

3 | 0.2 | 0 | 54 | 0.4 | 0 | 0.0400 | −0.0800 | 0.0400 |

4 | 0.3 | 0 | 36 | 0.6 | 0 | 0.0900 | −0.1800 | 0.0900 |

5^{b} | 0.4 | 0 | 15 | 0.8 | 0 | 0.1600 | −0.3200 | 0.1600 |

6 | 0.5 | 0 | 14 | 1.0 | 0 | 0.2500 | −0.5000 | 0.2500 |

7 | 0.6 | 0 | 16 | 1.2 | 0 | 0.3600 | −0.7200 | 0.3600 |

8 | 0.7 | 0 | 18 | 1.4 | 0 | 0.4900 | −0.9800 | 0.4900 |

9 | 0.8 | 0 | 8 | 1.6 | 0 | 0.6400 | −1.2800 | 0.6400 |

10 | 0.9 | 0 | 3 | 1.8 | 0 | 0.8100 | −1.6200 | 0.8100 |

11 | 0.95 | 0 | 2 | 1.9 | 0 | 0.9025 | −1.8050 | 0.9025 |

k | p_{k} | o | n_{k} | ${f}^{\prime}\left({p}_{k}\right)$ | f(o) | f(p_{k}) | $\left(o-{p}_{k}\right)\xb7{f}^{\prime}\left({p}_{k}\right)$ | D_{B}(1||p_{k}) |
---|---|---|---|---|---|---|---|---|

1 | 0.05 | 1 | 1 | 0.1 | 1 | 0.0025 | 0.0950 | 0.9025 |

2 | 0.1 | 1 | 1 | 0.2 | 1 | 0.0100 | 0.1800 | 0.8100 |

3 | 0.2 | 1 | 5 | 0.4 | 1 | 0.0400 | 0.3200 | 0.4400 |

4 | 0.3 | 1 | 5 | 0.6 | 1 | 0.0900 | 0.4200 | 0.4900 |

5^{b} | 0.4 | 1 | 4 | 0.8 | 1 | 0.1600 | 0.4800 | 0.3600 |

6 | 0.5 | 1 | 8 | 1.0 | 1 | 0.2500 | 0.5000 | 0.2500 |

7 | 0.6 | 1 | 6 | 1.2 | 1 | 0.3600 | 0.4800 | 0.1600 |

8 | 0.7 | 1 | 16 | 1.4 | 1 | 0.4900 | 0.4200 | 0.0900 |

9 | 0.8 | 1 | 16 | 1.6 | 1 | 0.6400 | 0.3200 | 0.0400 |

10 | 0.9 | 1 | 8 | 1.8 | 1 | 0.8100 | 0.1800 | 0.0100 |

11 | 0.95 | 1 | 11 | 1.9 | 1 | 0.9025 | 0.0950 | 0.0025 |

^{a}Notation: k, forecast category index; p

_{k}, probability forecast for rain (reference value, at which the tangent is calculated), probability forecast for no-rain is the complement; o, comparison value, at which the divergence is calculated; n

_{k}, number of observations (total no-rain observations = 265, total rain observations = 81); ${f}^{\prime}\left({p}_{k}\right)$, slope of the tangent to f(p) at p

_{k}; $f\left(o\right)-f\left({p}_{k}\right)$− $\left(o-{p}_{k}\right)\cdot {f}^{\prime}\left({p}_{k}\right)$ = D

_{B}(0||p

_{k}) (no-rain, o = 0), or $f\left(o\right)-f\left({p}_{k}\right)$ − $\left(o-{p}_{k}\right)\cdot {f}^{\prime}\left({p}_{k}\right)$ = D

_{B}(1||p

_{k}) (rain, o = 1);

^{b}See Figure 1A.

k | p_{k} | o | n_{k} | ${f}^{\prime}\left({p}_{k}\right)$ | f(o) | f(p_{k}) | $\left(o-{p}_{k}\right)\xb7{f}^{\prime}\left({p}_{k}\right)$ | D_{B}(0||p_{k}) |
---|---|---|---|---|---|---|---|---|

1 | 0.05 | 0 | 45 | −2.9444 | 0 | −0.1985 | 0.1472 | 0.0513 |

2 | 0.1 | 0 | 54 | −2.1972 | 0 | −0.3251 | 0.2197 | 0.1054 |

3 | 0.2 | 0 | 54 | −1.3863 | 0 | −0.5004 | 0.2773 | 0.2231 |

4 | 0.3 | 0 | 36 | −0.8473 | 0 | −0.6109 | 0.2542 | 0.3567 |

5^{b} | 0.4 | 0 | 15 | −0.4055 | 0 | −0.6730 | 0.1622 | 0.5108 |

6 | 0.5 | 0 | 14 | 0.0000 | 0 | −0.6931 | 0.0000 | 0.6931 |

7 | 0.6 | 0 | 16 | 0.4055 | 0 | −0.6730 | −0.2433 | 0.9163 |

8 | 0.7 | 0 | 18 | 0.8473 | 0 | −0.6109 | −0.5931 | 1.2040 |

9 | 0.8 | 0 | 8 | 1.3863 | 0 | −0.5004 | −1.1090 | 1.6094 |

10 | 0.9 | 0 | 3 | 2.1972 | 0 | −0.3251 | −1.9775 | 2.3026 |

11 | 0.95 | 0 | 2 | 2.9444 | 0 | −0.1985 | −2.7972 | 2.9957 |

k | p_{k} | o | n_{k} | ${f}^{\prime}\left({p}_{k}\right)$ | f(o) | f(p_{k}) | $\left(o-{p}_{k}\right)\xb7{f}^{\prime}\left({p}_{k}\right)$ | D_{B}(1||p_{k}) |
---|---|---|---|---|---|---|---|---|

1 | 0.05 | 1 | 1 | −2.9444 | 0 | −0.1985 | −2.7972 | 2.9957 |

2 | 0.1 | 1 | 1 | −2.1972 | 0 | −0.3251 | −1.9775 | 2.3026 |

3 | 0.2 | 1 | 5 | −1.3863 | 0 | −0.5004 | −1.1090 | 1.6094 |

4 | 0.3 | 1 | 5 | −0.8473 | 0 | −0.6109 | −0.5931 | 1.2040 |

5^{b} | 0.4 | 1 | 4 | −0.4055 | 0 | −0.6730 | −0.2433 | 0.9163 |

6 | 0.5 | 1 | 8 | 0.0000 | 0 | −0.6931 | 0.0000 | 0.6931 |

7 | 0.6 | 1 | 6 | 0.4055 | 0 | −0.6730 | 0.1622 | 0.5108 |

8 | 0.7 | 1 | 16 | 0.8473 | 0 | −0.6109 | 0.2542 | 0.3567 |

9 | 0.8 | 1 | 16 | 1.3863 | 0 | −0.5004 | 0.2773 | 0.2231 |

10 | 0.9 | 1 | 8 | 2.1972 | 0 | −0.3251 | 0.2197 | 0.1054 |

11 | 0.95 | 1 | 11 | 2.9444 | 0 | −0.1985 | 0.1472 | 0.0513 |

^{a}Notation: k, forecast category index; p

_{k}, probability forecast for rain (reference value, at which the tangent is calculated), probability forecast of no-rain is the complement; o, comparison value, at which the divergence is calculated; n

_{k}, number of observations (total no-rain observations = 265, total rain observations = 81); ${f}^{\prime}\left({p}_{k}\right)$, slope of the tangent to f(p) at p

_{k}; $f\left(o\right)-f\left({p}_{k}\right)$ − $\left(o-{p}_{k}\right)\cdot {f}^{\prime}\left({p}_{k}\right)$ = D

_{B}(0||p

_{k}) (no-rain, o = 0), or $f\left(o\right)-f\left({p}_{k}\right)$ − $\left(o-{p}_{k}\right)\cdot {f}^{\prime}\left({p}_{k}\right)$ = D

_{B}(1||p

_{k}) (rain, o = 1);

^{b}See Figure 1B.

k | p_{k} | ${\overline{o}}_{k}$ | n_{k} | ${f}^{\prime}\left({p}_{k}\right)$ | $f\left({\overline{o}}_{k}\right)$ | f(p_{k}) | $\left({\overline{o}}_{k}-{p}_{k}\right)\hspace{0.17em}\xb7{f}^{\prime}\left({p}_{k}\right)$ | ${D}_{B}\left({\overline{o}}_{k}\Vert {p}_{k}\right)$ |
---|---|---|---|---|---|---|---|---|

1 | 0.05 | 0.0217 | 46 | 0.1 | 0.0005 | 0.0025 | −0.0028 | 0.0008 |

2 | 0.1 | 0.0182 | 55 | 0.2 | 0.0003 | 0.0100 | −0.0164 | 0.0067 |

3 | 0.2 | 0.0847 | 59 | 0.4 | 0.0072 | 0.0400 | −0.0461 | 0.0133 |

4 | 0.3 | 0.1220 | 41 | 0.6 | 0.0149 | 0.0900 | −0.1068 | 0.0317 |

5 | 0.4 | 0.2105 | 19 | 0.8 | 0.0443 | 0.1600 | −0.1516 | 0.0359 |

6 | 0.5 | 0.3636 | 22 | 1.0 | 0.1322 | 0.2500 | −0.1364 | 0.0186 |

7^{b} | 0.6 | 0.2727 | 22 | 1.2 | 0.0744 | 0.3600 | −0.3927 | 0.1071 |

8 | 0.7 | 0.4706 | 34 | 1.4 | 0.2215 | 0.4900 | −0.3212 | 0.0526 |

9 | 0.8 | 0.6667 | 24 | 1.6 | 0.4444 | 0.6400 | −0.2133 | 0.0178 |

10 | 0.9 | 0.7273 | 11 | 1.8 | 0.5289 | 0.8100 | −0.3109 | 0.0298 |

11 | 0.95 | 0.8462 | 13 | 1.9 | 0.7160 | 0.9025 | −0.1973 | 0.0108 |

k | p_{k} | ${\overline{o}}_{k}$ | n_{k} | ${f}^{\prime}\left({p}_{k}\right)$ | $f\left({\overline{o}}_{k}\right)$ | f(p_{k}) | $\left({\overline{o}}_{k}-{p}_{k}\right)\hspace{0.17em}\xb7{f}^{\prime}\left({p}_{k}\right)$ | ${D}_{B}\left({\overline{o}}_{k}\Vert {p}_{k}\right)$ |
---|---|---|---|---|---|---|---|---|

1 | 0.05 | 0.0217 | 46 | −2.9444 | −0.1047 | −0.1985 | 0.0832 | 0.0106 |

2 | 0.1 | 0.0182 | 55 | −2.1972 | −0.0909 | −0.3251 | 0.1798 | 0.0544 |

3 | 0.2 | 0.0847 | 59 | −1.3863 | −0.2902 | −0.5004 | 0.1598 | 0.0504 |

4 | 0.3 | 0.1220 | 41 | −0.8473 | −0.3708 | −0.6109 | 0.1509 | 0.0892 |

5 | 0.4 | 0.2105 | 19 | −0.4055 | −0.5147 | −0.6730 | 0.0768 | 0.0815 |

6 | 0.5 | 0.3636 | 22 | 0.0000 | −0.6555 | −0.6931 | 0.0000 | 0.0377 |

7^{b} | 0.6 | 0.2727 | 22 | 0.4055 | −0.5860 | −0.6730 | −0.1327 | 0.2198 |

8 | 0.7 | 0.4706 | 34 | 0.8473 | −0.6914 | −0.6109 | −0.1944 | 0.1138 |

9 | 0.8 | 0.6667 | 24 | 1.3863 | −0.6365 | −0.5004 | −0.1848 | 0.0487 |

10 | 0.9 | 0.7273 | 11 | 2.1972 | −0.5860 | −0.3251 | −0.3795 | 0.1187 |

11 | 0.95 | 0.8462 | 13 | 2.9444 | −0.4293 | −0.1985 | −0.3058 | 0.0750 |

^{a}Notation: k, forecast category index; p

_{k}, probability forecast for rain (reference value, at which the tangent is calculated), probability forecast for no-rain is the complement; ${\overline{o}}_{k}$, average frequency of rain observations (comparison value, at which the divergence is calculated); n

_{k}, number of observations; ${f}^{\prime}\left({p}_{k}\right)$, slope of the tangent to f(p) at p

_{k}; $f\left({\overline{o}}_{k}\right)$ − f(p

_{k}) − $\left({\overline{o}}_{k}-{p}_{k}\right)\hspace{0.17em}\cdot {f}^{\prime}\left({p}_{k}\right)$ = ${D}_{B}\left({\overline{o}}_{k}\Vert {p}_{k}\right)$;

^{b}See Figure 2.

k | $\overline{o}$ | ${\overline{o}}_{k}$ | n_{k} | ${f}^{\prime}\left(\overline{o}\right)$ | $f\left({\overline{o}}_{k}\right)$ | $f\left(\overline{o}\right)$ | $\left({\overline{o}}_{k}-\overline{o}\right)\hspace{0.17em}\xb7{f}^{\prime}\left(\overline{o}\right)$ | ${D}_{B}\left({\overline{o}}_{k}\Vert \overline{o}\right)$ |
---|---|---|---|---|---|---|---|---|

1 | 0.2341 | 0.0217 | 46 | 0.4682 | 0.0005 | 0.0548 | −0.0994 | 0.0451 |

2 | 0.2341 | 0.0182 | 55 | 0.4682 | 0.0003 | 0.0548 | −0.1011 | 0.0466 |

3 | 0.2341 | 0.0847 | 59 | 0.4682 | 0.0072 | 0.0548 | −0.0699 | 0.0223 |

4 | 0.2341 | 0.1220 | 41 | 0.4682 | 0.0149 | 0.0548 | −0.0525 | 0.0126 |

5 | 0.2341 | 0.2105 | 19 | 0.4682 | 0.0443 | 0.0548 | −0.0110 | 0.0006 |

6 | 0.2341 | 0.3636 | 22 | 0.4682 | 0.1322 | 0.0548 | 0.0606 | 0.0168 |

7 | 0.2341 | 0.2727 | 22 | 0.4682 | 0.0744 | 0.0548 | 0.0181 | 0.0015 |

8 | 0.2341 | 0.4706 | 34 | 0.4682 | 0.2215 | 0.0548 | 0.1107 | 0.0559 |

9^{b} | 0.2341 | 0.6667 | 24 | 0.4682 | 0.4444 | 0.0548 | 0.2025 | 0.1871 |

10 | 0.2341 | 0.7273 | 11 | 0.4682 | 0.5289 | 0.0548 | 0.2309 | 0.2432 |

11 | 0.2341 | 0.8462 | 13 | 0.4682 | 0.7160 | 0.0548 | 0.2866 | 0.3746 |

k | $\overline{o}$ | ${\overline{o}}_{k}$ | n_{k} | ${f}^{\prime}\left(\overline{o}\right)$ | $f\left({\overline{o}}_{k}\right)$ | $f\left(\overline{o}\right)$ | $\left({\overline{o}}_{k}-\overline{o}\right)\hspace{0.17em}\xb7{f}^{\prime}\left(\overline{o}\right)$ | ${D}_{B}\left({\overline{o}}_{k}\Vert \overline{o}\right)$ |
---|---|---|---|---|---|---|---|---|

1 | 0.2341 | 0.0217 | 46 | −1.1853 | −0.1047 | −0.5442 | 0.2517 | 0.1877 |

2 | 0.2341 | 0.0182 | 55 | −1.1853 | −0.0909 | −0.5442 | 0.2559 | 0.1974 |

3 | 0.2341 | 0.0847 | 59 | −1.1853 | −0.2902 | −0.5442 | 0.1770 | 0.0769 |

4 | 0.2341 | 0.1220 | 41 | −1.1853 | −0.3708 | −0.5442 | 0.1329 | 0.0405 |

5 | 0.2341 | 0.2105 | 19 | −1.1853 | −0.5147 | −0.5442 | 0.0279 | 0.0016 |

6 | 0.2341 | 0.3636 | 22 | −1.1853 | −0.6555 | −0.5442 | −0.1535 | 0.0422 |

7 | 0.2341 | 0.2727 | 22 | −1.1853 | −0.5860 | −0.5442 | −0.0458 | 0.0040 |

8 | 0.2341 | 0.4706 | 34 | −1.1853 | −0.6914 | −0.5442 | −0.2803 | 0.1331 |

9^{b} | 0.2341 | 0.6667 | 24 | −1.1853 | −0.6365 | −0.5442 | −0.5127 | 0.4204 |

10 | 0.2341 | 0.7273 | 11 | −1.1853 | −0.5860 | −0.5442 | −0.5845 | 0.5428 |

11 | 0.2341 | 0.8462 | 13 | −1.1853 | −0.4293 | −0.5442 | −0.7255 | 0.8403 |

^{a}Notation: k, forecast category index; $\overline{o}$, overall average frequency of rain observations (see Table 2) (reference value, at which the tangent is calculated); ${\overline{o}}_{k}$, average frequency of rain observations (comparison value, at which the divergence is calculated); n

_{k}, number of observations; ${f}^{\prime}\left(\overline{o}\right)$, slope of the tangent to f(o) at $\overline{o}$; $f\left({\overline{o}}_{k}\right)$ − $f\left(\overline{o}\right)$ − $\left({\overline{o}}_{k}-\overline{o}\right)\hspace{0.17em}\cdot {f}^{\prime}\left(\overline{o}\right)$ = ${D}_{B}\left({\overline{o}}_{k}\Vert \overline{o}\right)$;

^{b}See Figure 3.

## References

- Lindley, D.V. Making Decisions, 2nd ed.; Wiley: Chichester, UK, 1985. [Google Scholar]
- Jolliffe, I.T.; Stephenson, D.B. (Eds.) Forecast Verification: A Practitioner’s Guide in Atmospheric Science, 2nd ed.; Wiley: Chichester, UK, 2012.
- Broecker, J. Probability forecasts. In Forecast Verification: A Practitioner’s Guide in Atmospheric Science, 2nd ed.; Jolliffe, I.T., Stephenson, D.B., Eds.; Wiley: Chichester, UK, 2012; pp. 119–139. [Google Scholar]
- Brier, G.W. Verification of forecasts expressed in terms of probability. Mon. Weather Rev.
**1950**, 78, 1–3. [Google Scholar] [CrossRef] - Good, I.J. Rational decisions. J. Roy. Stat. Soc. B
**1952**, 14, 107–114. [Google Scholar] - DeGroot, M.W.; Fienberg, S.E. The comparison and evaluation of forecasters. The Statistician
**1983**, 32, 12–22. [Google Scholar] [CrossRef] - Bröcker, J.; Smith, L.A. Scoring probabilistic forecasts: the importance of being proper. Weather Forecast.
**2007**, 22, 382–388. [Google Scholar] [CrossRef] - Bröcker, J. Reliability, sufficiency, and the decomposition of proper scores. Q. J. R. Meteorol. Soc.
**2009**, 135, 1512–1519. [Google Scholar] [CrossRef] - Murphy, A.H. A new vector partition of the probability score. J. Appl. Meteorol.
**1973**, 12, 595–600. [Google Scholar] [CrossRef] - Wilks, D.S. Statistical Methods in the Atmospheric Sciences, 3rd ed.; Academic Press: Oxford, UK, 2011. [Google Scholar]
- Weijs, S.V.; Schoups, G.; van de Giesen, N. Why hydrological predictions should be evaluated using information theory. Hydrol. Earth Syst. Sci.
**2010**, 14, 2545–2558. [Google Scholar] [CrossRef] - Weijs, S.V.; van Nooijen, R.; van de Giesen, N. Kullback-Leibler divergence as a forecast skill score with classic reliability-resolution-uncertainty decomposition. Mon. Weather Rev.
**2010**, 138, 3387–3399. [Google Scholar] [CrossRef] - Gneiting, T.; Katzfuss, M. Probabilistic forecasting. Annu. Rev. Stat. Appl.
**2014**, 1, 125–151. [Google Scholar] [CrossRef] - Verifying probability of precipitation—an example from Finland. http://www.cawcr.gov.au/projects/verification/POP3/POP3.html (accessed on 18 June 2015).
- Kullback, S. Information Theory and Statistics, 2nd ed.; Dover: New York, NY, USA, 1968. [Google Scholar]
- Cover, T.M.; Thomas, J.A. Elements of Information Theory, 2nd ed.; Wiley: Hoboken, NJ, USA, 2006. [Google Scholar]
- Shannon, C.E.; Weaver, W. The Mathematical Theory of Communication; University of Illinois Press: Urbana, IL, USA, 1949. [Google Scholar]
- Bregman, L.M. The relaxation method of finding the common points of convex sets and its application to the solution of problems in convex programming. USSR Comput. Math. Math. Phys.
**1967**, 7, 200–217. [Google Scholar] [CrossRef] - Gneiting, T.; Raftery, A.E. Strictly proper scoring rules, prediction, and estimation. J. Am. Stat. Assoc.
**2007**, 102, 359–378. [Google Scholar] [CrossRef] - Adamčik, M. The information geometry of Bregman divergences and some applications in multi-expert reasoning. Entropy
**2014**, 16, 6338–6381. [Google Scholar] [CrossRef] - Reid, M.D.; Williamson, R.C. Information, divergence and risk for binary experiments. J. Mach. Learn. Res.
**2011**, 12, 731–817. [Google Scholar] - Banerjee, A.; Merugu, S.; Dhillon, I.S.; Ghosh, J. Clustering with Bregman divergences. J. Mach. Learn. Res.
**2005**, 6, 1705–1749. [Google Scholar] - Theil, H. Statistical Decomposition Analysis; North-Holland Publishing Company: Amsterdam, The Netherlands, 1972. [Google Scholar]
- Benedetti, R. Scoring rules for forecast verification. Mon. Weather Rev.
**2010**, 138, 203–211. [Google Scholar] [CrossRef] - Tödter, J.; Ahrens, B. Generalization of the ignorance score: continuous ranked version and its decomposition. Mon. Weather Rev.
**2012**, 140, 2005–2017. [Google Scholar] [CrossRef] - Topp, C.F.E.; Wang, W.; Cloy, J.M.; Rees, R.M.; Hughes, G. Information properties of boundary line models for N
_{2}O emissions from agricultural soils. Entropy**2013**, 15, 972–987. [Google Scholar] [CrossRef] - Cardenas, L.M.; Gooday, R.; Brown, L.; Scholefield, D.; Cuttle, S.; Gilhespy, S.; Matthews, R.; Misselbrook, T.; Wang, J.; Li, C.; Hughes, G.; Lord, E. Towards an improved inventory of N
_{2}O from agriculture: model evaluation of N_{2}O emission factors and N fraction leached from different sources in UK agriculture. Atmos. Environ.**2013**, 79, 340–348. [Google Scholar] [CrossRef] - Rees, R.M.; Augustin, J.; Alberti, G.; Ball, B.C.; Boeckx, P.; Cantarel, A.; Castaldi, S.; Chirinda, N.; Chojnicki, B.; Giebels, M.; Gordon, H.; Grosz, B.; Horvath, L.; Juszczak, R.; Klemedtsson, Å.K.; Klemedtsson, L.; Medinets, S.; Machon, A.; Mapanda, F.; Nyamangara, J.; Olesen, J.E.; Reay, D.S.; Sanchez, L.; Sanz Cobena, A.; Smith, K.A.; Sowerby, A.; Sommer, J.M.; Soussana, J.F.; Stenberg, M.; Topp, C.F.E.; van Cleemput, O.; Vallejo, A.; Watson, C.A.; Wuta, M. Nitrous oxide emissions from European agriculture—an analysis of variability and drivers of emissions from field experiments. Biogeosciences
**2013**, 10, 2671–2682. [Google Scholar] - Hawkins, M.J.; Hyde, B.P.; Ryan, M.; Schulte, R.P.O.; Connolly, J. An empirical model and scenario analysis of nitrous oxide emissions from a fertilised and grazed grassland site in Ireland. Nutr. Cycl. Agroecosyst.
**2007**, 79, 93–101. [Google Scholar] [CrossRef] - Moran, D.; Lucas, A.; Barnes, A. Mitigation win-win. Nat. Clim. Change
**2013**, 3, 611–613. [Google Scholar] [CrossRef]

© 2015 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Hughes, G.; Topp, C.F.E.
Probabilistic Forecasts: Scoring Rules and Their Decomposition and Diagrammatic Representation via Bregman Divergences. *Entropy* **2015**, *17*, 5450-5471.
https://doi.org/10.3390/e17085450

**AMA Style**

Hughes G, Topp CFE.
Probabilistic Forecasts: Scoring Rules and Their Decomposition and Diagrammatic Representation via Bregman Divergences. *Entropy*. 2015; 17(8):5450-5471.
https://doi.org/10.3390/e17085450

**Chicago/Turabian Style**

Hughes, Gareth, and Cairistiona F.E. Topp.
2015. "Probabilistic Forecasts: Scoring Rules and Their Decomposition and Diagrammatic Representation via Bregman Divergences" *Entropy* 17, no. 8: 5450-5471.
https://doi.org/10.3390/e17085450