# Statistical Evidence Measured on a Properly Calibrated Scale for Multinomial Hypothesis Comparisons

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

“To a certain degree this scheme is typical for all theoretic knowledge: We begin with some general but vague principle, then find an important case where we can give that notion a concrete precise meaning, and from that case we gradually rise again to generality and if we are lucky we end up with an idea no less universal than the one from which we started. Gone may be much of its emotional appeal, but it has the same or even greater unifying power in the realm of thought and is exact instead of vague.”([8], p. 6, emphasis added)

## 2. A Vague but General Understanding of Statistical Evidence

- (i)
- Evidence as a function of changes in x/n for fixed n: If we hold n constant but allow x/n to increase from 0 up to ½, the evidence in favor of bias will at first diminish, and then at some point the evidence will begin to increase, as it shifts to favoring no bias. BBP(i) is illustrated in Figure 1a.
- (ii)
- Evidence as a function of changes in n for fixed x/n: For any given value of x/n, the evidence increases as n increases. The evidence may favor bias (e.g., if x/n = 0) or no bias (e.g., if x/n = ½), but in either case it increases with increasing n. Additionally, this increase in the evidence becomes smaller as n increases. For example, five tails in a row increase the evidence for bias by a greater amount if they are preceded by two tails, compared to if they are preceded by 100 tails. BBP(ii) is illustrated in Figure 1b.
- (iii)
- x/n as a function of changes in n (or vice versa) for fixed evidence: It follows from BBPs(i) and BBPs(ii) that in order for the evidence to remain constant, n and x/n must adjust to one another in a compensatory manner. For instance, if x/n increases from 0 to 0.05, in order for the evidence to remain the same, n must increase to compensate; otherwise, the evidence would go down following BBP(i). BBP(iii) is illustrated in Figure 1c.

## 3. A Precise but Non-general Definition of Statistical Evidence

_{1}: 0 ≤ θ < ½ vs. H

_{2}: θ = ½. Then the likelihood ratio (LR) on given data (n, x) is:

_{1}) does capture the set of behaviors we expect of the evidence. This presumably reflects some important underlying relationship between the MLR and the ALR, which to our knowledge has not been previously explored. It also illustrates that the most intuitively appealing mathematical definitions of evidence are not necessarily the best definitions, as they may exhibit behaviors that contradict our general concept of evidence.

_{1}appears to satisfy Weyl’s second stage: It is precise, but overly specific. The next question is, how general can we make our definition of evidence while maintaining this level of precision?

_{1}: θ < ½ (“linkage”) and H

_{2}: θ = ½ (“no linkage”).

_{1}in application to linkage analysis. Consider two data sets: D

_{1}(n = 50, x = 17), and D

_{2}(n = 85, x = 32). MLR = 13.5 for D

_{1}and 13.8 for D

_{2}. Thus we might conclude that both data sets support H

_{1}roughly equally well. (Recall that the MLR can never support H

_{2}, but can only indicate varying degrees of support for H

_{1}.) For these same data, we obtain e

_{1}= 6.1 and e

_{1}= 7.8, for D

_{1}, D

_{2}respectively. However, for D

_{1}the data fall to the left of the TrP, and therefore e

_{1}represents evidence for linkage; whereas for D

_{2}, the data fall to the right of the TrP and e

_{1}represents evidence against linkage. If we were to perform linkage analysis using the MLR, we would miss the fact (assuming that e

_{1}is behaving correctly) that one dataset supports linkage but the other doesn’t. This same point applies to the (one-sided) p-value (p-value = 0.016, 0.015 for D

_{1}, D

_{2}respectively), and the Bayesian e-value (Ev) of [16,17] (Ev = 0.009, 0.010 for D

_{1}, D

_{2}). A key feature of e

_{1}is that it represents evidence either in favor of H

_{1}or in favor of H

_{2}., depending on the side of the TrP on which the data fall, thereby satisfying BBP(i) (Figure 1a and Figure 2a).

## 4. First Generalization: From e_{1} to e_{2}

_{i}be the number of observations in the i

^{th}category, and θ

_{i}be the probability of the i

^{th}category. Let

**θ**= (θ

_{1},..,θ

_{m}) and let

**x/n**= (x

_{1}/n,…,x

_{m}/n). The multinomial likelihood in m categories can be written as:

_{i}and x

_{i}/n satisfy the constraints on the right hand side of Equation (3). We view the simplex as a function of

**θ**when specifying the hypotheses, and as a function of the data

**x/n**when considering the evidence.

^{2}is an equilateral triangle with vertices (1, 0, 0), (0, 1, 0) and (0, 0, 1). In general, the point

**θ**= (θ

_{1}= θ

_{2}= … = θ

_{m}) is called the centroid, which we denote ${\Delta}_{CENT}^{m-1}$, so that ${\Delta}_{CENT}^{2}$ = (⅓, ⅓, ⅓). When viewing the simplex as a function of

**θ**, we also use the shorthand

**θ**

_{CENT}; when viewed as a function of the data, the centroid similarly occurs at

**x/n**= (⅓, ⅓, ⅓). For any given

**x/n**, the simplex contains six points corresponding to the six possible permuted orders of the data, as shown in Figure 5b. Each of these six points falls into one of six regions, and these regions represent a symmetry group with respect to the likelihood, that is, each point in any one region has a (permuted) homologue in each of the remaining regions corresponding to the same likelihood.

**x/n**= (x

_{1}/n, x

_{2}/n, x

_{3}/n) along l that is 10% of the distance from the centroid to the boundary. In order to maintain the same orientation as used in the previous binomial graphs, we plot results along a given line l(t) with t running from 1 to 0 (left to right) along the x-axis. Although numerical details vary among different lines, the patterns of all results discussed here apply to all such lines. Hence we can plot results along a particular line without loss of generality.

_{1}: $\mathit{\theta}\ne {\Delta}_{CENT}^{m-1}$ vs. H

_{2}: $\mathit{\theta}={\Delta}_{CENT}^{m-1}$. In our current notation, we can rewrite the formula for evidence derived for the binomial in [9], while extending the notation to cover general m-dimensional multinomial HCs, as:

_{1}= x

_{1}/n,..,θ

_{m}= x

_{m}/n) , and:

^{n}B(x

_{1}+1,…,x

_{m}+1).

_{2}(Equation (4)), and confirms that e

_{2}continues to exhibit the expected behaviors. These included BBPs(i)–(iii), as described above. In addition, the TrP along any given line l(t) approaches ${\Delta}_{CENT}^{m}$ asymptotically, as it did in the original binomial setting.

## 5. Second Generalization: From e_{2} to e_{3}

^{m−1}, and the point t

_{α}which is α% of the distance along each such line. For the trinomial, these points form an embedded 2-simplex, which we denote ${\Delta}_{\alpha}^{2}$, having the same centroid as Δ

^{2}(Figure 7). Our new HC becomes H

_{1}:

**θ**∈ Δ

^{m−1}vs. H

_{2}:

**θ**∈ ${\Delta}_{\alpha}^{m-1}$, for a specified value of α.

_{i}be ${\mathrm{L}}_{\mathrm{H}i}(\widehat{\mathit{\theta}}\text{}|\text{}n,\text{}{x}_{1},\mathrm{..},{x}_{m})$, indicating that the likelihood is evaluated at the m.l.e. vector $\widehat{\mathit{\theta}}$ under any constraints imposed by H

_{i}. We now extend the calculating formula as follows:

**θ**. V (Equation (9)) is a straightforward generalization of VLR (Equation (6)); the only difference is that in the denominator, rather than fixing

**θ**at the single value stipulated by H

_{2}, we use the constrained m.l.e. under H

_{2}. S is a generalization of the log of the MLR (Equation (5)); again, the denominator is evaluated at the constrained m.l.e. rather than at a value fixed a priori. Equation (8) has also been called the generalized LR (GLR) and proposed as a definition of statistical evidence for composite vs. composite HCs [11,12]. The reasons for changing from “MLR” to “S” and placing S on the log scale will become clear below. Note that when α = 0, Equation (7) is identical to Equation (4). Thus in moving from e

_{1}to e

_{2}to e

_{3}, we are not changing the definition of evidence, but extending it to encompass more general cases.

_{2}(embedded) 2-simplex, converging asymptotically to the boundary of the inner triangle as n increases. This seems a natural extension of the behavior of the TrP in the composite vs. simple case.

_{1}simplex, the evidence for H

_{1}is larger the smaller is α, that is, the more incompatible the data are with H

_{2}; when the data are close to the centroid, the evidence for H

_{2}is larger the larger is α, that is, the more incompatible the data are with H

_{1}. Again, even unanticipated aspects of the behavior of e

_{3}appear to reflect appropriate behavior for an evidence measure.

## 6. Measurement Calibration

_{1}, e

_{2}or e

_{3}) only as an empirical measure, one that exhibits the correct behaviors. But measurement requires more than this, it requires proper calibration, so that a given measurement value always “means the same thing” with respect to the underlying object of measurement, across applications and across the measurement scale. In this section we continue to develop our argument [9,10] that Equation (7)—the most general form of the equation of state thus far—is not merely a good empirical measure of evidence, but also, that it is inherently on a properly calibrated, context-independent ratio scale. From this point forward we therefore refer to Equation (7) using “E”.

_{1}” and “evidence for H

_{2}.”

**x/n**vector along each possible line l that minimizes E; that is, TrL is the set of TrPs comprising the point along each l at which the directional derivative = 0. For the trinomial, the TrPs form a line (TrL), for the quadrinomial they form a plane (TrPL), etc.; Figure 11 illustrates the quadrinomial TrPL. More explicitly, the directional derivative $\overrightarrow{t}$ =

**x/n**− ${\Delta}_{CENT}^{m-1}$ becomes the inner product of t and the vector of partial derivatives of E for given data (n,

**x/n**). That is, ${\nabla}_{t}E=\nabla E\times \frac{\overrightarrow{t}}{|\overrightarrow{t}|}$. The TrP is the value of

**x/n**at which $\nabla E\times \overrightarrow{t}=0$.

**x/n**supports H

_{1}, while if it is negative,

**x/n**supports H

_{2}. In practice, it may be useful to show evidence using the sign of the directional derivative to indicate evidence for H

_{2}vs. evidence for H

_{1}. (This would be analogous to the convention in physics of showing mechanical work as either positive or negative, to indicate whether the given amount of work is being done by the system or to the system). The (trinomial) TrL separates those data values that constitute evidence for H

_{1}(outside the TrL) from those data values that constitute evidence for H

_{2}(inside the TrL). Note in particular that the TrL is not defined in advance or imposed based on any extraneous considerations, such as error probabilities. Rather, the TrL emerges from the underlying relationships inherent in Equation (7). This is in stark contrast to approaches to evidence measurement based on the MLR or the p-value, in which one specifies a threshold value beyond which the evidence is considered to (sufficiently) favor one hypothesis over the other based on external considerations, such as control of error rates. A natural question to ask, then, is what the TrL itself represents. What features of the system characterize the set of points that constitute the TrL?

^{2}significance test will also vary along the TrL. Additionally, for the BF the demarcation will depend upon the prior.) The demarcation indicating evidence for or against “die is fair” in our framework is not only conceptually distinct, but also mathematically different from the demarcation in pure likelihood, Bayesian or frequentist settings. Thus there is no obvious relationship between the TrL and readily identifiable quantities that play roles in other approaches to inference.

^{st}two laws of thermodynamics [19].

## 7. Discussion

“[There is] a notable dissimilarity between thermodynamics and the other branches of classical science….[Thermodynamics] reflects a commonality or universal feature of all laws…[it] is the study of the restrictions on the possible properties of matter that follow from the symmetry properties of the fundamental laws of physics. Thermodynamics inherits its universality…from its symmetry parentage.”([24], pp. 2–3)

## Acknowledgments

## Author Contributions

## Conflicts of Interest

## Abbreviations

LR | Likelihood Ratio |

BF | Bayes Factor |

MLR | Maximum LR |

ALR | Area under LR |

VLR | Volume under LR |

BBP | Basic Behavior Pattern (for evidence) |

HC | Hypothesis Contrast |

TrP | Transition Point |

TrL | Transition Line |

TrPL | Transition Plane |

## Appendix A. The constants d.f. and b

_{1}and H

_{2}, respectively, using subscripts 1, 2 on θ. For what we called Class II (nested, composite vs. simple and composite vs. composite) HCs, we defined d.f. as 1 + [θ

_{1r}− θ

_{1l}]+ [θ

_{2r}− θ

_{2l}], where [θ

_{ir}− θ

_{il}] is the length of the interval on θ

_{i}under the i

^{th}hypothesis (i = 1, 2). (So, e.g., for H

_{1}: ${\theta}_{1}\in \left[0,1\right]$ vs. H

_{2}: ${\theta}_{2}$ = ½, this would yield d.f. = 1 + 1 + 0 = 2.) In our current notation, the binomial is expressed as the m = 2 one-simplex, and the length of the interval under H

_{2}is replaced by α as defined above, which defines the length of H

_{2}for the binomial, the area within H

_{2}for the trinomial, etc. For the composite vs. simple HC, we can either continue to consider d.f. as a baseline of 1 + the usual statistical d.f. (m − 1), or we can express d.f. simply as m. For the composite vs. composite HCs considered in this paper, d.f. then becomes m + α.

_{2}of size α, the point t = α divides l into two sections: t < α corresponds to data vectors inside the region defined by H

_{2}, while t > α corresponds to data vectors outside the H

_{2}region. The point t = α also demarcates a change in the behavior of V (Equation (9)), which decreases moving from t = 0 to t = α, and then increases from t = α to the boundary. Thus the minimum V, V

_{MIN}, occurs at the point t = α. (For small n and/or large α, V

_{MIN}occurs very slightly outside t = α, but with negligible numerical effects.) We then have:

_{1}follows the curvature of V to make V-b a linearly increasing function of t for t = [0, α] and g

_{2}is a (decreasing) straight line connecting t = α and t = 1 (Figure A1).

**Figure A1.**Illustration of the constant b in relationship to V, for the α = 0.6 composite vs. composite HC (n = 100).

**x/n**, or equivalently, the observed value of t. The first two factors have a clear connection to the thermodynamic constant b as it appears in the Van der Waals equation of state, where b “corrects” for minimum volume as a function of physical d.f.; the third factor may also be related to the thermodynamic constant, for systems in which the minimum volume is also a function of the temperature, since the observed value of t also corresponds to how “hot” the system is (that is, the size of E).

_{3}) as computed from Equation (7), when applied to the binomial composite vs. composite HC, is virtually identical to E as calculated for that same binomial HC in [9], as long as

**x/n**is outside the H

_{2}region, i.e., for t > α. However, when

**x/n**is inside the H

_{2}region (t < α), the new formula for b yields larger values of E (Figure A2). As noted above in connection with Figure 9, when

**x/n**is near the centroid, E in favor of H

_{2}is larger the larger is α, that is, the more incompatible the data are with H

_{1}. This seems like the desired behavior. However, we now notice that the original formula for b produced the opposite behavior in the binomial case for x/n close to ½; see Figure 3a in [9]. Thus the new definition of b produces better behavior at least in this one respect.

**Figure A2.**Comparison of E as computed in [9], denoted “E[9],” using the original version of b, with E as computed from Equation 7, using the revised version of b, in application to the binomial HC H

_{1}: $\theta \in {\Delta}^{1}$ vs. H

_{2}: $\theta \in {\Delta}_{\alpha}^{1}$, for (

**a**) α = 0.0, (

**b**) α = 0.2, (

**c**) α = 0.6.

m = 3 | m = 4 | |||||||
---|---|---|---|---|---|---|---|---|

n | Max E | Min E | Diff | Ratio | Max E | Min E | Diff | Ratio |

50 | 3.6510 | 3.6449 | 0.0061 | 0.0017 | 4.6245 | 4.6147 | 0.0098 | 0.0021 |

100 | 4.4949 | 4.4896 | 0.0053 | 0.0012 | 5.8440 | 5.8342 | 0.0098 | 0.0017 |

500 | 7.5096 | 7.5077 | 0.0019 | 0.0003 | 10.4134 | 10.4094 | 0.0040 | 0.0004 |

## References

- Evans, M. Measuring Statistical Evidence Using Relative Belief (Monographs on Statistics and Applied Probability); CRC Press: Boca Raton, FL, USA, 2015. [Google Scholar]
- Hacking, I. Logic of Statistical Inference; Cambridge University Press: London, UK, 1965. [Google Scholar]
- Edwards, A. Likelihood; Johns Hopkins University Press: Baltimore, MD, USA, 1992. [Google Scholar]
- Royall, R. Statistical Evidence: A Likelihood Paradigm; Chapman & Hall: London, UK, 1997. [Google Scholar]
- Vieland, V.J. Thermometers: Something for statistical geneticists to think about. Hum. Hered.
**2006**, 61, 144–156. [Google Scholar] [CrossRef] [PubMed] - Vieland, V.J. Where’s the Evidence? Hum. Hered.
**2011**, 71, 59–66. [Google Scholar] [CrossRef] [PubMed] - Vieland, V.J.; Hodge, S.E. Measurement of Evidence and Evidence of Measurement. Stat. App. Genet. Molec. Biol.
**2011**, 10. [Google Scholar] [CrossRef] - Weyl, H. Symmetry; Princeton University Press: Princeton, NJ, USA, 1952. [Google Scholar]
- Vieland, V.J.; Seok, S.-C. Statistical Evidence Measured on a Properly Calibrated Scale across Nested and Non-nested Hypothesis Comparisons. Entropy
**2015**, 17, 5333–5352. [Google Scholar] [CrossRef] - Vieland, V.J.; Das, J.; Hodge, S.E.; Seok, S.-C. Measurement of statistical evidence on an absolute scale following thermodynamic principles. Theory Biosci.
**2013**, 132, 181–194. [Google Scholar] [CrossRef] [PubMed] - Bickel, D.R. The strength of statistical evidence for composite hypotheses: Inference to the best explanation. Available online: http://biostats.bepress.com/cobra/art71/ (accessed on 24 March 2016).
- Zhang, Z. A law of likelihood for composite hypotheses. Available online: http://arxiv.org/abs/0901.0463 (accessed on 24 March 2016).
- Jeffreys, H. Theory of Probability (The International Series of Monographs on Physics); The Clarendon Press: Oxford, UK, 1939. [Google Scholar]
- Kass, R.E.; Raftery, A.E. Bayes Factors. J. Am. Stat. Assoc.
**1995**, 90, 773–795. [Google Scholar] [CrossRef] - Fermi, E. Thermodynamics; Dover Publications: New York, NY, USA, 1956. [Google Scholar]
- Stern, J.M.; Pereira, C.A.D.B. Bayesian epistemic values: Focus on surprise, measure probability! Logic J. IGPL
**2014**, 22, 236–254. [Google Scholar] [CrossRef] - Borges, W.; Stern, J. The rules of logic composition for the bayesian epistemic e-values. Logic J. IGPL
**2007**, 15, 401–420. [Google Scholar] [CrossRef] - Kullback, S. Information Theory and Statistics; Dover Publications: Mineola, NY, USA, 1968. [Google Scholar]
- Vieland, V.J. Evidence, temperature, and the laws of thermodynamics. Hum. Hered.
**2014**, 78, 153–163. [Google Scholar] [CrossRef] [PubMed] - Landauer, R. Irreversibility and heat generation in the computing process. IBM J. Res. Dev.
**1961**, 5, 183–191. [Google Scholar] [CrossRef] - Jaynes, E.T. Information Theory and Statistical Mechanics. Phys. Rev.
**1957**, 106, 620–630. [Google Scholar] [CrossRef] - Duncan, T.L.; Semura, J.S. The deep physics behind the second law: Information and energy as independent forms of bookkeeping. Entropy
**2004**, 6, 21–29. [Google Scholar] [CrossRef] - Caticha, A. Relative Entropy and Inductive Inference. Available online: http://arxiv.org/abs/physics/0311093 (accessed on 24 March 2016).
- Callen, H.B. Thermodynamics and an Introduction to Thermostatistics, 2nd ed.; John Wiley & Sons: New York, NY, USA, 1985. [Google Scholar]

**Figure 1.**Basic Behavior Patterns (BBPs) for evidence regarding whether a coin toss is fair or biased towards tails: (

**a**) Evidence as a function of changes in x/n for fixed n; (

**b**) evidence as a function of changes in n for fixed x/n; (

**c**) changes in n and x/n for fixed evidence. No calculations are done here and no specific values are assigned to the evidence. Only the basic shapes of the three curves, respectively, are important.

**Figure 2.**Behavior of the first evidence measure, e

_{1}, in application to the one-sided coin-tossing experiment (coin toss is fair vs. biased towards tails): (

**a**) e

_{1}as a function of changes in x/n for fixed n (the circle at x/n = 0.351 marks the TrP); (

**b**) e

_{1}as a function of changes in n for fixed x/n; (

**c**) changes in n and x/n for fixed e

_{1}. This figure, which is based on actual numerical calculations, exhibits the three basic behavior patterns illustrated in Figure 1.

**Figure 3.**Behavior of the TrP as a function of n for the one-sided binomial hypothesis contrast, illustrating that the TrP converges from the left toward x/n = ½ as n increases.

**Figure 4.**Behavior of alternative evidence measures as a function of n for fixed x/n, for the one-sided binomial hypothesis contrast: (

**a**) log MLR; (

**b**) log ALR, which is proportional to the Bayes ratio using a uniform prior. Neither statistic exhibits the concave down pattern shown in Figure 1b. The log MLR is either constant (for x/n = 0.5) or linear in n (thus the MLR itself is exponential in n); the log ALR behaves appropriately for x/n = 0.5 (it is < 0 and decaying more slowly as n increases), but for values of x/n supporting “coin is biased” it is slightly concave up.

**Figure 5.**The trinomial 2-simplex: (

**a**) basic orientation of the simplex in

**θ**= (θ

_{1}, θ

_{2}, θ

_{3}); (

**b**) illustration of permutation symmetry within the simplex; (

**c**) illustration of parameterization of line l used in subsequent figures.

**Figure 6.**Behavior of the second evidence measure, e

_{2}, in application to the composite vs. simple trinomial HC: (

**a**) e

_{2}as a function of changes in

**x/n**for fixed n, with the TrP indicated by the circle; (

**b**) e

_{2}as a function of changes in n for fixed

**x/n**; (

**c**) changes in n and

**x/n**for fixed e

_{2}. e

_{2}continues to exhibit all three basic behavior patterns illustrated in Figure 1 and Figure 2.

**Figure 7.**Illustration of composite vs. composite hypothesis contrast: (

**a**) α = 0.6, (

**b**) α = 0.2, (

**c**) α = 0. As shown, the composite vs. simple HC considered above is the special case α = 0.

**Figure 8.**Behavior of e

_{3}in application to the composite vs. composite trinomial HC, for α = 0.4: (

**a**) e

_{3}as a function of changes in

**x/n**for fixed n, with the TrP indicated by the circle; (

**b**) e

_{3}as a function of changes in n for fixed

**x/n**; (

**c**) changes in n and

**x/n**for fixed e

_{3}. e

_{3}continues to exhibit all three basic behavior patterns illustrated in Figure 1, Figure 2 and Figure 6.

**Figure 9.**Behavior of e

_{3}for two different values of α. Of particular note is the relative evidence at

**x/n**near the boundary (t = 1) and near the centroid (t = 0).

**Figure 10.**Behavior of E in application to a composite vs. simple trinomial HC as a function of

**x/n**: (

**a**) ordinary 3D view, (

**b**) representation on the 2-simplex with evidence strength represented by shading. The black line in (b) indicates the TrL, which is the bottom of the trough created in (a) as the evidence shifts (moving from the centroid out) from supporting H

_{2}to supporting H

_{1}.

**Figure 11.**Illustration of the transition plane (TrPL) for the quadrinomial composite vs. simple HC, H

_{1}: $\mathit{\theta}\in {\Delta}^{3}$ vs. H

_{2}:

**θ**= (¼, ¼, ¼, ¼). The 3-simplex consists of 4 planes, each of which is defined by 3 of the 4 vertices as labeled in (a). (

**a**) The TrPL along a single plane defined by the vertices (1,0,0,0), (0,1,0,0), and (0,0,1,0); or equivalently, the set of TrPs corresponding to all possible lines l extending from the centroid to that plane. (

**b**) The full TrPL considering all 4 planes, labeled in different colors.

**Figure 12.**Various proposed evidence measures plotted along the TrL, traveling from the TrP on the line l which extends to the boundary point

**x/n**= (1, 0, 0) a full 360° around the TrL. Only E is constant along the TrL.

© 2016 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons by Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Vieland, V.J.; Seok, S.-C.
Statistical Evidence Measured on a Properly Calibrated Scale for Multinomial Hypothesis Comparisons. *Entropy* **2016**, *18*, 114.
https://doi.org/10.3390/e18040114

**AMA Style**

Vieland VJ, Seok S-C.
Statistical Evidence Measured on a Properly Calibrated Scale for Multinomial Hypothesis Comparisons. *Entropy*. 2016; 18(4):114.
https://doi.org/10.3390/e18040114

**Chicago/Turabian Style**

Vieland, Veronica J., and Sang-Cheol Seok.
2016. "Statistical Evidence Measured on a Properly Calibrated Scale for Multinomial Hypothesis Comparisons" *Entropy* 18, no. 4: 114.
https://doi.org/10.3390/e18040114