# The Logical Consistency of Simultaneous Agnostic Hypothesis Tests

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

- Monotonicity: if A implies B, then a test that does not reject A should not reject B.
- Invertibility: A test should reject A if and only if it does not reject not-A.
- Union consonance: If a test rejects A and B, then it should reject $A\cup B$.
- Intersection consonance: If a test does not reject A and does not reject B, then it should not reject $A\cap B$.

## 2. Agnostic Testing Schemes

**Definition 1 (Agnostic Testing Scheme; ATS).**

**Example**

**1.**

**Example 2 (ATS based on posterior probabilities).**

**Example 3 (Likelihood Ratio Tests with fixed threshold).**

**Example 4 (FBST ATS).**

**Example 5 (Region Estimator-based ATS).**

**Example**

**6.**

**Example 7 (Region Likelihood Ratio ATS).**

**Example 8 (Region FBST ATS).**

## 3. Coherence Properties

#### 3.1. Monotonicity

**Definition 2 (Monotonicity).**

- if $\mathcal{L}$ accepts A, then it also accepts B.
- if $\mathcal{L}$ remains agnostic about A, then it either remains agnostic about B or accepts B.

**Example 9 (Tests based on posterior probabilities).**

**Example 10 (Likelihood Ratio Tests with fixed threshold).**

**Example 11 (FBST).**

**Theorem**

**1.**

**Example 12 (Region Estimator).**

#### 3.2. Union Consonance

**Definition 3 (Weak Union Consonance).**

**Remark**

**1.**

**Definition 4 (Strong Union Consonance).**

**Example 13 (Tests based on posterior probabilities).**

**Example 14 (Likelihood Ratio Tests with fixed threshold).**

**Example 15 (FBST).**

**Example 16 (Region Estimator).**

#### 3.3. Intersection Consonance

**Definition 5 (Weak Intersection Consonance).**

**Definition 6 (Strong Intersection Consonance).**

**Example 17 (Tests based on posterior probabilities).**

**Example 18 (Region Estimator).**

**Example 19 (ANOVA).**

#### 3.4. Invertibility

**Definition 7 (Invertibility).**

**Example 20 (Tests based on posterior probabilities).**

**Example 21 (Region Estimator).**

## 4. Satisfying All Properties

#### 4.1. Weak Desiderata

**Definition 8 (Weakly Consistent ATS).**

**Example 22 (Region Estimator).**

**Example 23 (Tests based on posterior probabilities).**

**Theorem**

**2.**

- 1.
- Monotonicity for every prior distribution if, and only if, for every $A,B\in \sigma \left(\mathsf{\Theta}\right)$ with $A\subseteq B,$ ${c}_{2}^{A}\ge {c}_{2}^{B}$ and ${c}_{1}^{A}\ge {c}_{1}^{B}$
- 2.
- Weak union consonance for every prior distribution if, and only if, for every $A,B\in \sigma \left(\mathsf{\Theta}\right)$ such that $A\ne B$, ${c}_{2}^{A}+{c}_{2}^{B}\le {c}_{1}^{A\cup B}$
- 3.
- Weak intersection consonance for every prior distribution if, and only if, for every $A,B\in \sigma \left(\mathsf{\Theta}\right)$ such that $A\ne B$, ${c}_{1}^{A}+{c}_{1}^{B}-1\ge {c}_{2}^{A\cap B}$
- 4.
- Invertibility for every prior distribution if, and only if, for every $A\in \sigma \left(\mathsf{\Theta}\right)$, ${c}_{1}^{A}=1-{c}_{2}^{{A}^{c}}$

#### 4.2. Strong Desiderata

**Definition 9 (Fully Consistent ATS).**

**Theorem**

**3.**

#### 4.3. n-Weak Desiderata

**Definition 10 (Weak n-union Consonance)**

**Definition 11 (Weak n-intersection Consonance)**

**Definition 12 (n-Weakly Consistent ATS)**

**Example 24 (Region Estimator).**

**Example 25 (Tests based on posterior probabilities).**

## 5. Decision-Theoretic Perspective

**Definition 13 (ATS generated by a family of loss functions).**

**Example 26 (Bayesian ATS generated by a family of error-wise constant loss functions).**

**Definition 14 (Proper losses).**

**Example 27 (Bayesian ATS generated by a family of error-wise constant loss functions).**

#### 5.1. Monotonicity

**Definition 15 (Relative Loss).**

**Definition 15 (Relative Loss).**

**Example**

**28.**

**Definition 17 (Balanced Relative Loss).**

**Lemma**

**1.**

**Theorem**

**4.**

**Example**

**29.**

#### 5.2. Union Consonance

**Definition**

**18.**

**no**$A,B\in \sigma \left(\mathsf{\Theta}\right)$, ${\theta}_{1},{\theta}_{2},{\theta}_{3}\in \mathsf{\Theta}$ and ${p}_{1},{p}_{2},{p}_{3}\ge 0$ such that ${p}_{1}+{p}_{2}+{p}_{3}=1$ and

**Theorem**

**5.**

**Example**

**30.**

**Definition 19 (Union consonance-balanced relative losses [7]).**

**Corollary**

**1.**

#### 5.3. Intersection Consonance

**Definition**

**20.**

**no**$A,B\in \sigma \left(\mathsf{\Theta}\right)$, ${\theta}_{1},{\theta}_{2},{\theta}_{3}\in \mathsf{\Theta}$ and ${p}_{1},{p}_{2},{p}_{3}\ge 0$ such that ${p}_{1}+{p}_{2}+{p}_{3}=1$ and

**Theorem**

**6.**

**Example**

**31.**

**Definition 21 (Intersection consonance-balanced relative losses [7]).**

**Corollary**

**2.**

#### 5.4. Invertibility

**Definition 22 (Invertible Relative Losses).**

**Theorem**

**7.**

**Example**

**32.**

## 6. Final Remarks

## Acknowledgments

## Author Contributions

## Conflicts of Interest

## Appendix A

**Proof of Theorem 1.**

- ${\mathcal{L}}_{A}\left(x\right)=0\Rightarrow {s}_{A}\left(x\right)>{c}_{1}\Rightarrow {s}_{B}\left(x\right)>{c}_{1}\Rightarrow {\mathcal{L}}_{B}\left(x\right)=0$.
- ${\mathcal{L}}_{A}\left(x\right)=\frac{1}{2}\Rightarrow {c}_{1}\ge {s}_{A}\left(x\right)>{c}_{2}\Rightarrow {s}_{B}\left(x\right)>{c}_{2}\Rightarrow {L}_{B}\left(x\right)\in \{0,\frac{1}{2}\}$.
- ${\mathcal{L}}_{A}\left(x\right)=1\Rightarrow {\mathcal{L}}_{B}\left(x\right)\le {\mathcal{L}}_{A}\left(x\right)=1.$

**Proof of Theorem 2.**

- If for every $A,B\in \sigma \left(\mathsf{\Theta}\right)$ with $A\subseteq B,$ ${c}_{2}^{A}\ge {c}_{2}^{B}$ and ${c}_{1}^{A}\ge {c}_{1}^{B}$, then for every $x\in \mathcal{X}$, $\mathbb{P}\left(A\right|x)>{c}_{2}^{A}\Rightarrow \mathbb{P}\left(B\right|x)>{c}_{2}^{B}$, and $\mathbb{P}\left(A\right|x)>{c}_{1}^{A}\Rightarrow \mathbb{P}\left(B\right|x)>{c}_{1}^{B}$. It follows that monotonicity holds.
- If for every $A,B\in \sigma \left(\mathsf{\Theta}\right)$ such that $A\ne B$, ${c}_{2}^{A}+{c}_{2}^{B}\le {c}_{1}^{A\cup B}$, then for every $x\in \mathcal{X}$, $\mathbb{P}\left(A\right|x)\le {c}_{2}^{A}$ and $\mathbb{P}\left(B\right|x)\le {c}_{2}^{B}$ implies that $\mathbb{P}(A\cup B|x)\le \mathbb{P}\left(A\right|x)+\mathbb{P}\left(B\right|x)={c}_{2}^{A}+{c}_{2}^{B}\le {c}_{1}^{A\cup B}$. It follows that union consonance holds.
- If for every $A,B\in \sigma \left(\mathsf{\Theta}\right)$ such that $A\ne B$, ${c}_{1}^{A}+{c}_{1}^{B}-1\ge {c}_{2}^{A\cap B}$, then for every $x\in \mathcal{X}$, $\mathbb{P}\left(A\right|x)>{c}_{1}^{A}$ and $\mathbb{P}\left(B\right|x)>{c}_{1}^{B}$ implies that $\mathbb{P}(A\cap B|x)\ge \mathbb{P}\left(A\right|x)+\mathbb{P}\left(B\right|x)-1>{c}_{1}^{A}+{c}_{1}^{B}-1\ge {c}_{2}^{A\cap B}$. It follows that intersection consonance holds.
- If for every $A\in \sigma \left(\mathsf{\Theta}\right)$, ${c}_{1}^{A}=1-{c}_{2}^{{A}^{c}}$, then for every $x\in \mathcal{X}$, $\mathbb{P}\left(A\right|x)\le {c}_{1}^{A}$ if, and only if, $\mathbb{P}\left({A}^{c}\right|x)\ge 1-{c}_{1}^{A}={c}_{2}^{{A}^{c}}$. Similarly, $\mathbb{P}\left(A\right|x)\le {c}_{2}^{A}$ if, and only if, $\mathbb{P}\left({A}^{c}\right|x)\ge 1-{c}_{2}^{A}={c}_{1}^{{A}^{c}}$. It follows that invertibility holds.

**Lemma**

**A1.**

**Proof of Lemma A1.**

**Proof of Theorem 3.**

**Lemma**

**A2.**

**Proof.**

**Lemma**

**A3.**

**Proof.**

**Theorem**

**A1.**

**Proof.**

**Lemma**

**A4.**

- If $E\left[{L}_{A}(1,\theta )\right|x]<E\left[{L}_{A}(\frac{1}{2},\theta )\right|x]$, then $E\left[{L}_{A}(1,\theta )\right|x]<E\left[{L}_{A}(0,\theta )\right|x]$.
- If $E\left[{L}_{A}(0,\theta )\right|x]<E\left[{L}_{A}(\frac{1}{2},\theta )\right|x]$, then $E\left[{L}_{A}(0,\theta )\right|x]<E\left[{L}_{A}(1,\theta )\right|x]$.

**Proof of Lemma A4.**

**Proof of Lemma 1.**

**Lemma**

**A5.**

**Proof of Lemma A5.**

**Proof of Theorem 4.**

**Proof of Theorem 5.**

**Proof of Corollary 1.**

**Proof of Theorem 6.**

**Proof of Corollary 2.**

**Proof of Theorem 7.**

## References

- Wiener, Y.; El-Yaniv, R. Agnostic selective classification. In Advances in Neural Information Processing Systems; MIT Press: Cambridge, MA, USA, 2011; pp. 1665–1673. [Google Scholar]
- Balsubramani, A. Learning to abstain from binary prediction. 2016; arXiv:1602.08151. [Google Scholar]
- Izbicki, R.; Esteves, L.G. Logical consistency in simultaneous statistical test procedures. Logic J. IGPL
**2015**, 23, 732–758. [Google Scholar] [CrossRef] - Finner, H.; Strassburger, K. The partitioning principle: A powerful tool in multiple decision theory. Ann. Stat.
**2002**, 30, 1194–1213. [Google Scholar] [CrossRef] - Sonnemann, E. General solutions to multiple testing problems. Biom. J.
**2008**, 50, 641–656. [Google Scholar] [CrossRef] [PubMed] - Patriota, A.G. S-value: An alternative measure of evidence for testing general null hypotheses. Cienc. Nat.
**2014**, 36, 14–22. [Google Scholar] - Da Silva, G.M.; Esteves, L.G.; Fossaluza, V.; Izbicki, R.; Wechsler, S. A bayesian decision-theoretic approach to logically-consistent hypothesis testing. Entropy
**2015**, 17, 6534–6559. [Google Scholar] [CrossRef] - Berg, N. No-decision classification: An alternative to testing for statistical significance. J. Socio-Econ.
**2004**, 33, 631–650. [Google Scholar] [CrossRef] - Babb, J.; Rogatko, A.; Zacks, S. Bayesian sequential and fixed sample testing of multihypothesis. In Asymptotic Methods in Probability and Statistics; Elsevier: Amsterdam, The Netherlands, 1998; pp. 801–809. [Google Scholar]
- Ripley, B.D. Pattern Recognition and Neural Networks; Cambridge University Press: Cambridge, UK, 1996. [Google Scholar]
- De Bragança Pereira, C.A.; Stern, J.M. Evidence and credibility: Full bayesian significance test for precise hypotheses. Entropy
**1999**, 1, 99–110. [Google Scholar] [CrossRef] - Berger, J.O.; Delampady, M. Testing precise hypotheses. Stat. Sci.
**1987**, 2, 317–335. [Google Scholar] [CrossRef] - Benjamini, Y.; Hochberg, Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. R. Stat. Soc.
**1995**, 57, 289–300. [Google Scholar] - Jaynes, E.T. Confidence intervals vs. Bayesian intervals. In Foundations of Probability Theory, Statistical Inference, and Statistical Theories of Science; Springer: Dodrecht, The Netherlands, 1976. [Google Scholar]
- Gabriel, K.R. Simultaneous test procedures—Some theory of multiple comparisons. Ann. Math. Stat.
**1969**, 41, 224–250. [Google Scholar] [CrossRef] - Fossaluza, V.; Izbicki, R.; da Silva, G.M.; Esteves, L.G. Coherent hypothesis testing. Am. Stat.
**2016**. submitted for publication. [Google Scholar] - Sonnemann, E.; Finner, H. Vollständigkeitssätze für multiple testprobleme. In Multiple Hypothesenprüfung; Bauer, P., Hommel, G., Sonnemann, E., Eds.; Springer: Berlin, Germany, 1988; pp. 121–135. (In German) [Google Scholar]
- Lavine, M.; Schervish, M. Bayes factors: What they are and what they are not. Am. Stat.
**1999**, 53, 119–122. [Google Scholar] - Izbicki, R.; Fossaluza, V.; Hounie, A.G.; Nakano, E.Y.; Pereira, C.A.B. Testing allele homogeneity: The problem of nested hypotheses. BMC Genet.
**2012**, 13. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Schervish, M.J. p values: What they are and what they are not. Am. Stat.
**1996**, 50, 203–206. [Google Scholar] [CrossRef] - Hochberg, Y.; Tamhane, A.C. Multiple Comparison Procedures; John Wiley & Sons: New York, NY, USA, 1987. [Google Scholar]
- Borges, W.; Stern, J.M. The rules of logic composition for the bayesian epistemic e-values. Logic J. IGPL
**2007**, 15, 401–420. [Google Scholar] [CrossRef] - Johnson, R.A.; Wichern, D.W. Applied Multivariate Statistical Analysis, 6th ed.; Pearson: Upper Saddle River, NJ, USA, 2007. [Google Scholar]
- Schervish, M.J. Theory of Statistics; Springer: New York, NY, USA, 1997. [Google Scholar]
- Robert, C. The Bayesian Choice: From Decision-Theoretic Foundations to Computational Implementation, 2nd ed.; Springer: New York, NY, USA, 2007. [Google Scholar]

**Figure 2.**Illustrations of the performance of the agnostic region testing scheme (Example 5) for three different hypotheses (specified on the top of each picture). The pictures present the probability of each decision, $\mathbb{P}\left(\mathcal{L}\right(A\left)\right(X)=d|\mu )$ for $d\in \{0,\frac{1}{2},1\}$, as a function of the mean, μ.

Decision | State of Nature | |
---|---|---|

$\theta \in A$ | $\theta \notin A$ | |

0 (accept A) | 0 | ${b}_{A}$ |

$\frac{1}{2}$ (remain agnostic about A) | ${a}_{A}$ | ${c}_{A}$ |

1 (reject A) | ${d}_{A}$ | 0 |

© 2016 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Esteves, L.G.; Izbicki, R.; Stern, J.M.; Stern, R.B.
The Logical Consistency of Simultaneous Agnostic Hypothesis Tests. *Entropy* **2016**, *18*, 256.
https://doi.org/10.3390/e18070256

**AMA Style**

Esteves LG, Izbicki R, Stern JM, Stern RB.
The Logical Consistency of Simultaneous Agnostic Hypothesis Tests. *Entropy*. 2016; 18(7):256.
https://doi.org/10.3390/e18070256

**Chicago/Turabian Style**

Esteves, Luís G., Rafael Izbicki, Julio M. Stern, and Rafael B. Stern.
2016. "The Logical Consistency of Simultaneous Agnostic Hypothesis Tests" *Entropy* 18, no. 7: 256.
https://doi.org/10.3390/e18070256