Nonparametric Full Bayesian Significance Testing for Bayesian Histograms ^†

Abstract

In this article, we present an extension of the Full Bayesian Significance Test (FBST) for nonparametric settings, termed NP-FBST, which is constructed using the limit of finite dimension histograms. The test statistics for NP-FBST are based on a plug-in estimate of the cross-entropy between the null hypothesis and a histogram. This method shares similarities with Kullback–Leibler and entropy-based goodness-of-fit tests, but it can be applied to a broader range of hypotheses and is generally less computationally intensive. We demonstrate that when the number of histogram bins increases slowly with the sample size, the NP-FBST is consistent for Lipschitz continuous data-generating densities. Additionally, we propose an algorithm to optimize the NP-FBST. Through simulations, we compare the performance of the NP-FBST to traditional methods for testing uniformity. Our results indicate that the NP-FBST is competitive in terms of power, even surpassing the most powerful likelihood-ratio-based procedures for very small sample sizes.

1. Introduction and General Setting

Full Bayesian Significance Testing (FBST) [1] is a Bayesian method for testing if a parameter $\theta$ belongs to some set ${\mathrm{\Theta }}_0$. In traditional statistical setting, researchers analyze a collection of n observations ${\mathbf{X}}_n=(X_1,\dots ,X_n)$ that are presumed to conform to a specified distribution $f_{\theta }$ characterized by an unobserved parameter $\theta$. A Bayesian statistician makes inferences about $\theta$ by updating a prior density $\pi \left(\theta \right)$, supported by the set of all possibilities $\mathrm{\Theta }$. After observing ${\mathbf{X}}_n$, one obtains a posterior density $f_{\theta |{\mathbf{X}}_n}$. Often, one needs to determine whether $f_{\theta |{\mathbf{X}}_n}$ supports a scientific hypothesis framed with respect to $\theta$ belonging to some subset ${\mathrm{\Theta }}_0\subset \mathrm{\Theta }$, written $H_0:\theta \in {\mathrm{\Theta }}_0$. FBST tests $H_0$ by comparing the posterior density $f_{\theta |{\mathbf{X}}_n}$ of points inside and outside ${\mathrm{\Theta }}_0$. This comparison is represented by the posterior probability of the tangential set:

$$T\left({\mathrm{\Theta }}_0\right)=\{\theta :f_{\theta |X}\left(\theta \right)\le \underset{t\in {\mathrm{\Theta }}_0}{sup}{f}_{\theta |X}\left(t\right)\}.$$

(1)

$T\left({\mathrm{\Theta }}_0\right)$ encompasses all points in the parameter space that exhibit lower posterior density compared to those in ${\mathrm{\Theta }}_0$. The FBST methodology posits that if the posterior probability of $T\left({\mathrm{\Theta }}_0\right)$ is low, the hypothesis $H_0$ should be rejected, as it is located in a region characterized by low posterior density.

Definition 1.

In a standard Bayesian statistical model, let Θ be a finite dimensional parametric space, ${\mathbf{X}}_n$ an observed sample, $\mathcal{L}$ the likelihood function, π be the prior distribution, and $f_{\theta |{\mathbf{X}}_n}$ the posterior density proportional to $\pi \left(\theta \right)\mathcal{L}\left(\theta \right)$. Also, let ${\mathrm{\Pi }}_{\theta |{\mathbf{X}}_n}$ be the measure on Θ induced by $f_{\theta |{\mathbf{X}}_n}$. The Full Bayesian Significance Test (FBST) for testing $H_0:\theta \in {\mathrm{\Theta }}_0$ consists on rejecting the null hypothesis based on the e-value statistic

$$ev({\mathrm{\Theta }}_0;{\mathbf{X}}_n)={\mathrm{\Pi }}_{\theta |{\mathbf{X}}_n}\left(T\left({\mathrm{\Theta }}_0\right)\right)$$

(2)

where the tangential set $T\left({\mathrm{\Theta }}_0\right)$ is given by Equation (1). $H_0$ is rejected if $ev({\mathrm{\Theta }}_0;{\mathbf{X}}_n)<\alpha$ for some fixed significance level $\alpha \in [0,1]$.

In other words, the e-value quantifies the credibility of a hypothesis $H_0$ using the maximum probability argument, whereby a system is optimally represented by its most probable realization. This probability is defined as the posterior density $f_{\theta |{\mathbf{X}}_n}$, which quantifies the continuous probability associated with a specific point $\theta \in {\mathrm{\Theta }}_0$. The e-value directly addresses the question “What is the posterior probability of observing a $\theta$ with a posterior density exceeding that of any point in ${\mathrm{\Theta }}_0$?”. A higher e-value signifies that $H_0:\theta \in {\mathrm{\Theta }}_0$ is deemed more credible, whereas a lower e-value suggests that $H_0$ is considered less credible.

In this paper, we extend this concept to a nonparametric framework for density estimation using histograms. Bayesian nonparametric approaches for density estimation can be divided into two main categories. The first type focuses on defining priors ${\mathrm{\Pi }}_0$ in the infinite-dimensional space of probability densities. Upon observing the data ${\mathbf{X}}_n$, these priors are updated into infinite-dimensional posteriors, facilitating an adapted approach to Bayesian inference. Well-established examples of such priors include the Dirichlet Process Mixtures (DPM) and its extensions [2]. In contrast, the second type of Bayesian nonparametric approach employs regular finite-dimensional Bayesian modeling in parameter spaces ${\mathrm{\Theta }}_{k\left(n\right)}$ that maintain a fixed dimension $k\left(n\right)$, while allowing $k\left(n\right)$ for gradual expansion as the sample sizes increase. This includes truncated versions of infinite-dimensional priors and histograms with a fixed number of bins that increases with the sample size. This paper specifically examines a variant of the FBST applied in the context of increasing dimensionality.

In this paper, we propose an FBST for the problem of Bayesian density estimation using Dirichlet-Multinomial models, interpreted as histograms where the number of bins increases with the sample size. This methodology is in alignment with the Bayesian frameworks outlined by [3,4,5]. Therefore, we will use consistent notation to leverage results from the existing literature. The primary advantages of leveraging the Dirichlet-Multinomial model include (1) the feasibility of deriving an explicit formula for the FBST test statistic in a nonparametric context, (2) the implicit relation between the formula for the FBST test statistic and the differential entropy estimation, and (3) the potential to extend frequentist consistency results from the literature to this method. These attributes collectively establish a robust framework for nonparametric hypothesis testing that is mathematically rigorous, interpretable through the lens of information geometry, and consistent from a frequentist standpoint.

This paper is structured as follows. Section 2 outlines the essential definitions and properties of our proposed methodology. Section 3 provides simulations demonstrating the statistical power of our test. Finally, Section 4 offers a discussion of our findings and potential avenues for future research. The proof of our results are presented in Appendix A.

2. FBST for Random Histograms

We start with a formal definition of our model. To maintain clarity, we will restrict our analysis to densities on $[0,1]$.

Definition 2.

For $k\in \mathbb{N}$, consider the set of densities with support on $[0,1]$, defined as

$${\mathcal{H}}_k=\left\{f\in L_1\left([0,1]\right):f\left(t\right)=k\sum _{i=1}^k{\mathbf{1}}_{I_i}\left(t\right)w_i\mathit{and}\sum _{i=1}^k{w}_i=1,w_i\ge 0\right\}$$

where $I_j=[(j-1)/k,j/k)$ for $j=1,2,\dots ,k$. A random histogram θ is a random variable that selects a random element of ${\mathcal{H}}_{k\left(n\right)}$.

The distribution of $\theta$ is fully characterized by the distribution of the vector of random weights $W=(W_1,\dots ,W_k)$. Bayesian posterior inference on $\theta$ may also be conducted with respect to W if the likelihood is given by

$$\mathcal{L}\left(\theta \right)=\prod _{i=1}^n\theta \left(X_i\right)\propto w_1^{{\sum }_{i=1}^n{\mathbf{1}}_{I_1}\left(X_i\right)}\dots w_k^{{\sum }_{i=1}^n{\mathbf{1}}_{I_k}\left(X_i\right)}$$

which corresponds to the assumption that the sample values $X_1,\dots ,X_n|\theta$ are conditionally independent and share an identical density $\theta$. In this paper, we shall consider random histograms sampled implicitly by Dirichlet priors of the weights W. This approach guarantees that the posterior inference on $\theta$ is conjugate and computationally tractable, as it is equivalent to inference on a Dirichlet-Multinomial Bayesian model.

Proposition 1.

Consider θ a random histogram with weights $W\sim \mathit{Dirichlet}({\alpha }_1,\dots ,{\alpha }_k)$. If $X_i\perp X_j|\theta$, $i\ne j$ and $X_i|\theta \sim \theta$, then the posterior $\theta |{\mathbf{X}}_n$ remains a random histogram with weights

$$W|{\mathbf{X}}_n\sim \mathit{Dirichlet}({\alpha }_1+N_1,\dots ,{\alpha }_k+N_k)$$

(3)

where $N_j={\sum }_{i=1}^n{\mathbf{1}}_{I_j}\left(X_i\right),1\le j\le k.$

A usual approach for Bayesian nonparametric inference on a histogram is letting $k\left(n\right)$ grow slowly with the sample size n. This may be interpreted as a data-dependent prior; the full parameter space being considered is the set of all densities and, contingent on n, random histograms puts mass only on specific subsets of this set. One could define priors that do not depend on n, but this would come at a heavy computational cost. Moreover, meaningful and computationally sound inference might be conducted both in frequentist and Bayesian perspective if we also require the priors of w to depend on n [2,4].

Fixing n and $k\left(n\right)$, the original definition of $ev({\mathrm{\Theta }}_0;{\mathbf{X}}_n)$ may be adapted to conduct tests regarding $\theta |{\mathbf{X}}_n$. Given that there exists a bijection between an element of ${\mathcal{H}}_{k\left(n\right)}$ and its corresponding weights $(W_1,\dots ,W_{k\left(n\right)})$, the FBST test statistic may be defined in terms of the Dirichlet distribution defined in Equation (3). However, this approach comes at the price of being able to only test hypothesis of the form ${\mathrm{\Theta }}_0\subset {\mathcal{H}}_{k\left(n\right)}$. Therefore, if a researcher is interested in testing a hypothesis framed in terms of a general ${\mathrm{\Theta }}_0$, our proposed procedure specifies a test statistic based on its finite-dimensional counterpart. As $k\left(n\right)$ is permitted to increase with the sample size, this translation process becomes increasingly negligible.

Definition 3

(FBST for random histograms). Let θ be a random histogram defined by Dirichlet weights, and ${\mathbf{X}}_n|\theta$ represent an i.i.d. sample drawn from θ. The FBST test statistic for testing a hypothesis $H_0:\theta \in {\mathrm{\Theta }}_0$, where ${\mathrm{\Theta }}_0$ is an arbitrary set of densities on $[0,1]$, is given by

$$ev({\mathrm{\Theta }}_0;{\mathbf{X}}_n)={\mathrm{\Pi }}_{\theta |{\mathbf{X}}_n}\left(\sum _{j=1}^{k\left(n\right)}(N_j+{\alpha }_j)log\left(W_j\right)\le \underset{\mathbf{w}\in \mathcal{S}\left({\mathrm{\Theta }}_0\right)}{sup}\sum _{i=1}^{k\left(n\right)}(N_j+{\alpha }_j)log{w}_j\right)$$

(4)

where $\mathcal{S}\left({\mathrm{\Theta }}_0\right)$ denotes probabilities attributed to the sets $I_1,\dots ,I_{k\left(n\right)}$ by each element of ${\mathrm{\Theta }}_0$:

$$\mathcal{S}\left({\mathrm{\Theta }}_0\right)=\left\{\left({\int }_{I_1}f\left(t\right)dt,{\int }_{I_2}f\left(t\right)dt,\dots ,{\int }_{I_{k\left(n\right)}}f\left(t\right)dt\right):f\in {\mathrm{\Theta }}_0\right\}$$

The FBST for random histograms may be interpreted in the context of information theory. Let p and q be two m dimensional probability vectors. We recall that the cross-entropy divergence $H(p,q)$ between those vectors is given by $-{\sum }_{j=1}^m{p}_{i}log{q}_i$ and the Kullback–Leibler divergence $D_{KL}\left(p\right|\left|q\right)$ is given by $\sum p_{i}log\left({\displaystyle \frac{p_i}{q_i}}\right)$. By introducing ${\widehat{w}}_n=\left({\displaystyle \frac{N_1+{\alpha }_j}{n+{\sum }_{j=1}^{k\left(n\right)}{\alpha }_j}},\dots ,{\displaystyle \frac{N_{k\left(n\right)}+{\alpha }_j}{n+{\sum }_{j=1}^{k\left(n\right)}{\alpha }_j}}\right)$, Equation (3) can be articulated as

$$ev({\mathrm{\Theta }}_0;{\mathbf{X}}_n)={\mathrm{\Pi }}_{\theta |{\mathbf{X}}_n}\left(-H({\widehat{w}}_n,W)\le \underset{w\in \mathcal{S}\left({\mathrm{\Theta }}_0\right)}{sup}-H({\widehat{w}}_n,w)\right),$$

(5)

$$ev({\mathrm{\Theta }}_0;{\mathbf{X}}_n)={\mathrm{\Pi }}_{\theta |{\mathbf{X}}_n}\left(D_{KL}({\widehat{w}}_n\left|\right|W)\ge \underset{w\in \mathcal{S}\left({\mathrm{\Theta }}_0\right)}{inf}{D}_{KL}({\widehat{w}}_n\left|\right|w)\right).$$

(6)

These equations demonstrate that the application of the FBST definition leads to statistical tests grounded in an information-theoretic measure of divergence between a distribution of the sample into $k\left(n\right)$ bins and the expected value of counts on those same bins under the assumption that $\theta$ is some hypothesized density f. Indeed, in the context of this particular test, a related concept has emerged in the literature on goodness-of-fit testing, notably in G-tests [6] and other methodologies rooted in frequentist nonparametric estimates of the continuous variant of the Kullback–Leibler divergence for probability densities [7]. Both tests utilize a ${\chi }^2$ asymptotic distribution under the null hypothesis. For the FBST, there are specific rates of increase of $k\left(n\right)$ that ensure the presence of analogous results.

Theorem 1.

If (1) ${\mathbf{X}}_n$ is an independent and identically distributed sample of $X_1$ with density $f^{*}$ Lipchitz continuous on $[0,1]$; (2) θ is a random histogram satisfying $M>{\alpha }_i>m$ for all i and fixed quantities $m,M$ and (3) $k\left(n\right)={\displaystyle \frac{n^{1/6}}{{(logn)}^{1/6+ϵ}}}$, for any $ϵ>0$, then the FBST for random histograms with $H_0:\theta =f_0$ satisfies

1. 

$ev({\mathrm{\Theta }}_0;{\mathbf{X}}_n){\to }^{\mathcal{L}}\mathrm{Unif}(0,1)$ if $f_0=f^{*}$ and

2. 

$ev({\mathrm{\Theta }}_0;{\mathbf{X}}_n){\to }^{\mathbb{P}}0$ if $f_0\ne f^{*}$, where ${\to }^{\mathbb{P}}$ denotes convergence in probability with respect to ${\mathbf{X}}_n$.

One particular virtue of Equation (4) is the simplicity of the optimization step in the FBST. This fact is due the convexity of the cross-entropy functional. Also, this optimization will be able to reject false null hypotheses as the sample size grows larger, as we exemplify for the case of fixed parametric families.

Theorem 2.

Let $P_{\alpha }$ be a parametric family of differentiable distribution functions $f_{\alpha }$, such that $min\left|\right|f_{\alpha }-f^{*}{\left|\right|}_2^2=ϵ>0$ and ${\mathrm{\Theta }}_0^k=\{(F_{\alpha }\left(I_1\right),...,F_{\alpha }\left(I_{k\left(n\right)}\right)):F_{\alpha }\in P_{\alpha }\}$ is the corresponding subset of the $k\left(n\right)$ dimensional simplex. Then, the FBST on histograms for goodness-of-fit of this parametric family satisfies $ev\left({\mathrm{\Theta }}_0^k\right)\to 0$ in $f^{*}$ probability.

This procedure is similar to other nonparametric methods that do not rely on maximum likelihood estimates for testing, but instead optimize specific statistics. This idea dates back to Berkson’s suggestion to minimize chi-squared rather than maximize likelihood [8], although there have been few attempts to directly optimize test statistics, such as the Kolmogorov–Smirnov statistic [9]. This may be because optimizing usual test statistics for goodness-of-fit, such as Kolmogorov–Smirnov, Anderson–Darling, and Cramer–von Mises [10], requires specialized optimization procedures, like the one developed in [9].

Alternatively, the most common approach for testing adherence to a parametric family of distributions involves estimating parameters by maximum likelihood and then deriving the null distribution of an existing test through resampling [11]. Our new test, as we will demonstrate in simulations, could also require corrections when the optimization suggested by Theorem 2 is used.

3. Simulations

In this section, we will compare the statistical power of our test with that of other available techniques through simulations. Both simple and composite null hypotheses will be considered. Simulations will be conducted using the R programming language [12] and its public repository of packages. For simple hypotheses, the following tests are compared:

• The e-value for histograms, as defined in Definition 3, adopting with ${\alpha }_{i,j}=1$, with the number of bins defined as the hypothesis of Theorem 1;
• Classic Kolmogorov–Smirnov (KS) test, as described in [6];
• Alternative versions of KS, AD, and CV, constructed by [10], implemented by the R Package [13].

Following [10], we shall compare the NP-FBST of Definition 3 with

$$k\left(n\right)=max\{2,n^{1/6}log{\left(n\right)}^{-1/6-1}\}$$

using sample sizes $n=10,20,30,50,70,100,150,200,300$. The null hypothesis tested shall be $H_0:f^{*}=Unif(0,1)$ and $H_1$ will be simulated in 4 scenarios: $Beta(1.6,1.6)$, $Beta(1.3,1.3)$, $Beta(0.8,0.8)$, and $Beta(0.6,0.6)$. We calculate the statistical power as the % of correct rejection of $H_0$, with rejection at the $5\%$ level, on 500 Monte Carlo sample. The results are summarized in Figure 1.

Analyzing Figure 1, we observe the following:

• For $\alpha >1$ and $\beta >1$, the NP-FBST power may be much more powerful for small sample sizes, but it is still competitive for large sample sizes.
• For $\alpha <1$ and $\beta <1$, which are non-Lipchitz alternative hypotheses, the test performs worse than Zhang’s alternatives [10]. However, it still shows comparable or superior power compared to the usual Kolmogorov–Smirnov statistic.
• For sample sizes below 1000, $k\left(n\right)$ will usually be very small, such as 2 or 3, so the test is just a regular multinomial test.

To showcase the approximation properties of the NP-FBST based on Lemma A1, we shall simulate one last example, but this time we will adopt $k\left(n\right)={log}_2\left(n\right)$, usually referred as Sturge’s Law, for histogram binning [14]. This is, of course, appropriate asymptotically, as ${log}_2\left(n\right)\in o(n^{1/6}/log{\left(n\right)}^{-1/6-ϵ})$ for all $ϵ>0$. However, this produces very competitive statistical power for testing if a $Beta(2,2)$ is a $\text{Irwin-Hall}\left(2\right)$, as highlighted in Figure 2.

4. Conclusions

In this paper, we presented a new nonparametric Bayesian procedure extending the usual FBST for Bayesian histograms. We summarize our results as practical and theoretical.

On the practical and applied front, we draw the following conclusions:

• For small sample sizes, our method is competitive in terms of statistical power, even compared to sophisticated alternatives such as Zhang’s tests [10].
• For larger sample sizes, the very slow sample size growth required by Theorem 1 harms the statistical power. Therefore, other binning rules could be considered. Further research shall look for an adaptable number of bins. Desirable binning rules should be larger than $n^{1/6}log{\left(n\right)}^{1/6+ϵ}$ for small sample sizes, but smaller for large sample sizes. Our simulations suggest that the usual $k\left(n\right)=O\left({log}_2\left(n\right)\right)$, known as Sturge’s Law, is a competitive alternative for moderate sample sizes lower than 1000.
• Unlike previous attempts, our method is computationally inexpensive with competitive statistical power.

From a theoretical perspective, we derive the following conclusions:

• The natural Dirichlet-Multinomial formulation of Bayesian histograms induces statistical tests based on estimates of Kullback–Leibler divergences. This formulation logically follows from the definition of the FBST, and the same logic could be applied to other Bayesian density estimation methods. The frequentist properties of other versions of this NP-FBST remain to be studied in future works, but our results highlight the Kullback–Leibler divergence as a possible “canonical” statistic for nonparametric versions of the FBST for density estimation.
• Our results show that taking the limit of a slowly increasing finite-dimensional parameter space is a viable strategy for building nonparametric versions of the FBST. The frequentist properties of the FBST are intimately related to the Bernstein–von Mises theorem. Therefore, if these types of Gaussian approximations are available, our arguments should also hold. In fact, all the main references of this paper build specific growth rates of the dimension of the parameter space and could be used to find other versions of nonparametric FBSTs [3,4,5].
• For composite hypotheses, our method can be used both for testing based on the maximum likelihood estimate of nuisance parameters and for directly optimizing the test statistics, which may be interpreted as a weighted likelihood function. Usual numerical methods for optimizing the likelihood will work for our test statistic, which is not the case for other usual statistics such as Anderson–Darling, Cramer–von Mises, or Kolmogorov–Smirnov.

For future research, we highlight that adaptively choosing the number of bins $k\left(n\right)$ is crucial, as the statistical power is heavily influenced by this quantity. Additionally, Theorem 1 requires a Lipschitz continuous data-generating density, a usual assumption for histograms, but excludes unbounded densities, which are important from a practical and theoretical point of view. Extending our results to Hölder continuous densities is particularly important but requires the derivation of other versions of Bernstein–von Mises theorems.