A Bayesian One-Sample Test for Proportion

Abstract

This paper deals with a new Bayesian approach to the one-sample test for proportion. More specifically, let $x=(x_1,\dots ,x_n)$ be an independent random sample of size n from a Bernoulli distribution with an unknown parameter $\theta$. For a fixed value ${\theta }_0$, the goal is to test the null hypothesis $H_0:\theta ={\theta }_0$ against all possible alternatives. The proposed approach is based on using the well-known formula of the Kullback–Leibler divergence between two binomial distributions chosen in a certain way. Then, the difference of the distance from a priori to a posteriori is compared through the relative belief ratio (a measure of evidence). Some theoretical properties of the method are developed. Examples and simulation results are included.

1. Introduction

The one-sample test for proportion is a topic that is taught in most introductory statistics courses. Let $x=(x_1,\dots ,x_n)$ be an independent random sample of size n from a Bernoulli distribution with unknown parameter $\theta$, where $\theta \in [0,1]$. The interest is to test the null hypothesis $H_0:\theta ={\theta }_0$, where ${\theta }_0$ is a known value.

In this problem, within the classical frequentist framework, the approximate z-test is commonly used for testing $H_0$. Specifically, if $n{\theta }_0\ge 5$ and $n(1-{\theta }_0)\ge 5$ (i.e., a large-sample test), the test statistic is given by

$$\begin{array}{c}\hfill z=\frac{\widehat{\theta }-{\theta }_0}{\sqrt{{\theta }_0(1-{\theta }_0)/n}},\end{array}$$

where $\widehat{\theta }={\sum }_{i=1}^n{x}_i/n$ represents the maximum likelihood estimator under $H_0$. For example, to test $H_0$ against the alternative hypothesis $H_1:\theta \ne {\theta }_0$, the two-sided p-value is equal to $2P(Z>|z\left|\right)$, where Z has the standard normal distribution. Then, $H_0$ is rejected if the p-value is less than a given significance level $\alpha$. As for a small-sample test, the exact binomial distribution (rather than a normal approximation) is typically used. We refer the reader to Chapter 12 of [1] for general methods of dealing with hypothesis testing problems.

A standard alternative method to the earlier procedures is the Bayesian approach by means of the Bayes factor [2,3]. This requires the specification of prior distributions for parameters. It is noticed that most of the Bayesian methods in this area are presented for comparing the proportion in two-sample problems. For instance, ref. [4] proposed an approximation for the Bayes factor to test the equality of two binomial proportions. Furthermore, ref. [5] provided some recommendations for using Bayes factors in testing the equality of two proportions to improve the sensitivity of the test. Several studies can also be found in the literature on developing the Bayesian two-sample proportion test in contingency tables for testing the independence between rows and columns. See, for example, ref. [6] and the references therein.

For the one-sample problem, ref. [7] used an improper prior for the proportion in clinical trials. They computed the probability of being under the null and alternative hypotheses and then compared them to decide whether to accept the null hypothesis. Ref. [8] developed the BayesFactor R package, which can be used to compute the Bayes factor for several research designs and hypotheses including a test for one-sample proportion. Ref. [9] discussed this problem in detail based on a direct computation of the relative belief ratio. In this paper, a different Bayesian approach to the one-sample test for proportion is proposed. First, to avoid a prior-data conflict (see Section 2), the uniform $[0,1]$ is placed as a prior on the true proportion, then the Kullback–Leibler divergence between two binomial distributions (sampling and hypothesized distributions selected in an appropriate way). The difference of the distance from a priori to a posteriori is compared such that, if the posterior distribution is more concentrated around the hypothesized distribution than the prior distribution, then there is evidence in favor of the null hypothesis. Furthermore, if the posterior distribution is less concentrated, then it indicates evidence against the null hypothesis. This comparison is made via a relative belief ratio, which measures the evidence in the observed data for or against the null hypothesis. We also discuss a measure of the strength of this evidence, and therefore, the proposed test is completely based on a direct measure of statistical evidence.

We highlight here some key motivations of the proposed technique. First, it is simple, and the computations are straightforward. Second, unlike the classical approach, it gives evidence in favor of the null hypothesis and does not require choosing the significance level. Third, it avoids any possibility of the prior for prior-data conflict and, hence, provides robustness in the analysis [10]. Fourth, the proposed method uses the relative belief and computes the strength separately, and it has been proven to work as a calibration of the relative belief [9]. Fifth, similar to the exact binomial test, the proposed test works for small sample sizes. Lastly, it can be extended to the case of the equality of k independent sample binomial proportions.

The remainder of this paper is organized as follows. Inferences using the relative belief ratio are introduced in Section 2. In Section 3, a Bayesian test is proposed. In this section, we also cover checking for prior-data conflict, prior elicitation, and the prior for bias. In Section 4, we consider several examples to illustrate the approach. A comparison with other methods is also considered in this section. Finally, some concluding remarks are given in Section 5.

2. Inferences Using Relative Belief

The relative belief ratio was developed in [9], and it has become a widely used tool in the theory of Bayesian statistical hypothesis testing. For some references, see, for example, [11,12,13,14]. Suppose a statistical model is given by the density function $\{f_{\theta }\left(x\right):\theta \in \mathsf{\Theta }\}$ with respect to Lebesgue measure on the parameter space $\mathsf{\Theta }$. Let $\pi \left(\theta \right)$ be a prior on $\mathsf{\Theta }$. After observing the data x, the posterior distribution of $\theta$ is given by the density $\pi \left(\theta \right|x)=\frac{f_{\theta }\left(x\right)\pi \left(\theta \right)}{m\left(x\right)},$ where $m\left(x\right)={\int }_{\mathsf{\Theta }}{f}_{\theta }\left(x\right)\pi \left(\theta \right)d\theta$. If the interest is to make an inference about the parameter $\theta$, when $\pi (·)$ and $\pi (·|x)$ are continuous at $\theta$, the relative belief ratio for a hypothesized value ${\theta }_0$ of $\theta$ is

$$RB\left({\theta }_0\right|x)=\frac{\pi \left({\theta }_0\right|x)}{\pi \left({\theta }_0\right)}.$$

(1)

If $\pi (·)$ and $\pi (·|x)$ are discrete, the relative belief ratio is defined through limits [9]. Clearly, (1) measures how beliefs have changed such that ${\theta }_0$ is the true value from a priori to a posteriori. Accordingly, $RB\left({\theta }_0\right|x)$ is a measure of evidence that ${\theta }_0$ is the true value. If $RB\left({\theta }_0\right|x)>1$, then the probability of ${\theta }_0$ being the true value increases from a priori to a posteriori, and so, there is evidence based on the data that ${\theta }_0$ is the true value; therefore, there is evidence in favor of ${\theta }_0$. If $RB\left({\theta }_0\right|x)<1$, then the probability of ${\theta }_0$ being the true value decreases from a priori to a posteriori. Thus, the data provide evidence against ${\theta }_0$ being the true value. When $RB\left({\theta }_0\right|x)=1$, then there is no evidence either way.

Let T denote a minimal sufficient statistic for $\{f_{\theta }:\theta \in \mathsf{\Theta }\}$ with density $f_{\theta T}\left(t\right)$ and $m_T\left(t\right)={\int }_{\mathsf{\Theta }}{f}_{\theta T}\left(t\right)\pi \left(\theta \right)d\theta$. Ref. [15] showed that

$$RB\left({\theta }_0\right|x)=\frac{f_{{\theta }_{0}T}\left(x\right)}{m_T\left(T\left(x\right)\right)}=\frac{f_{{\theta }_0}\left(x\right))}{m\left(x\right)}.$$

(2)

The relationship in (2) is known as the Savage–Dickey ratio.

The next step after computing (2) is to calibrate whether this value indicates strong or weak evidence for or against $H_0:\theta ={\theta }_0$. A standard calibration of $RB\left({\theta }_0\right|x)$ is obtained by computing the following probability [9]:

$$Str\left({\theta }_0\right|x)=\mathsf{\Pi }(RB\left(\theta \right|x)\le RB\left({\theta }_0\right|x)|x)={\int }_{\{\theta \in \mathsf{\Theta }:RB\left(\theta \right|x)\le RB\left({\theta }_0\right|x)\}}\pi \left(\theta \right|x)d\theta .$$

(3)

Here, $\mathsf{\Pi }(·|x)$ is the posterior cumulative distribution function with posterior density $\pi (·|x)$. As $Str\left({\theta }_0\right|x)$ represents the posterior probability that the true value of $\theta$ has a relative belief ratio of at most the relative belief ratio at ${\theta }_0$, when $RB\left({\theta }_0\right|x)<1,$ a small value for $Str\left({\theta }_0\right|x)$ gives strong evidence against $H_0$ (most of the values of $\theta$ have relative belief ratios greater than the relative belief ratio at ${\theta }_0$). On the other side, if $RB\left({\theta }_0\right|x)>1,$ a large value for $Str\left({\theta }_0\right|x)$ gives strong evidence in favor of ${\theta }_0$ (most of the values of $\theta$ have relative belief ratios smaller than the relative belief ratio at ${\theta }_0$), while a small value of $Str\left({\theta }_0\right|x)$ indicates weak evidence in favor of ${\theta }_0$.

Another basic issue when dealing with Bayesian inference methods is the possibility to bias the analysis by the prior. To assess any possible bias, following [9], when $H_0$ is true, the bias against $H_0:\theta ={\theta }_0$ can be measured by computing the prior probability:

$$M_T(RB\left({\theta }_0\right|x)\le 1|{\theta }_0),$$

(4)

where $M_T(·|{\theta }_0)$ denotes the prior predictive distribution of T when the true value is ${\theta }_0$. Large values of (4) imply there is a bias against $H_0$ and there is a concern about any evidence against $H_0$. On the other side, the bias in favor of $H_0$ is given by the prior probability

$$M_T(RB\left({\theta }_0\right|x)\ge 1|{\theta }_0^{\prime })$$

(5)

for values ${\theta }_0\ne {\theta }_0^{\prime }$ such that the difference between ${\theta }_0$ and ${\theta }_0^{\prime }$ represents the smallest difference of practical importance. When the bias in favor is large, then there is evidence in favor of $H_0$. As a result, any evidence in favor of $H_0$ is not convincing. For a fixed prior, both biases decrease as the sample size increases.

Another concern that appears in Bayesian analysis is whether a chosen prior may be strongly contradicted by the data [9]. A possible contradiction between the data and the prior is known as a prior-data conflict. A common method for checking the prior was developed by [16], which involves computing the probability:

$$\begin{array}{c}\hfill M_T\left(m_T\left(t\right)\le m_T\left(T\left(x\right)\right)\right),\end{array}$$

(6)

where T is a minimal sufficient statistic of the model and $M_T$ is the prior predictive probability distribution of T with density $m_T$ and is given by $M_T\left(A\right)={\int }_A{\int }_{\mathsf{\Theta }}\pi \left(\theta \right)f_{\theta T}\left(t\right)d\theta dt={\int }_A{m}_T\left(t\right)dt$. If (6) is small, then $T\left(x\right)$ lies in a region of low prior probability, such as a tail or anti-mode, which indicates a conflict. Furthermore, if (6) is large, then prior-data conflict should not be an issue. For different methods of checking prior-data conflict, see [17]. Furthermore, for relevant robustness results of inferences based on the relative belief ratio, see [10,18], where a strong connection between robustness and prior-data conflict is found.

3. One-Sample Bayesian Test for Proportion

3.1. The Approach

The proposed test is based on using the Kullback–Leibler (KL) distance. The KL distance between two discrete cumulative distribution functions (cdfs) P and Q defined on the same probability space with corresponding probability mass functions (pmfs) p and q (with respect to the counting measure) is defined by

$$\begin{array}{c}\hfill d(P,Q)=\sum _{x\in \phi }p\left(x\right)log\left(\frac{p\left(x\right)}{q\left(x\right)}\right).\end{array}$$

It is well known that $d_{KL}(P,Q)\ge 0$, and the equality holds if and only if $p=q$. However, it is not symmetric and does not satisfy the triangle inequality [19]. For example, if P and Q are the cdfs of binomial $(n,\theta )$ and binomial $(n,{\theta }_0)$, respectively, then the KL divergence between P and Q is given by

$$\begin{array}{c}\hfill d(P,Q)=n\left(\theta log\left({\displaystyle \frac{\theta }{{\theta }_0}}\right)+(1-\theta )log\left({\displaystyle \frac{1-\theta }{1-{\theta }_0}}\right)\right).\end{array}$$

(7)

As $d(P,Q)=0$ if and only if $\theta ={\theta }_0$, testing $H_0:\theta ={\theta }_0$ is equivalent to testing $H_0^{\prime }:d(P,Q)=0$. For the proposed Bayesian approach, to test $H_0:\theta ={\theta }_0$, let Beta(${\alpha }_0$, ${\beta }_0$) be the prior of $\theta$. Then, the posterior distribution of $\theta$ given $x=(x_1,\dots ,x_n)$ is Beta(${\alpha }_x$, ${\beta }_x$), where ${\alpha }_x={\alpha }_0+{\sum }_{i=1}^n{x}_i$ and ${\beta }_x={\beta }_0+n-{\sum }_{i=1}^n{x}_i$. If $\theta \sim$ Beta$({\alpha }_0,{\beta }_0)$, we denote $d(P,Q)$ by $D=d^{\mathrm{prior}}(P,Q)$, and if $\theta \sim$ Beta$({\alpha }_x,{\beta }_x)$, then we write $D_x=d^{\mathrm{post}}(P,Q)$. Note that, as $n\to \infty$, by the weak law of large numbers, $𝔼\left(\theta \right|x)=\frac{{\alpha }_x}{{\alpha }_x+{\beta }_x}=\frac{{\alpha }_0+{\sum }_{i=1}^n{x}_i}{n+{\alpha }_0+{\beta }_0}\stackrel{p}{\to }{\theta }_{\mathrm{true}}$, the true population proportion. Thus, by (7), if $H_0$ is true, we have $D_x\stackrel{a.s.}{\to }0$. On the other hand, if $H_0$ is not true, then

$$\begin{array}{c}\hfill D_x\stackrel{a.s.}{\to }c,\end{array}$$

where $c>0$. It follows that, if $H_0$ is true, then that distribution of $D_x$ should be concentrated about 0 more than D. Therefore, the proposed test is constructed based on a comparison of the distribution of D and $D_x$ about 0 via the relative belief ratio:

$$\begin{array}{c}\hfill R{B}_D\left(0\right|x)={\displaystyle \frac{{\pi }_{{}_D}\left(0\right|x)}{{\pi }_{{}_D}\left(0\right)}},\end{array}$$

(8)

where ${\pi }_{{}_D}\left(0\right)$ and ${\pi }_{{}_D}\left(0\right|x)$ are the probability density functions of D and $D_x$, respectively. If the distribution of $D_x$ is more concentrated about 0, then $R{B}_D\left(0\right|x)>1$; hence, this is evidence in favor of $H_0$, while if it is less concentrated about 0, then $R{B}_D\left(0\right|x)<1$, and there is evidence against $H_0$. Furthermore, the strength of evidence, whether for or against $H_0$, is also computed.

Note that, if $d=g\left(\theta \right)$ denotes the KL divergence function in (7), then

$${\pi }_D\left(d\right)=\left|\frac{1}{g_1^{\prime }\left(g_1^{-1}\left(d\right)\right)}\right|\pi \left(g_1^{-1}\left(d\right)\right)+\left|\frac{1}{g_2^{\prime }\left(g_2^{-1}\left(d\right)\right)}\right|\pi \left(g_2^{-1}\left(d\right)\right),$$

where $g_1$ and $g_2$ are the two parts of g on the left and right sides of ${\theta }_0$, respectively. See Figure 1 below.

It follows that

$$\begin{array}{c}\hfill R{B}_D\left(0\right|x)={\displaystyle \frac{\left|\frac{g_2^{\prime }\left(g_2^{-1}\left(d\right)\right)}{g_1^{\prime }\left(g_1^{-1}\left(d\right)\right)}\right|\pi \left(g_1^{-1}\left(d\right)\right|x)+\pi \left(g_2^{-1}\left(d\right)\right|x)}{\left|\frac{g_2^{\prime }\left(g_2^{-1}\left(d\right)\right)}{g_1^{\prime }\left(g_1^{-1}\left(d\right)\right)}\right|\pi \left(g_1^{-1}\left(d\right)\right)+\pi \left(g_2^{-1}\left(d\right)\right)}}.\end{array}$$

Now, as $d\to 0$, $\left|\frac{g_2^{\prime }\left(g_2^{-1}\left(d\right)\right)}{g_1^{\prime }\left(g_1^{-1}\left(d\right)\right)}\right|$ could $\to 0$, or $\to \infty$, or $\to c$, $c>0$. In either of theses three cases, $R{B}_D\left(0\right|x)=RB\left({\theta }_0\right|x)$. The difference between $R{B}_D\left(0\right|x)$ and $R{B}_D\left(0\right|x)$ may occur when $d\ne 0$. Accordingly, we develop Algorithm A in Section 3.4 to compute $R{B}_D\left(0\right|x)$ and the corresponding strength $St{r}_D\left(0\right|x)={\mathsf{\Pi }}_D(RB\left(d\right|x)\le RB\left(0\right|x)|x)$, where ${\mathsf{\Pi }}_D(·|x)$ is the cumulative distribution function of $D_x$.

3.2. Checking for Prior-Data Conflict

It is known that $T\left(x\right)={\sum }_{i=1}^n{x}_i$ is the minimal sufficient statistic for $\theta$. To check for prior-data conflict, we compute that tail probability in (6). If $\theta \sim \mathrm{Beta}({\alpha }_0,{\beta }_0)$, then the prior prediction function of $T\left(x\right)$ is given by

$$\begin{array}{ccc}\hfill m_T\left(t\right)& =& {\int }_0^1\left(\genfrac{}{}{0pt}{}{n}{t}\right){\theta }^n{(1-\theta )}^{n-t}\frac{\mathsf{\Gamma }({\alpha }_0+{\beta }_0)}{\mathsf{\Gamma }\left({\alpha }_0\right)\mathsf{\Gamma }\left({\beta }_0\right)}{\theta }^{{\alpha }_0-1}{(1-\theta )}^{{\beta }_0-1}d\theta \hfill \\ & =& \left(\genfrac{}{}{0pt}{}{n}{t}\right)\frac{\mathsf{\Gamma }({\alpha }_0+{\beta }_0)}{\mathsf{\Gamma }\left({\alpha }_0\right)\mathsf{\Gamma }\left({\beta }_0\right)}\frac{\mathsf{\Gamma }(t+{\alpha }_0)\mathsf{\Gamma }(n-t+{\beta }_0)}{\mathsf{\Gamma }(n+{\alpha }_0+{\beta }_0)}.\hfill \end{array}$$

We have

$$\begin{array}{ccc}\hfill & & M_T\left(m_T\left(x\right)\le m_T\left(T\left(x\right)\right)\right)\hfill \\ & =& M_T\left(\frac{\mathsf{\Gamma }(t+{\alpha }_0)\mathsf{\Gamma }(n-t+{\beta }_o)}{\mathsf{\Gamma }(t+1)\mathsf{\Gamma }(n-t+1)}\le \frac{\mathsf{\Gamma }(T\left(x\right)+{\alpha }_0)\mathsf{\Gamma }(n-T\left(x\right)+{\beta }_0)}{\mathsf{\Gamma }\left(T\right(x)+1)\mathsf{\Gamma }(n-T(x)+1)}\right).\hfill \end{array}$$

(9)

Clearly, computing (9) should be performed by simulation. For this, generate $\theta \sim \mathrm{Beta}({\alpha }_0,{\beta }_0)$. Then, generate $t\sim \mathrm{Binomial}(n,\theta )$. Repeat this many times, and record the proportion of values of $m_T\left(t\right)$ that are less than or equal to $m_T\left(T\left(x\right)\right)$ giving a Monte Carlo estimate of (9). Clearly, when ${\alpha }_0={\beta }_0=1$, then both sides of the inequality in (9) are equal and the probability is 1. That is, with Beta$(1,1)$ prior (i.e., the uniform $[0,1]$ prior), there is never any prior-data conflict. Accordingly, we choose the prior to be ${\alpha }_0={\beta }_0=1$, which will relax the need to check for prior-data conflict.

The following proposition finds the expected value of D and $D_x$ for the uniform $[0,1]$ prior. This proposition could be used, for example, to compare the theoretical and the computed values of D and $D_x$.

Proposition 1.

Let $D=d^{prior}(P,Q)$ and $D_x=d^{post}(P,Q)$ be as defined in Section 3.1:

(i)

If $\theta \sim$ Beta$(1,1)$, then

$$\begin{array}{c}\hfill E\left(d^{\mathrm{prior}}(P,Q)\right)=-\frac{n\left(1+log({\theta }_0(1-{\theta }_0)\right)}{2}.\end{array}$$

(ii)

If $\theta \sim$ Beta$({\alpha }_x=1+{\sum }_{i=1}^n{x}_i,{\beta }_x=1+n-{\sum }_{i=1}^n{x}_i)$, then

$$\begin{array}{c}\hfill E\left(d^{post}(P,Q)\right)=\frac{n}{{\alpha }_x+{\beta }_x+1}[{\alpha }_x(\psi ({\alpha }_x+1)-\psi ({\alpha }_x+{\beta }_x+1)\\ \hfill -log\left({\theta }_0\right))+{\beta }_x\left(\psi ({\beta }_x+1)-\psi ({\alpha }_x+{\beta }_x+1)-log(1-{\theta }_0)\right)],\end{array}$$

where $\psi \left(x\right)=\frac{d}{dx}log\mathsf{\Gamma }\left(x\right)$ is the digamma function.

Proof. 

Note,

$$\begin{array}{ccc}\hfill E\left(d^{\mathrm{prior}}\right)& =& n{\int }_0^1\left[\theta log\left({\displaystyle \frac{\theta }{{\theta }_0}}\right)+(1-\theta )log\left({\displaystyle \frac{1-\theta }{1-{\theta }_0}}\right)\right]d\theta .\hfill \end{array}$$

The proof (i) follows by using integration by parts.

To prove (ii), notice that

$$\begin{array}{ccc}\hfill d^{\mathrm{post}}(P,Q)& =& n[\theta log(\theta )+(1-\theta )log(1-\theta )\hfill \\ & & -\theta log\left({\theta }_0\right)-(1-\theta )log(1-{\theta }_0)].\hfill \end{array}$$

(10)

Now,

$$\begin{array}{ccc}\hfill E(\theta log(\theta \left)\right)& =& {\int }_0^1\theta log\left(\theta \right){\displaystyle \frac{\mathsf{\Gamma }({\alpha }_x+{\beta }_x)}{\mathsf{\Gamma }\left({\alpha }_x\right)\mathsf{\Gamma }\left({\beta }_x\right)}}{\theta }^{{\alpha }_x-1}{(1-\theta )}^{{\beta }_x-1}d\theta \hfill \\ \hfill & & ={\displaystyle \frac{\mathsf{\Gamma }({\alpha }_x+{\beta }_x)}{\mathsf{\Gamma }\left({\alpha }_x\right)\mathsf{\Gamma }\left({\beta }_x\right)}}{\int }_0^{1}log\left(\theta \right){\theta }^{{\alpha }_x}{(1-\theta )}^{{\beta }_x-1}d\theta \hfill \\ \hfill & & ={\displaystyle \frac{\mathsf{\Gamma }({\alpha }_x+{\beta }_x)}{\mathsf{\Gamma }\left({\alpha }_x\right)\mathsf{\Gamma }\left({\beta }_x\right)}}{\int }_0^1{\displaystyle \frac{\partial }{\partial {\alpha }_x}}\left({\theta }^{{\alpha }_x}{(1-\theta )}^{{\beta }_x-1}\right)d\theta \hfill \\ & & ={\displaystyle \frac{\mathsf{\Gamma }({\alpha }_x+{\beta }_x)}{\mathsf{\Gamma }\left({\alpha }_x\right)\mathsf{\Gamma }\left({\beta }_x\right)}}{\displaystyle \frac{\partial }{\partial {\alpha }_x}}\left({\theta }^{{\alpha }_x}{(1-\theta )}^{{\beta }_x-1}\right)\hfill \\ \hfill & & ={\displaystyle \frac{{\alpha }_x}{{\alpha }_x+{\beta }_x}}{\displaystyle \frac{\mathsf{\Gamma }({\alpha }_x+{\beta }_x+1)}{\mathsf{\Gamma }({\alpha }_x+1)\mathsf{\Gamma }\left({\beta }_x\right)}}{\displaystyle \frac{\partial }{\partial {\alpha }_x}}\left({\displaystyle \frac{\mathsf{\Gamma }({\alpha }_x+1)\mathsf{\Gamma }\left({\beta }_x\right)}{\mathsf{\Gamma }({\alpha }_x+{\beta }_x+1)}}\right)\hfill \\ \hfill & & ={\displaystyle \frac{{\alpha }_x}{{\alpha }_x+{\beta }_x}}{\displaystyle \frac{\partial }{\partial {\alpha }_x}}log\left({\displaystyle \frac{\mathsf{\Gamma }({\alpha }_x+1)\mathsf{\Gamma }\left({\beta }_x\right)}{\mathsf{\Gamma }({\alpha }_x+{\beta }_x+1)}}\right)\hfill \\ \hfill & & ={\displaystyle \frac{{\alpha }_x}{{\alpha }_x+{\beta }_x}}\left[\psi ({\alpha }_x+1)-\psi ({\alpha }_x+{\beta }_x+1)\right]\hfill \end{array}$$

(11)

Similarly,

$$\begin{array}{c}\hfill E\left((1-\theta )log(1-\theta )\right)={\displaystyle \frac{{\beta }_x}{{\alpha }_x+{\beta }_x}}\left[\psi ({\beta }_x+1)-\psi ({\alpha }_x+{\beta }_x+1)\right].\end{array}$$

(12)

Furthermore,

$$\begin{array}{ccc}\hfill E\left(\theta log{\theta }_0\right)& =& log\left({\theta }_0\right)E\left(\theta \right)\hfill \\ & =& {\displaystyle \frac{{\alpha }_x}{{\alpha }_x+{\beta }_x}}log\left({\theta }_0\right)\hfill \end{array}$$

(13)

and

$$\begin{array}{ccc}\hfill E\left((1-\theta )log(1-{\theta }_0)\right)& =& log(1-{\theta }_0)E(1-\theta )\hfill \\ & =& (1-{\theta }_0){\displaystyle \frac{{\beta }_x}{{\alpha }_x+{\beta }_x}}\hfill \end{array}$$

(14)

Now, combining the above expectations (12)–(14) in (10) gives the result.    □

3.3. Checking the Prior for Bias

As mentioned in Section 2, there is a need to check if the prior Beta(1,1) induces any bias in the problem. Note that, with this prior, the relative belief ratio $RB\left(\theta \right|x)$ is derived by [9]:

$$\begin{array}{c}\hfill RB\left(\theta \right|x)=\frac{\mathsf{\Gamma }(n+2){\theta }^{{\sum }_{i=1}^n{x}_i}{(1-\theta )}^{n-{\sum }_{i=1}^n{x}_i}}{\mathsf{\Gamma }(1+{\sum }_{i=1}^n{x}_i)\mathsf{\Gamma }(1+n-{\sum }_{i=1}^n{x}_i)}.\end{array}$$

(15)

The bias against the hypothesis $H_0:\theta ={\theta }_0$ is measured by computing (4) using $RB\left(\theta \right|x)$ as given in (15). Here, $M_T(·|{\theta }_0)$ is the $\mathrm{Binomial}(n,{\theta }_0)$ distribution. On the other hand, the bias in favor of the hypothesis $H_0:\theta ={\theta }_0$ is measured by computing (5) when ${\theta }_0\ne {\theta }_0^{\prime }$ is true and $RB\left({\theta }_0^{\prime }\right|x)$ is as defined in (15). Here, $M_T(·|{\theta }_0^{\prime })$ is the $\mathrm{Binomial}(n,{\theta }_0^{\prime })$ distribution. The interpretation of the bias was given in Section 2.

To compute the bias in this problem, it should be performed via simulation. For example, to compute the bias against, generate $t\sim \mathrm{Binomial}(n,{\theta }_0)$, and compute $RB\left({\theta }_0\right|x)$ as defined in (15). Repeat this many times, and record the proportion of values of $RB\left({\theta }_0\right|x)$ that are less than 1. The bias in favor can be estimated similarly.

3.4. The Algorithm

As ${\pi }_{{}_D}(·|x)$ and ${\pi }_{{}_D}(·)$ in (8) have no closed forms, the relative belief ratio and the strength need to be estimated via approximation. The following algorithm summarizes the steps required to test $H_0:\theta ={\theta }_0$.

Algorithm A (RB test for proportion).

(i)

Generate $\theta$ from Beta$(1,1)$, and compute D.

(ii)

Repeat Step (ii) to obtain a sample of $r_1$ values of D.

(iii)

Generate $\theta |x$ from Beta$({\alpha }_x=1+{\sum }_{i=1}^n{x}_i,{\beta }_x=1+n-{\sum }_{i=1}^n{x}_i)$, and compute $D_x$.

(iv)

Repeat Step (iv) to obtain a sample of $r_2$ values of $D_x$.

(v)

Compute the relative belief ratio and the strength as follows:

(a)

Let L be a positive number. Let ${\widehat{F}}_D$ denote the empirical cdf of D based on the prior sample in (3), and for $i=0,\dots ,L,$ let ${\widehat{d}}_{i/L}$ be the estimate of $d_{i/L},$ the $(i/L)$-prior quantile of D. Here, ${\widehat{d}}_0=0$, and ${\widehat{d}}_1$ is the largest value of D. Let ${\widehat{F}}_D(·|x)$ denote the empirical cdf based on $D_x$. For $d\in [{\widehat{d}}_{i/L},{\widehat{d}}_{(i+1)/L})$, estimate $R{B}_D\left(d\right|x)={\pi }_D\left(d\right|x)/{\pi }_D\left(d\right)$ by

$${\widehat{RB}}_D\left(d\right|x)=L\{{\widehat{F}}_D({\widehat{d}}_{(i+1)/L}|x)-{\widehat{F}}_D({\widehat{d}}_{i/L}|x)\},$$

(16)

the ratio of the estimates of the posterior and prior contents of $[{\widehat{d}}_{i/L},{\widehat{d}}_{(i+1)/L}).$ Thus, we estimate $R{B}_D\left(0\right|x)={\pi }_D\left(0\right|x)/{\pi }_D\left(0\right)$ by ${\widehat{RB}}_D\left(0\right|x)=$$L{\widehat{F}}_D({\widehat{d}}_{p_0}|x)$, where $p_0=i_0/L$ and $i_0$ are chosen so that $i_0/L$ is not too small (typically, $i_0/L\approx 0.05)$.

(b)

Estimate the strength ${\mathsf{\Pi }}_D(R{B}_D\left(d\right|x)\le R{B}_D\left(0\right|x)|x)$ by the finite sum:

$$\sum _{\{i\ge i_0:{\widehat{RB}}_D\left({\widehat{d}}_{i/L}\right|x)\le {\widehat{RB}}_D\left(0\right|x)\}}({\widehat{F}}_D\left({\widehat{d}}_{(i+1)/L}\right|x)-{\widehat{F}}_D\left({\widehat{d}}_{i/L}\right|x)).$$

(17)

For fixed $L,$ as $r_1\to \infty ,r_2\to \infty ,$ then ${\widehat{d}}_{i/L}$ converges almost surely to $d_{i/L}$, and (16) and (17) converge almost surely to $R{B}_D\left(d\right|x)$ and ${\mathsf{\Pi }}_D(R{B}_D\left(d\right|x)\le R{B}_D\left(0\right|x)|x)$, respectively. See [20] for the details.

4. Examples

The approach is illustrated through two examples. In Algorithm A, we set $L=20$ and $i_0=1$. For the sake of comparison, the Bayes factor (BF) of [8] and the p-values of the exact binomial test and the approximate z-test were also considered. The R package BayesFactor was used to compute the BF of [21], while the R functions binom.test() and prop.test() were used to compute the p-values of the exact and the approximate tests, respectively. For these tests, we set $\alpha =0.05$. To interpret the Bayes factor, we used the following rule of thumb introduced by Jefferys in [22]. See also [23]. This scale gives evidence against $H_0$. To this end, if $BF=1$, no evidence; $1<BF<3$, anecdotal; $3<BF\le 10$, moderate; $10<BF\le 30$, strong; $30<BF\le 100$, very strong; and $BF>100$, extreme. On the other side, if, for example, $BF=0.2$, then 1/0.2 = 45 is interpreted as there is moderate evidence in favor of $H_0$.

Example 1.

(Clinical trial; [24]).

A method currently used by doctors to screen women for breast cancer fails to detect cancer in 20% of the women who actually have the disease. A new method has been developed that researchers hope will detect cancer more accurately. This new method was used to screen a random sample of 140 women known to have breast cancer. Of these, the new method failed to detect cancer in 12 women. The researchers want to test if the sample provides evidence that the failure rate of the new method differs from the one currently in use.

First, the bias in the prior is assessed by computing (4) with $\theta ={\theta }_0=0.2$. In this case, the bias against the null hypothesis is 0.2233. On the other hand, the bias in favor of the null hypothesis is measured by computing (5) with ${\theta }_0=0.2\pm 0.05$, which gives 0.2706 (for ${\theta }_0=0.25$) and 0.01737 (for ${\theta }_0=0.15$). This shows relatively equal bias either for or against the null hypothesis for the uniform $[0,1]$ prior. The results of the tests are reported in

Table 1. Test results of Example 1.

Test	Values	Decision
RB (strength)	0.0400 (0.0021)	Strong evidence to reject $H_0$
RB (strength) using (15)	0.0166 (0.0002)	Strong evidence to reject $H_0$
BF	167.8429	Extreme evidence to reject $H_0$
p-value (exact)	0.0003	Reject $H_0$
p-value (approximate)	0.0012	Reject $H_0$

. In particular, the relative belief ratio of the proposed test is 0.0040, which is less than 1, and the strength is 0.0021, which is close to 0. Hence, there is strong evidence to conclude that the new female breast cancer detection method is better than the used method.

Example 2.

(Production defective; [25]).

A machine in a factory must be repaired if it produces more than 10% defective items among the large lot of items that it produces in a day. A random sample of 100 items from the day’s production contains 15 defective items, and the supervisor says that the machine must be repaired. The goal is to test if the sample provides evidence to support his/her decision.

As in Example 1, the bias in the prior is assessed by computing (4) with $\theta ={\theta }_0=0.1$. In this case, the bias against the null hypothesis is 0.1182. On the other hand, the bias in favor of the null hypothesis is measured by computing (5) with ${\theta }_0=0.1\pm 0.05$, which gives 0.0504 (for ${\theta }_0=0.15$) and 0.0204 (for ${\theta }_0=0.05$). This shows an acceptable bias either for or against the null hypothesis for the uniform $[0,1]$ prior. The results of the tests are reported

Table 2. Test results of Example 2.

Test	Values	Decision
RB (strength)	4.094 (0.406)	Moderate evidence in favor of $H_0$
RB (strength) using (15)	2.790 (0.1177)	Moderate/weak evidence in favor of $H_0$
BF	0.9567	Anecdotal evidence for $H_0$
p-value (exact)	0.0962	Fail to reject $H_0$
p-value (approximate)	0.1336	Fail to reject $H_0$

. As $RB=4.094>1$ and the strength is 0.406, there is moderate evidence to conclude that the factory produces 10% defective items in a day.

5. Simulation Study

In this section, we run a Monte Carlo simulation study of size 1000 to determine the proportion of false positives (false rejection of the null hypothesis) and the proportion of true positives (true rejection of the null hypothesis) when random data with various sample sizes $n\in \{10,30,50,100\}$ are generated from the Bernoulli distribution with parameter $\theta$, where $\theta \in \{0,0.05,0.1,\dots ,0.9,0.95,1\}$. In Figure 2, we plot $\theta$ against the proportion of rejecting $H_0:\theta =0.5$ (i.e., when $RB<1$). Figure 2 reveals that when $\theta$ is close to 0.5 (i.e., $H_0$ is true), the proportion of rejecting $H_0$ is close to 0. However, as $\theta$ moves away from 0.5, the curves move toward 1. Furthermore, as the sample size increases, the computed proportions converge faster to the true values.

We also conducted a simulation study to compare the performance of the RB test, the BF test, and the exact and approximate tests. The proportion of rejecting $H_0$ of the exact and approximate tests is computed by the number of times the p-value is less than $0.05$ divided by 1000. For sample sizes $n\in \left\{10,30,50,100\right\}$, we generated data from Bernoulli $\left({\theta }_0\right)$, where ${\theta }_0$ is given in

Table 3. The proportion of rejecting $H_0$ out of 1000 replications based on samples of size $n=10$ and $n=30$ is reported. Here, ${}^{†}$ represents the ratio of false positives for the tests and ${}^{‡}$ represents the ratio of the true positives.

n = 10
Proportion	$\mathit{RB}$	$\mathit{BF}$	Exact Test	z-Test
${\theta }_0=0.1$	0.922 ${}^{‡}$	0.984 ${}^{‡}$	0.719 ${}^{‡}$	0.835 ${}^{‡}$
${\theta }_0=0.15$	0.832 ${}^{‡}$	0.934 ${}^{‡}$	0.535 ${}^{‡}$	0.689 ${}^{‡}$
${\theta }_0=0.2$	0.696 ${}^{‡}$	0.869 ${}^{‡}$	0.368 ${}^{‡}$	0.530 ${}^{‡}$
${\theta }_0=0.25$	0.539 ${}^{‡}$	0.772 ${}^{‡}$	0.236 ${}^{‡}$	0.381 ${}^{‡}$
${\theta }_0=0.3$	0.428 ${}^{‡}$	0.654 ${}^{‡}$	0.150 ${}^{‡}$	0.256 ${}^{‡}$
${\theta }_0=0.35$	0.276 ${}^{‡}$	0.534 ${}^{‡}$	0.081 ${}^{‡}$	0.161 ${}^{‡}$
${\theta }_0=0.4$	0.214 ${}^{‡}$	0.437 ${}^{‡}$	0.044 ${}^{‡}$	0.098 ${}^{‡}$
${\theta }_0=0.45$	0.149 ${}^{‡}$	0.351 ${}^{‡}$	0.029 ${}^{‡}$	0.062 ${}^{‡}$
${\theta }_0=0.5$	0.133 ${}^{†}$	0.341 ${}^{†}$	0.021 ${}^{†}$	0.050 ${}^{†}$
${\theta }_0=0.55$	0.148 ${}^{‡}$	0.382 ${}^{‡}$	0.034 ${}^{‡}$	0.062 ${}^{‡}$
${\theta }_0=0.6$	0.230 ${}^{‡}$	0.441 ${}^{‡}$	0.06 ${}^{‡}$	0.098 ${}^{‡}$
${\theta }_0=0.65$	0.315 ${}^{‡}$	0.530 ${}^{‡}$	0.088 ${}^{‡}$	0.161 ${}^{‡}$
${\theta }_0=0.7$	0.402 ${}^{‡}$	0.663 ${}^{‡}$	0.139 ${}^{‡}$	0.256 ${}^{‡}$
${\theta }_0=0.75$	0.526 ${}^{‡}$	0.771 ${}^{‡}$	0.240 ${}^{‡}$	0.381 ${}^{‡}$
${\theta }_0=0.8$	0.694 ${}^{‡}$	0.887 ${}^{‡}$	0.390 ${}^{‡}$	0.530 ${}^{‡}$
${\theta }_0=0.85$	0.818 ${}^{‡}$	0.950 ${}^{‡}$	0.551 ${}^{‡}$	0.689 ${}^{‡}$
${\theta }_0=0.9$	0.937 ${}^{‡}$	0.988 ${}^{‡}$	0.704 ${}^{‡}$	0.835 ${}^{‡}$
${\theta }_0=0.95$	0.990 ${}^{‡}$	0.998 ${}^{‡}$	0.906 ${}^{‡}$	0.943 ${}^{‡}$
n = 30
${\theta }_0=0.1$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$	0.999 ${}^{‡}$
${\theta }_0=0.15$	0.998 ${}^{‡}$	0.999 ${}^{‡}$	0.993 ${}^{‡}$	0.989 ${}^{‡}$
${\theta }_0=0.2$	0.953 ${}^{‡}$	0.974 ${}^{‡}$	0.929 ${}^{‡}$	0.941 ${}^{‡}$
${\theta }_0=0.25$	0.864 ${}^{‡}$	0.884 ${}^{‡}$	0.791 ${}^{‡}$	0.818 ${}^{‡}$
${\theta }_0=0.3$	0.652 ${}^{‡}$	0.712 ${}^{‡}$	0.574 ${}^{‡}$	0.615 ${}^{‡}$
${\theta }_0=0.35$	0.453 ${}^{‡}$	0.502 ${}^{‡}$	0.352 ${}^{‡}$	0.386 ${}^{‡}$
${\theta }_0=0.4$	0.278 ${}^{‡}$	0.297 ${}^{‡}$	0.188 ${}^{‡}$	0.197 ${}^{‡}$
${\theta }_0=0.45$	0.121 ${}^{‡}$	0.153 ${}^{‡}$	0.079 ${}^{‡}$	0.085 ${}^{‡}$
${\theta }_0=0.5$	0.078 ${}^{†}$	0.101 ${}^{†}$	0.037 ${}^{†}$	0.050 ${}^{†}$
${\theta }_0=0.55$	0.114 ${}^{‡}$	0.152 ${}^{‡}$	0.072 ${}^{‡}$	0.085 ${}^{‡}$
${\theta }_0=0.6$	0.237 ${}^{‡}$	0.297 ${}^{‡}$	0.184 ${}^{‡}$	0.197 ${}^{‡}$
${\theta }_0=0.65$	0.499 ${}^{‡}$	0.489 ${}^{‡}$	0.337 ${}^{‡}$	0.386 ${}^{‡}$
${\theta }_0=0.7$	0.657 ${}^{‡}$	0.721 ${}^{‡}$	0.572 ${}^{‡}$	0.616 ${}^{‡}$
${\theta }_0=0.75$	0.873 ${}^{‡}$	0.891 ${}^{‡}$	0.801 ${}^{‡}$	0.818 ${}^{‡}$
${\theta }_0=0.8$	0.962 ${}^{‡}$	0.979 ${}^{‡}$	0.952 ${}^{‡}$	0.941 ${}^{‡}$
${\theta }_0=0.85$	0.994 ${}^{‡}$	0.998 ${}^{‡}$	0.993 ${}^{‡}$	0.989 ${}^{‡}$
${\theta }_0=0.9$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$	0.999 ${}^{‡}$
${\theta }_0=0.95$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$

and

Table 4. The proportion of rejecting $H_0$ out of 1000 replications based on samples of size $n=50$ and $n=100$ is reported. Here, ${}^{†}$ represents the ratio of false positives for the tests and ${}^{‡}$ represents the ratio of the true positives.

n = 50
Proportion	$\mathit{RB}$	$\mathit{BF}$	Exact Test	z-Test
${\theta }_0=0.1$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$
${\theta }_0=0.15$	1 ${}^{‡}$	1 ${}^{‡}$	0.998 ${}^{‡}$	1 ${}^{‡}$
${\theta }_0=0.2$	0.998 ${}^{‡}$	0.997 ${}^{‡}$	0.988 ${}^{‡}$	0.995 ${}^{‡}$
${\theta }_0=0.25$	0.959 ${}^{‡}$	0.983 ${}^{‡}$	0.940 ${}^{‡}$	0.959 ${}^{‡}$
${\theta }_0=0.3$	0.834 ${}^{‡}$	0.902 ${}^{‡}$	0.770 ${}^{‡}$	0.829 ${}^{‡}$
${\theta }_0=0.35$	0.605 ${}^{‡}$	0.720 ${}^{‡}$	0.499 ${}^{‡}$	0.577 ${}^{‡}$
${\theta }_0=0.4$	0.309 ${}^{‡}$	0.422 ${}^{‡}$	0.226 ${}^{‡}$	0.296 ${}^{‡}$
${\theta }_0=0.45$	0.131 ${}^{‡}$	0.208 ${}^{‡}$	0.071 ${}^{‡}$	0.109 ${}^{‡}$
${\theta }_0=0.5$	0.051 ${}^{†}$	0.129 ${}^{†}$	0.034 ${}^{†}$	0.050 ${}^{†}$
${\theta }_0=0.55$	0.105 ${}^{‡}$	0.218 ${}^{‡}$	0.089 ${}^{‡}$	0.109 ${}^{‡}$
${\theta }_0=0.6$	0.306 ${}^{‡}$	0.449 ${}^{‡}$	0.242 ${}^{‡}$	0.296 ${}^{‡}$
${\theta }_0=0.65$	0.589 ${}^{‡}$	0.741 ${}^{‡}$	0.514 ${}^{‡}$	0.577 ${}^{‡}$
${\theta }_0=0.7$	0.848 ${}^{‡}$	0.913 ${}^{‡}$	0.779 ${}^{‡}$	0.829 ${}^{‡}$
${\theta }_0=0.75$	0.961 ${}^{‡}$	0.989 ${}^{‡}$	0.946 ${}^{‡}$	0.959 ${}^{‡}$
${\theta }_0=0.8$	0.996 ${}^{‡}$	1 ${}^{‡}$	0.995 ${}^{‡}$	0.995 ${}^{‡}$
${\theta }_0=0.85$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$
${\theta }_0=0.9$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$
${\theta }_0=0.95$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$
n = 100
${\theta }_0=0.1$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$
${\theta }_0=0.15$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$
${\theta }_0=0.2$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$
${\theta }_0=0.25$	1 ${}^{‡}$	1 ${}^{‡}$	0.999 ${}^{‡}$	0.999 ${}^{‡}$
${\theta }_0=0.3$	0.980 ${}^{‡}$	0.99 ${}^{‡}$	0.977 ${}^{‡}$	0.984 ${}^{‡}$
${\theta }_0=0.35$	0.806 ${}^{‡}$	0.91 ${}^{‡}$	0.838 ${}^{‡}$	0.861 ${}^{‡}$
${\theta }_0=0.4$	0.434 ${}^{‡}$	0.629 ${}^{‡}$	0.478 ${}^{‡}$	0.521 ${}^{‡}$
${\theta }_0=0.45$	0.124 ${}^{‡}$	0.230 ${}^{‡}$	0.133 ${}^{‡}$	0.170 ${}^{‡}$
${\theta }_0=0.5$	0.043 ${}^{†}$	0.092 ${}^{†}$	0.039 ${}^{†}$	0.050 ${}^{†}$
${\theta }_0=0.55$	0.126 ${}^{‡}$	0.248 ${}^{‡}$	0.133 ${}^{‡}$	0.170 ${}^{‡}$
${\theta }_0=0.6$	0.461 ${}^{‡}$	0.633 ${}^{‡}$	0.469 ${}^{‡}$	0.521 ${}^{‡}$
${\theta }_0=0.65$	0.821 ${}^{‡}$	0.907 ${}^{‡}$	0.833 ${}^{‡}$	0.861 ${}^{‡}$
${\theta }_0=0.7$	0.974 ${}^{‡}$	0.989 ${}^{‡}$	0.970 ${}^{‡}$	0.984 ${}^{‡}$
${\theta }_0=0.75$	0.997 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$	0.999 ${}^{‡}$
${\theta }_0=0.8$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$
${\theta }_0=0.85$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$
${\theta }_0=0.9$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$
${\theta }_0=0.95$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$	1 ${}^{‡}$

. The performance to accept or reject the null hypothesis is reported in these tables. The results in Table 3 and Table 4 show that the RB test performs well.

6. Conclusions

In this paper, a new Bayesian approach to the one-sample test for proportion was developed. The approach is based on using the Kullback–Leibler divergence between two binomial distributions. Then, the change of the distance from a priori to a posteriori was compared through the relative belief. The approach is simple, and the computations are straightforward. Additionally, we anticipate that the approach presented here can be extended to the two-sample and k-sample tests. We leave this extension for future research.