Abstract
This paper deals with a new Bayesian approach to the one-sample test for proportion. More specifically, let be an independent random sample of size n from a Bernoulli distribution with an unknown parameter . For a fixed value , the goal is to test the null hypothesis against all possible alternatives. The proposed approach is based on using the well-known formula of the Kullback–Leibler divergence between two binomial distributions chosen in a certain way. Then, the difference of the distance from a priori to a posteriori is compared through the relative belief ratio (a measure of evidence). Some theoretical properties of the method are developed. Examples and simulation results are included.
1. Introduction
The one-sample test for proportion is a topic that is taught in most introductory statistics courses. Let be an independent random sample of size n from a Bernoulli distribution with unknown parameter , where . The interest is to test the null hypothesis , where is a known value.
In this problem, within the classical frequentist framework, the approximate z-test is commonly used for testing . Specifically, if and (i.e., a large-sample test), the test statistic is given by
where represents the maximum likelihood estimator under . For example, to test against the alternative hypothesis , the two-sided p-value is equal to , where Z has the standard normal distribution. Then, is rejected if the p-value is less than a given significance level . As for a small-sample test, the exact binomial distribution (rather than a normal approximation) is typically used. We refer the reader to Chapter 12 of [1] for general methods of dealing with hypothesis testing problems.
A standard alternative method to the earlier procedures is the Bayesian approach by means of the Bayes factor [2,3]. This requires the specification of prior distributions for parameters. It is noticed that most of the Bayesian methods in this area are presented for comparing the proportion in two-sample problems. For instance, ref. [4] proposed an approximation for the Bayes factor to test the equality of two binomial proportions. Furthermore, ref. [5] provided some recommendations for using Bayes factors in testing the equality of two proportions to improve the sensitivity of the test. Several studies can also be found in the literature on developing the Bayesian two-sample proportion test in contingency tables for testing the independence between rows and columns. See, for example, ref. [6] and the references therein.
For the one-sample problem, ref. [7] used an improper prior for the proportion in clinical trials. They computed the probability of being under the null and alternative hypotheses and then compared them to decide whether to accept the null hypothesis. Ref. [8] developed the BayesFactor R package, which can be used to compute the Bayes factor for several research designs and hypotheses including a test for one-sample proportion. Ref. [9] discussed this problem in detail based on a direct computation of the relative belief ratio. In this paper, a different Bayesian approach to the one-sample test for proportion is proposed. First, to avoid a prior-data conflict (see Section 2), the uniform is placed as a prior on the true proportion, then the Kullback–Leibler divergence between two binomial distributions (sampling and hypothesized distributions selected in an appropriate way). The difference of the distance from a priori to a posteriori is compared such that, if the posterior distribution is more concentrated around the hypothesized distribution than the prior distribution, then there is evidence in favor of the null hypothesis. Furthermore, if the posterior distribution is less concentrated, then it indicates evidence against the null hypothesis. This comparison is made via a relative belief ratio, which measures the evidence in the observed data for or against the null hypothesis. We also discuss a measure of the strength of this evidence, and therefore, the proposed test is completely based on a direct measure of statistical evidence.
We highlight here some key motivations of the proposed technique. First, it is simple, and the computations are straightforward. Second, unlike the classical approach, it gives evidence in favor of the null hypothesis and does not require choosing the significance level. Third, it avoids any possibility of the prior for prior-data conflict and, hence, provides robustness in the analysis [10]. Fourth, the proposed method uses the relative belief and computes the strength separately, and it has been proven to work as a calibration of the relative belief [9]. Fifth, similar to the exact binomial test, the proposed test works for small sample sizes. Lastly, it can be extended to the case of the equality of k independent sample binomial proportions.
The remainder of this paper is organized as follows. Inferences using the relative belief ratio are introduced in Section 2. In Section 3, a Bayesian test is proposed. In this section, we also cover checking for prior-data conflict, prior elicitation, and the prior for bias. In Section 4, we consider several examples to illustrate the approach. A comparison with other methods is also considered in this section. Finally, some concluding remarks are given in Section 5.
2. Inferences Using Relative Belief
The relative belief ratio was developed in [9], and it has become a widely used tool in the theory of Bayesian statistical hypothesis testing. For some references, see, for example, [11,12,13,14]. Suppose a statistical model is given by the density function with respect to Lebesgue measure on the parameter space . Let be a prior on . After observing the data x, the posterior distribution of is given by the density where . If the interest is to make an inference about the parameter , when and are continuous at , the relative belief ratio for a hypothesized value of is
If and are discrete, the relative belief ratio is defined through limits [9]. Clearly, (1) measures how beliefs have changed such that is the true value from a priori to a posteriori. Accordingly, is a measure of evidence that is the true value. If , then the probability of being the true value increases from a priori to a posteriori, and so, there is evidence based on the data that is the true value; therefore, there is evidence in favor of . If , then the probability of being the true value decreases from a priori to a posteriori. Thus, the data provide evidence against being the true value. When , then there is no evidence either way.
Let T denote a minimal sufficient statistic for with density and . Ref. [15] showed that
The relationship in (2) is known as the Savage–Dickey ratio.
The next step after computing (2) is to calibrate whether this value indicates strong or weak evidence for or against . A standard calibration of is obtained by computing the following probability [9]:
Here, is the posterior cumulative distribution function with posterior density . As represents the posterior probability that the true value of has a relative belief ratio of at most the relative belief ratio at , when a small value for gives strong evidence against (most of the values of have relative belief ratios greater than the relative belief ratio at ). On the other side, if a large value for gives strong evidence in favor of (most of the values of have relative belief ratios smaller than the relative belief ratio at ), while a small value of indicates weak evidence in favor of .
Another basic issue when dealing with Bayesian inference methods is the possibility to bias the analysis by the prior. To assess any possible bias, following [9], when is true, the bias against can be measured by computing the prior probability:
where denotes the prior predictive distribution of T when the true value is . Large values of (4) imply there is a bias against and there is a concern about any evidence against . On the other side, the bias in favor of is given by the prior probability
for values such that the difference between and represents the smallest difference of practical importance. When the bias in favor is large, then there is evidence in favor of . As a result, any evidence in favor of is not convincing. For a fixed prior, both biases decrease as the sample size increases.
Another concern that appears in Bayesian analysis is whether a chosen prior may be strongly contradicted by the data [9]. A possible contradiction between the data and the prior is known as a prior-data conflict. A common method for checking the prior was developed by [16], which involves computing the probability:
where T is a minimal sufficient statistic of the model and is the prior predictive probability distribution of T with density and is given by . If (6) is small, then lies in a region of low prior probability, such as a tail or anti-mode, which indicates a conflict. Furthermore, if (6) is large, then prior-data conflict should not be an issue. For different methods of checking prior-data conflict, see [17]. Furthermore, for relevant robustness results of inferences based on the relative belief ratio, see [10,18], where a strong connection between robustness and prior-data conflict is found.
3. One-Sample Bayesian Test for Proportion
3.1. The Approach
The proposed test is based on using the Kullback–Leibler (KL) distance. The KL distance between two discrete cumulative distribution functions (cdfs) P and Q defined on the same probability space with corresponding probability mass functions (pmfs) p and q (with respect to the counting measure) is defined by
It is well known that , and the equality holds if and only if . However, it is not symmetric and does not satisfy the triangle inequality [19]. For example, if P and Q are the cdfs of binomial and binomial , respectively, then the KL divergence between P and Q is given by
As if and only if , testing is equivalent to testing . For the proposed Bayesian approach, to test , let Beta(, ) be the prior of . Then, the posterior distribution of given is Beta(, ), where and . If Beta, we denote by , and if Beta, then we write . Note that, as , by the weak law of large numbers, , the true population proportion. Thus, by (7), if is true, we have . On the other hand, if is not true, then
where . It follows that, if is true, then that distribution of should be concentrated about 0 more than D. Therefore, the proposed test is constructed based on a comparison of the distribution of D and about 0 via the relative belief ratio:
where and are the probability density functions of D and , respectively. If the distribution of is more concentrated about 0, then ; hence, this is evidence in favor of , while if it is less concentrated about 0, then , and there is evidence against . Furthermore, the strength of evidence, whether for or against , is also computed.
Note that, if denotes the KL divergence function in (7), then
where and are the two parts of g on the left and right sides of , respectively. See Figure 1 below.
Figure 1.
Plot of , the KL divergence function in (7). In the plot, we set and .
It follows that
Now, as , could , or , or , . In either of theses three cases, . The difference between and may occur when . Accordingly, we develop Algorithm A in Section 3.4 to compute and the corresponding strength , where is the cumulative distribution function of .
3.2. Checking for Prior-Data Conflict
It is known that is the minimal sufficient statistic for . To check for prior-data conflict, we compute that tail probability in (6). If , then the prior prediction function of is given by
We have
Clearly, computing (9) should be performed by simulation. For this, generate . Then, generate . Repeat this many times, and record the proportion of values of that are less than or equal to giving a Monte Carlo estimate of (9). Clearly, when , then both sides of the inequality in (9) are equal and the probability is 1. That is, with Beta prior (i.e., the uniform prior), there is never any prior-data conflict. Accordingly, we choose the prior to be , which will relax the need to check for prior-data conflict.
The following proposition finds the expected value of D and for the uniform prior. This proposition could be used, for example, to compare the theoretical and the computed values of D and .
Proposition 1.
Let and be as defined in Section 3.1:
- (i)
- If Beta, then
- (ii)
- If Beta, thenwhere is the digamma function.
3.3. Checking the Prior for Bias
As mentioned in Section 2, there is a need to check if the prior Beta(1,1) induces any bias in the problem. Note that, with this prior, the relative belief ratio is derived by [9]:
The bias against the hypothesis is measured by computing (4) using as given in (15). Here, is the distribution. On the other hand, the bias in favor of the hypothesis is measured by computing (5) when is true and is as defined in (15). Here, is the distribution. The interpretation of the bias was given in Section 2.
To compute the bias in this problem, it should be performed via simulation. For example, to compute the bias against, generate , and compute as defined in (15). Repeat this many times, and record the proportion of values of that are less than 1. The bias in favor can be estimated similarly.
3.4. The Algorithm
As and in (8) have no closed forms, the relative belief ratio and the strength need to be estimated via approximation. The following algorithm summarizes the steps required to test .
Algorithm A (RB test for proportion).
- (i)
- Generate from Beta, and compute D.
- (ii)
- Repeat Step (ii) to obtain a sample of values of D.
- (iii)
- Generate from Beta, and compute .
- (iv)
- Repeat Step (iv) to obtain a sample of values of .
- (v)
- Compute the relative belief ratio and the strength as follows:
- (a)
- Let L be a positive number. Let denote the empirical cdf of D based on the prior sample in (3), and for let be the estimate of the -prior quantile of D. Here, , and is the largest value of D. Let denote the empirical cdf based on . For , estimate bythe ratio of the estimates of the posterior and prior contents of Thus, we estimate by , where and are chosen so that is not too small (typically, .
- (b)
- Estimate the strength by the finite sum:
4. Examples
The approach is illustrated through two examples. In Algorithm A, we set and . For the sake of comparison, the Bayes factor (BF) of [8] and the p-values of the exact binomial test and the approximate z-test were also considered. The R package BayesFactor was used to compute the BF of [21], while the R functions binom.test() and prop.test() were used to compute the p-values of the exact and the approximate tests, respectively. For these tests, we set . To interpret the Bayes factor, we used the following rule of thumb introduced by Jefferys in [22]. See also [23]. This scale gives evidence against . To this end, if , no evidence; , anecdotal; , moderate; , strong; , very strong; and , extreme. On the other side, if, for example, , then 1/0.2 = 45 is interpreted as there is moderate evidence in favor of .
Example 1.
(Clinical trial; [24]).
A method currently used by doctors to screen women for breast cancer fails to detect cancer in 20% of the women who actually have the disease. A new method has been developed that researchers hope will detect cancer more accurately. This new method was used to screen a random sample of 140 women known to have breast cancer. Of these, the new method failed to detect cancer in 12 women. The researchers want to test if the sample provides evidence that the failure rate of the new method differs from the one currently in use.
First, the bias in the prior is assessed by computing (4) with . In this case, the bias against the null hypothesis is 0.2233. On the other hand, the bias in favor of the null hypothesis is measured by computing (5) with , which gives 0.2706 (for ) and 0.01737 (for ). This shows relatively equal bias either for or against the null hypothesis for the uniform prior. The results of the tests are reported in Table 1. In particular, the relative belief ratio of the proposed test is 0.0040, which is less than 1, and the strength is 0.0021, which is close to 0. Hence, there is strong evidence to conclude that the new female breast cancer detection method is better than the used method.
Table 1.
Test results of Example 1.
Example 2.
(Production defective; [25]).
A machine in a factory must be repaired if it produces more than 10% defective items among the large lot of items that it produces in a day. A random sample of 100 items from the day’s production contains 15 defective items, and the supervisor says that the machine must be repaired. The goal is to test if the sample provides evidence to support his/her decision.
As in Example 1, the bias in the prior is assessed by computing (4) with . In this case, the bias against the null hypothesis is 0.1182. On the other hand, the bias in favor of the null hypothesis is measured by computing (5) with , which gives 0.0504 (for ) and 0.0204 (for ). This shows an acceptable bias either for or against the null hypothesis for the uniform prior. The results of the tests are reported Table 2. As and the strength is 0.406, there is moderate evidence to conclude that the factory produces 10% defective items in a day.
Table 2.
Test results of Example 2.
5. Simulation Study
In this section, we run a Monte Carlo simulation study of size 1000 to determine the proportion of false positives (false rejection of the null hypothesis) and the proportion of true positives (true rejection of the null hypothesis) when random data with various sample sizes are generated from the Bernoulli distribution with parameter , where . In Figure 2, we plot against the proportion of rejecting (i.e., when ). Figure 2 reveals that when is close to 0.5 (i.e., is true), the proportion of rejecting is close to 0. However, as moves away from 0.5, the curves move toward 1. Furthermore, as the sample size increases, the computed proportions converge faster to the true values.
Figure 2.
Power curve of the proposed test for various sample sizes .
We also conducted a simulation study to compare the performance of the RB test, the BF test, and the exact and approximate tests. The proportion of rejecting of the exact and approximate tests is computed by the number of times the p-value is less than divided by 1000. For sample sizes , we generated data from Bernoulli , where is given in Table 3 and Table 4. The performance to accept or reject the null hypothesis is reported in these tables. The results in Table 3 and Table 4 show that the RB test performs well.
Table 3.
The proportion of rejecting out of 1000 replications based on samples of size and is reported. Here, represents the ratio of false positives for the tests and represents the ratio of the true positives.
Table 4.
The proportion of rejecting out of 1000 replications based on samples of size and is reported. Here, represents the ratio of false positives for the tests and represents the ratio of the true positives.
6. Conclusions
In this paper, a new Bayesian approach to the one-sample test for proportion was developed. The approach is based on using the Kullback–Leibler divergence between two binomial distributions. Then, the change of the distance from a priori to a posteriori was compared through the relative belief. The approach is simple, and the computations are straightforward. Additionally, we anticipate that the approach presented here can be extended to the two-sample and k-sample tests. We leave this extension for future research.
Author Contributions
Conceptualization, L.A.-L.; Methodology, L.A.-L.; Software, K.L.; Validation, Y.C. and Y.W.; Formal analysis, L.A.-L. and F.F.-A.; Investigation, K.L.; Resources, L.A.-L. and F.F.-A.; Data curation, K.L.; Writing—original draft preparation, L.A.-L. and K.L.; Writing—review and editing, F.F.-A. and K.L.; Supervision, L.A.-L.; Project administration, L.A.-L.; Funding acquisition, L.A.-L. All authors have read and agreed to the published version of the manuscript.
Funding
This research received no external funding.
Institutional Review Board Statement
There was no ethical approval is required.
Informed Consent Statement
This study did not involve human or any living organisms.
Data Availability Statement
No dataset had been used.
Acknowledgments
The authors thank the Editor and three anonymous Referees for their important and constructive comments, which led to a significant improvement of the paper, In particular, pointing out the connection between and in Section 3.1 is highly appreciated.
Conflicts of Interest
There was no conflict of interest.
References
- Bain, L.J.; Engelhardt, M. Introduction to Probability and Mathematical Statistics; Duxbury Press: Belmont, CA, USA, 1992; Volume 4. [Google Scholar]
- Kass, R.E.; Raftery, A.E. Bayes factors. J. Am. Stat. Assoc. 1995, 90, 773–795. [Google Scholar] [CrossRef]
- Rouder, J.N.; Morey, R.D.; Speckman, P.L.; Province, J.M. Default Bayes factors for ANOVA designs. J. Math. Psychol. 2012, 56, 356–374. [Google Scholar] [CrossRef] [Scilit]
- Kass, R.E.; Vaidyanathan, S.K. Approximate Bayes factors and orthogonal parameters, with application to testing equality of two binomial proportions. J. R. Stat. Soc. Ser. B (Methodol.) 1992, 54, 129–144. [Google Scholar] [CrossRef] [Scilit]
- Dablander, F.; Huth, K.; Gronau, Q.F.; Etz, A.; Wagenmakers, E.J. A puzzle of proportions: Two popular Bayesian tests can yield dramatically different conclusions. Stat. Med. 2022, 41, 1319–1333. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Jamil, T.; Ly, A.; Morey, R.D.; Love, J.; Marsman, M.; Wagenmakers, E.J. Default “Gunel and Dickey” Bayes factors for contingency tables. Behav. Res. Methods 2017, 49, 638–652. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Nieto, A.; Extremera, S.; Gómez, J. Bayesian hypothesis testing for proportions. In Paper SP08; PhUSE: Madrid, Spain, 2011; Available online: https://www.lexjansen.com/phuse/2011/sp/SP08.pdf (accessed on 1 October 2022).
- Morey, R.D.; Rouder, J.N.; Jamil, T.; Urbanek, S.; Forner, K.; Ly, A. Package ‘BayesFactor’. 2022. Available online: https://cran.rproject.org/web/packages/BayesFactor/BayesFactor.pdf (accessed on 1 October 2022).
- Evans, M. Measuring Statistical Evidence Using Relative Belief; Monographs on Statistics and Applied Probability 144; Taylor & Francis Group, CRC Press: Boca Raton, FL, USA, 2015. [Google Scholar]
- Al-Labadi, L.; Evans, M. Optimal robustness results for relative belief inferences and the relationship to prior-data conflict. Bayesian Anal. 2017, 12, 705–728. [Google Scholar] [CrossRef] [Scilit]
- Abdelrazeq, I.; Al-Labadi, L.; Alzaatreh, A. On one-sample Bayesian tests for the mean. Statistics 2020, 54, 424–440. [Google Scholar] [CrossRef] [Scilit]
- Al-Labadi, L.; Berry, S. Bayesian estimation of extropy and goodness of fit tests. J. Appl. Stat. 2020, 49, 357–370. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Al-Labadi, L.; Patel, V.; Vakiloroayaei, K.; Wan, C. Kullback–Leibler divergence for Bayesian nonparametric model checking. J. Korean Stat. Soc. 2020, 50, 272–289. [Google Scholar] [CrossRef] [Scilit]
- Al-Labadi, L. The two-sample problem via relative belief ratio. Comput. Stat. 2021, 36, 1791–1808. [Google Scholar] [CrossRef] [Scilit]
- Baskurt, Z.; Evans, M. Hypothesis assessment and inequalities for Bayes factors and relative belief ratios. Bayesian Anal. 2013, 8, 569–590. [Google Scholar] [CrossRef] [Scilit]
- Evans, M.; Moshonov, H. Checking for prior-data conflict. Bayesian Anal. 2006, 1, 893–914. [Google Scholar] [CrossRef] [Scilit]
- Nott, D.J.; Seah, M.; Al-Labadi, L.; Evans, M.; Ng, H.K.; Englert, B. Using prior expansion for prior-data conflict checking. Bayesian Anal. 2021, 16, 203–231. [Google Scholar] [CrossRef] [Scilit]
- Al-Labadi, L.; Asl, F.F. On robustness of the relative belief ratio and the strength of its evidence with respect to the geometric contamination prior. J. Korean Stat. Soc. 2022, 1–15. [Google Scholar] [CrossRef] [Scilit]
- Cover, T.M.; Thomas, J.A. Elements of Information Theory, 2nd ed.; Wiley: Hoboken, NJ, USA, 2006. [Google Scholar]
- Al-Labadi, L.; Evans, M. Prior-based model checking. Can. J. Stat. 2018, 46, 380–398. [Google Scholar] [CrossRef] [Scilit]
- Rouder, J.; Speckman, P.; Sun, D.; Morey, R.; Iverson, G. Bayesian t tests for accepting and rejecting the null hypothesis. Psychon. Bull. Rev. 2009, 16, 225–237. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Jeffreys, H. Theory Probability; Oxford University Press: Oxford, UK, 1961. [Google Scholar]
- Raftery, A.E. Bayesian model selection in social research. Sociol. Methodol. 1995, 25, 111–164. [Google Scholar] [CrossRef] [Scilit]
- McClave, J.; Sincich, T. Statistics, 13th ed.; Person: Boston, MA, USA, 2017. [Google Scholar]
- Wackerly, D.; Mendenhall, W.; Scheaffer, R.L. Mathematical Atatistics with Applications; Cengage Learning: Belmont, CA, USA, 2014. [Google Scholar]
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

