Algorithm Providing Ordered Integer Sequences for Sampling with Replacement Confidence Intervals

Abstract

Sampling with replacement occurs when drawing without removing individuals from finite populations. It is a common distribution technique used in physics, biology, and medicine. It is used in state analysis of qubits, the physics of particle interactions, studies of genetic variation and variability, and analyzing the treatment effects from clinical trial analyses. When applied, sample statistics should be accompanied by confidence intervals. The major difficulty in expressing the confidence intervals in sampling with replacement consists of discreetness regarding the probability distribution. As a result, no mathematical formula can handle an optimum solution. Using a simple algorithm is proposed in order to obtain confidence intervals for sampling with replacement variables (x from m trials with replacement) and their proportion ($x/m$). A question-based discussion is presented. Traditional confidence intervals often require large sample sizes. Confidence intervals, constructed in a deterministic way provided by the proposed algorithm for sampling with replacement, allow constructing intervals without constraints.

1. Introduction

The sampling strategy is foundational when dealing with finite populations. In “sampling with replacement,” after extracting an individual (or a case) from a population, it can be selected for insertion in the population, having a non-zero probability of being extracted again in the future. In medical studies, sampling a dichotomous outcome emulates a Bernoulli trial [1], and the ratio between the desired (positive) outcome (x) and the number of trials (or sample size, m) leads to a proportion ($0\le x/m\le 1$).

Fundamental in this context is the probability associated with the reproduction of the observed outcome. Imposing the size of a second sample to have the value of the first one, the distribution function for the probability of having y as the outcome is binomial (referring to the $\left({\displaystyle \genfrac{}{}{0pt}{}{m}{y}}\right)$ notation; see [2]), as in Equation (1):

$$f_B(y;x,m)=\left({\displaystyle \genfrac{}{}{0pt}{}{m}{y}}\right){\left({\displaystyle \frac{x}{m}}\right)}^y{\left(1-{\displaystyle \frac{x}{m}}\right)}^{m-y}$$

(1)

In Equation (1), the true proportion rate from the population has been replaced by the observed value of it from the sample. In the absence of any other observation, all the information from the sampling has been used. The question is “How far away from x can y be?”. The result can be expressed using a confidence interval (CI) assuming a success rate (usually 95%).

Calculating the CI (with a conventional risk of 5% or otherwise) for a discrete probability distribution is a problem of combinatorics, which is a fact that has been recognized for quite a long time [3], but the complexity of the problem denied the availability of an exact method for its calculation until much later [4].

A series of approximations have been proposed, speculating the normal asymptotic behavior of the binomial distribution [5]. The construction of the exact CI has been proposed as well [4,6]. Feeding an optimization algorithm with the values obtained from a pool of variants is yet another explored alternative [7].

Turning to the fields of survey sampling and robust estimation, readers can find very useful results, including Raghav’s, communicated in [8] (reporting a novel class of estimators and several new member estimators, combining the ratio and product forms, within the framework of adaptive cluster sampling), as well as those in [9] (reporting new estimators utilizing ranked set sampling to assess the population mean when faced with both correlated and uncorrelated measurement errors), and those in [10] (reporting a case study on a Poisson distribution).

An algorithm balancing the actual error around the imposed level of error is proposed here for variables taken from sampling with replacement and their proportions.

2. Constructing a CI

In many ways, Wald’s CI [5], derived under the assumption of an infinite population, is a reference example of a CI. Anticipating the results communicated here, a CI for a sampled with replacement variable (x out of m) should be seen as the outcome of an algorithm rather than of a simple mathematical formula, and the outcome of an such algorithm should be a series of numbers, indexed from 0 to m: $N_0$, $N_1$, …, $N_m$.

It is proposed that, by symmetry, a CI should be constructed with these numbers for x out of m as $[N_x,m-N_{m-x}]$.

Let us take an example with Wald’s CI [5]. Let $z_{\alpha }=\Phi (1-\alpha /2)$, where $\Phi$ is the cumulative function of the normal distribution. For $\alpha$ = 0.05, $z_{\alpha }$ is about 1.95996. A Wald CI will be calculated as in Equation (2):

$$C{I}_W(x;m,\alpha )=x\pm z_{\alpha }\sqrt{x(m-x)/m}$$

(2)

It is nonsensical to have boundaries smaller than 0 or larger than m. Furthermore, any non-integer value of the boundary makes no difference since x takes only integer values.

Table 1. From Wald’s CI to ordered integer sequences ($m=10$ and $\alpha =5\%$).

x	${\mathbf{CI}}_{\mathit{W}}$	${\mathbf{CI}}_{\mathit{I}}$	Ordered Integer Sequences	Coverage
1	$\pm 1.9$	[0, 2]	$0=N_1$, $10-N_{10-1}=2$	92.98%
2	$\pm 2.5$	[0, 4]	$0=N_2$, $10-N_{10-2}=4$	96.72%
3	$\pm 2.8$	[1, 5]	$1=N_3$, $10-N_{10-3}=5$	92.44%
4	$\pm 3.0$	[1, 7]	$1=N_4$, $10-N_{10-4}=7$	98.17%
5	$\pm 3.1$	[2, 8]	$2=N_5$, $10-N_{10-5}=8$	97.85%
6	$\pm 3.0$	[3, 9]	$3=N_6$, $10-N_{10-6}=9$	98.17%
7	$\pm 2.8$	[5, 9]	$5=N_7$, $10-N_{10-7}=9$	92.44%
8	$\pm 2.5$	[6, 10]	$6=N_8$, $10-N_{10-8}=10$	96.72%
9	$\pm 1.9$	[8, 10]	$8=N_9$, $10-N_{10-9}=10$	92.98%

$C{I}_W=[W_L,W_U]$, $C{I}_I=[I_L,I_U]$, $I_L\left(x\right)=\lceil \max (W_L,0)\rceil$, $I_U\left(x\right)=\lfloor \min (W_L,m)\rfloor$.

lists the CIs for variables (x in Table 1) with nonnegative integers as boundaries calculated with Equation (2) and provided along with their true coverage probabilities.

One should notice the symmetry in the intervals provided in Table 1. However, the solution provided in Table 1 is not optimal. Thus,

• CIs provided in Table 1 are not optimal; 2, 3, 5, 6, and 8 instead of 1, 2, 4, 5, and 7 provide better estimates;
• If it is optimal to provide an at least $1-\alpha$ coverage, then $CI(1,10,0.05)$ should be [0, 3] instead of [0, 2] with a coverage of 98.72% instead of 92.98%; $CI(3,10,0.05)$ should be [0, 5] or [1, 6] instead of [1, 5] with coverage of 95.27% and 96.12% instead of 92.44%;
• If it is optimal to provide an at most $1-\alpha$ coverage, then $CI(2,10,0.05)$ should be [0, 3] instead of [0, 4] with a coverage of 87.91% instead of 97.72%; $CI(4,10,0.05)$ should be [0, 6] instead of [1, 7] with a coverage of 94.52% or [1, 6] with a coverage of 93.92% instead of 97.72%;
• As can be observed, imposing a single rule (such as “at least” or “at most”) is not enough; see alternatives for $CI(3,10,0.05)$: [0, 5] has a coverage of 95.27% and [1, 6] has a coverage of 96.12% or alternatives for $CI(4,10,0.05)$: [0, 6] has a coverage of 94.52% and [1, 6] has a coverage of 93.92%;
• Being closest to the imposed coverage but at least equal (or at most equal) to it seems a reasonable criterion but does not work all the time; thus, for it to be the closest to the imposed coverage but at most equal, $CI(3,10,0.05)$ should be [1, 5] with a coverage of 92.44% and $CI(4,10,0.05)$ should be [0, 6] with a coverage of 94.52%. However, since the left boundary of $CI(3,10,0.05)$ is 1, another common sense rule says that the left boundary of $CI(4,10,0.05)$ should be at least 1.

To summarize, there is no unique recipe for an ideal CI for binomial distribution. There are many optimal solutions depending on what is considered to be optimal. In [7], for instance, ${\sum }_{x=0}^m{(CCI(x,m,\alpha )+\alpha -1)}^8\to$ min has been suggested, where $CCI(x,m,\alpha )$ is the coverage of $CI(x,m,\alpha )$ as criteria.

3. Balanced CI with Ordered Integer Sequences

The way in which a confidence interval for a discrete distribution is constructed should be changed. In the case of a binomial distribution, a mathematical formula is not able to encompass the complexity of the issue. Three algorithms have been proposed in [11].

Here, one solution is given, balancing between the imposed and actual level of residual error.

To fix the terminology, $\alpha$ is the imposed level, x is the number of successes, m is the number of trials, $[i,j]$ is the confidence interval, and q is its actual coverage.

Algorithm 1 systematically keeps the error closest to the imposed level. There is a symmetry in the confidence intervals provided by the algorithm. For each significance level $\alpha$ and sample size m, the solution provided by running Algorithm 1 is a series of numbers, $N_0$, …, $N_m$. The CI of a pair of data $(x,m)$ for $\alpha$ is $[N_x,m-N_{m-x}]$, where $x=0,1,\dots ,m$. Some $N_x$ series are listed in the Appendix A for convenience.

The $N_x$ series provided by Algorithm 1 are monotonic. The actual coverage probability series are symmetric. Any of the series defined by $\{i:(i,j,q)\leftarrow \mathtt{BalancedCI}(x,m,\alpha ),0\le x\le m\}$ and by $\{j:(i,j,q)\leftarrow \mathtt{BalancedCI}(x,m,\alpha ),0\le x\le m\}$ provide the same but in the reversed order, and this is the beauty of the proposed solution.

The values of the numbers constructing CIs with $[N_x,30-N_{30-x}]$ are given in Appendix A at $m=30$ entry and designated as $N_0$, …, $N_{30}$. The plot of Algorithm 1 CI non-coverage probabilities is given in Figure 1.

Algorithm 1: BalancedCI

The actual non-coverage of CIs provided by Algorithm 1 is not monotone (it alternates, on average, around the imposed level) but symmetric and convergent to it. With data in Appendix A,

Table 2. Actual non-coverage probability tendencies ($\alpha =5\%$).

Stat.	$\mathit{m}=30$	$\mathit{m}=45$	$\mathit{m}=100$	$\mathit{m}=900$
$Avg$	5.1000	5.0045	4.9566	4.9961
$Dev$	0.8793	0.6591	0.4939	0.1561

${ϵ}_x={\displaystyle \sum _{y\notin CI}}{f}_{Bino}(y;x,m)$, $Avg={\displaystyle \sum _{x=1}^{m-1}}{\displaystyle \frac{{ϵ}_x}{m-1}}$, $Dev={\displaystyle \sum _{x=1}^{m-1}}{\displaystyle \frac{|\alpha -{ϵ}_x|}{m-1}}$.

statistics can be calculated; the average values converge slowly, $\approx \sqrt{m}$).

In terms of computational complexity, Algorithm 1 for given $\alpha$, m, and x requires a number of steps to arrive from x ($i\leftarrow x$ and $j\leftarrow x$ in Algorithm 1) to $N_x$ and $N_{m-x}$ (see the series of $N_x$ for m = 30, 45, 100, and 900 in Appendix A). The solution is nearby. In fact, Equation (2) is an estimate of the asymptotic interval. Computational time is then proportional to $2z_{\alpha }\sqrt{x(m-x)/m}$. The shortest computational time is for $x=0$ and $x=m$ (near 0; only evaluation of the $f_{RB}$ function time since $N_0=0$ and $N_m=m$ always), while the longest is for x near ${\displaystyle \frac{m}{2}}$ (where $\sqrt{m}$ evaluations are needed). Thus, Algorithm 1 complexity is $\mathcal{O}\left(\sqrt{m}\right)$. The algorithm speed can be increased too if an approximate solution is provided before the While loop. Algorithm 1 from [11] or $\max (W_L,0)\rceil$, $\lfloor \min (W_L,m)\rfloor$ with Equation (2) can be used for this task.

4. Discussion

One major point is to provide simple answers to simple questions. Here are some:

4.1. Question 1

How to express the confidence interval for a variable?

Let us take $x=26$ with $m=30$. Using Appendix A, we get $N_x=23$ (this is $N_{26}$ in the series) and $N_{m-x}=1$ (this is $N_4$ in the series). The BalancedCI at 95% desired coverage ($\alpha =5\%$) is [23, 29], and its actual coverage is 94.8%.

4.2. Question 2

How should one deal with proportions instead of variables?

The main issue is people’s tendency to simplify. For instance, ${\displaystyle \frac{1}{2}}$ is often written as $0.5$; for ${\displaystyle \frac{1}{3}}$, we write $0.\left(3\right)$ or $0.33\dots$, but it is, in fact, not the same thing when it comes to sampling.

The preference for real numbers is obvious: people are much more comfortable with them and they quickly indicate the order of magnitude (0.0…or 0.1…is close to 0, 0.8…or 0.9…is close to 1, 0.4…or 0.5…is close to 0.5), which allows for instant interpretation. At the same time, however, it lets something slip through the cracks. One can express (with three decimal places, for example) any real sub-unit number, but its exact value cannot be obtained as a proportion of two integers, except under certain conditions and very few situations (one of these being that the sample size is 1000; for the sample size 997, however, there is no non-trivial situation).

Thus, using the same numbers, for $x=26$ positive outcomes from $m=30$ trials, in order to pass along all information, a convention is first needed to express the proportion as is without simplifying it (proportion is then ${\displaystyle \frac{26}{30}}$), and the confidence interval is expressed accordingly with fractions too.

The BalancedCI at 95% desired coverage ($\alpha =5\%$) for ${\displaystyle \frac{26}{30}}$ is $\left[{\displaystyle \frac{23}{30}},{\displaystyle \frac{29}{30}}\right]$, and its actual coverage is 94.8%.

4.3. Question 3

How should a real-world data analysis be conducted?

Let us provide the answer by taking one case study from the literature. In [12], a study was conducted with 20 patients subject to dermal regeneration template placement over areas with exposed bone/tendon prior split-thickness skin grafting, all being confirmed clinically and microbiologically negative for previously treated infection. Utilizing non-invasive fluorescence imaging, bacterial load was identified using a defined threshold in eight out of twenty patients. Out of eight, four developed an infection. Three of the four infections were positive for Pseudomonas.

There are certain sampling with replacement variables in the study data. Five research questions can be formulated (the null hypothesis is that the effect being studied does not exist; the observation can be merely by chance):

• Q3.1: Was the detection of the bacterial load with non-invasive fluorescence imaging statistically significant in the group, or can it be asserted as being observed by chance?
  Answer: The observed proportion was 8 out of 20. The sequence of integers for $\alpha =0.05$ and $m=20$ is 0 0 0 1 1 2 3 4 4 5 6 7 8 9 10 11 13 14 16 18 20. The CI for $x=8$ is [4, 12]. The answer is Yes (detection of the bacterial load with non-invasive fluorescence imaging was statistically significant in the group), with a true risk (of being in error in the coverage) of 3.6%.
• Q3.2: Was the development of a bacterial infection a real risk for the patients following the procedure, or can it be asserted as being observed by chance?
  Answer: The observed proportion was 4 out of 20. Using the same sequence of integers, the CI for $x=4$ is [1, 7], and the answer is Yes (developing a bacterial infection was a real risk for the patients following the procedure), with a true risk of 4.3%.
• Q3.3: Was Pseudomonas a real threat for the patients following the procedure, or can it be asserted as being observed by chance?
  Answer: The observed proportion was 3 out of 20. Using the same sequence of integers, the CI for $x=3$ is [1, 6], and the answer is Yes (Pseudomonas was a real threat for the patients following the procedure), with a true risk of 6.1%.
• Q3.4: Was the development of a bacterial infection a real risk for the patients possessing bacterial load detected by non-invasive fluorescence imaging, or can it be asserted as being observed by chance?
  Answer: The observed proportion was 4 out of 8. The sequence of integers for $\alpha =0.05$ and $m=8$ is 0 0 0 1 2 3 4 6 8. The CI for $x=4$ is [2, 6], and the answer is Yes (developing a bacterial infection was a real risk for the patients possessing bacterial load detected by non-invasive fluorescence imaging), with a true risk (of being in error in the coverage) of 7.0%.
• Q3.5: Was infection with Pseudomonas a real risk for the patients possessing bacterial load detected by non-invasive fluorescence imaging, or can it be asserted as being observed by chance?
  Answer: The observed proportion was 4 out of 8. The sequence of integers for $\alpha =0.05$ and $m=8$ is 0 0 0 1 2 3 4 6 8. The CI for $x=3$ is [1, 5]. The answer is Yes (infection with Pseudomonas was a real risk for the patients possessing bacterial load detected by non-invasive fluorescence imaging), with a true risk (of being in error in the coverage) of 5.9%.

With the sampled data, the answer was Yes in all cases since $0\notin CI$.

4.4. Question 4

How is the proposed method positioned relatively to other methods from the scientific literature?

Any formula-based approach will have the same deficiency against proposed method as the Wald confidence interval (Equation (2)) does. In this category, included are Agresti-Coull [13] and Wilson [14] CIs, and lesser-known approaches, such as arcsine and logit [15].

Any probabilistic-based approach, such as Clopper–Pearson [3] and Blyth–Still–Casella [16], imposes an actual risk always less than or equal to imposed risk. Because of that, the actual risk is severely penalized and on average tends to be biased relative to the imposed value (their ideal actual coverage is approximated by $1-{0.197}_{\pm 0.009}\ln (-{18.0}_{\pm 3.6}\ln \left(m\right))$ for $\alpha =5\%$, being about 4.1% instead of 5% for $m=30$).

A probabilistic balanced approach, such as Jeffrey’s [17], will provide a good solution on average, but not the one with minimal deviation from the imposed level, as Algorithm 1 does.

5. Conclusions

For sampling with replacement, an algorithm was proposed for calculating the confidence interval for the variable that defines the number of successes. The interval is constructed so that it is as close as possible to the required significance level. The algorithm is useful for small samples, for which the approximation to normality cannot be accepted, but the simplicity of its construction makes it applicable to practically any sample volume from real-world applications. The solution proposed by the algorithm is the construction of a sequence of monotonically increasing integers (between 0 and the volume of the sample), with which the confidence interval is constructed. Thus, if $N_x$ is the series of numbers (with x from 0 to m, several cases being listed in Appendix A), then the sequence of confidence intervals for the variable x extracted with replacement from the sample of volume m is $[N_x,m-N_{m-x}]$.