Indicators of Evidence for Bioequivalence

Abstract

Some equivalence tests are based on two one-sided tests, where in many applications the test statistics are approximately normal. We define and find evidence for equivalence in Z-tests and then one- and two-sample binomial tests as well as for t-tests. Multivariate equivalence tests are typically based on statistics with non-central chi-squared or non-central F distributions in which the non-centrality parameter λ is a measure of heterogeneity of several groups. Classical tests of the null $\lambda \ge {\lambda }_0$ versus the equivalence alternative $\lambda <{\lambda }_0$ are available, but simple formulae for power functions are not. In these tests, the equivalence limit ${\lambda }_0$ is typically chosen by context. We provide extensions of classical variance stabilizing transformations for the non-central chi-squared and F distributions that are easy to implement and which lead to indicators of evidence for equivalence. Approximate power functions are also obtained via simple expressions for the expected evidence in these equivalence tests.

1. Introduction

Our purpose is to extend the concept of “evidence for the alternative hypothesis”, already available in classical one-sided testing, to contexts where that alternative is “equivalence” of two or more distributions. We abbreviate the term “bioequivalence” to “equivalence” for simplicity and because these results are of much more general applicability.

1.1. Background and Summary

Why should we introduce another approach to equivalence testing? Because, even though some equivalence tests [1] are well established and embraced by the USA Food and Drug Administration (FDA) and the European Medicines Agency (EMA), there are substantial critiques [2,3,4], as well as novel, competing approaches to multivariate equivalence testing [4,5,6,7,8].

We endorse the proposal by [4] to define a hierarchy of bioequivalence models, “average bioequivalence” within “population bioequivalence” within “individual bioequivalence”, in terms of the Kullback–Leibler symmetrized distance (KLD) between distributions arising in equivalence tests. Somewhat surprisingly, we advocate estimating these distances indirectly using variance stabilized test statistics (VSTs) rather than plug-in parameter estimates for parameters appearing in the KLD. The close tie between the mean of a VST and the KLD for non-central chi-squared and F distributions will be illustrated in Appendix A.

In the remainder of this introduction we describe the notion of “evidence for the alternative” in the context of Z-tests and show how it is connected to level and power of such tests; these ideas were first introduced in [9]. After these preliminaries, we extend the notion of evidence for equivalence to two one-sided Z-tests (TOSTs), and show how it is related to a one-sided test based on a non-central chi-squared statistic with one degree of freedom.

This notion of evidence is applicable to many situations because VSTs can carry many test statistics into normally distributed statistics Z. For exponential families, Reference [10] show that the expected evidence of the variance stabilized statistic T is approximately equal to the signed square root of the Kullback–Leibler symmetrized divergence. Examples not from exponential families are in [11,12,13]. These results gives support to calling T the “evidence for the alternative hypothesis”.

Throughout the paper we employ standard notation and properties of the non-central t, ${\chi }^2$ and F distributions which are introduced and studied in depth in [14,15]. Below we first discuss specific examples of two one-sided tests (TOST) in Section 2, namely for binomial, two by two tables and t-tests; all methods are illustrated using data examples from the literature. In the multivariate setting, chi-squared tests for equivalence are given in Section 3 and F-tests for equivalence of K normal populations in Section 4. New methods for choosing ${\lambda }_0$ and for variance stabilizing the test statistics are provided. Discussion follows in Section 5, and R scripts for implementing some procedures are found in the Appendix B.

1.2. Properties of Evidence in One-Sided Z-Tests

Univariate equivalence tests are often based on statistics having the non-central t distribution or normal approximations to the binomial distribution, so we begin our introduction to evidence contained in such tests with the approximating, and simpler, Z-tests.

In the prototypical model $X\sim N(\mu ,1)$ where μ is unknown, and one tests the null hypothesis $\mu \le {\mu }_0$ against the alternative $\mu >{\mu }_0$ by rejecting the null if the p-value is sufficiently small. For an observed $X=x$, the p-value is $\Phi ({\mu }_0-x),$ where Φ is the standard normal distribution function. The ”evidence for the alternative” $\mu >{\mu }_0$ is defined to be $T\equiv X-{\mu }_0$. It is normally distributed with mean $\mu -{\mu }_0$, which is linearly increasing in μ. In addition, T estimates its mean with standard error 1, regardless of the value of $\mu .$ For reasons given in [9,16] values of T near 1.645, 3.3 and 5 are called “weak”, “moderate” and “strong” evidence for the alternative. Note that for an observed $T=t=x-{\mu }_0$, the p-value can be recovered from $1-\Phi (t).$

In

Table 1. p-values for testing $\mu =0$ against $\mu >0$ and evidence estimates for alternatives based on one observation $T=t$, where $T\sim N(\mu ,1)$. Keep in mind that the standard error of the observed value of t is equal to 1.

t	0	1.281	1.645	2.326	3.090	3.3	3.719	5
p-value	0.5	0.10	0.05	0.01	0.001	0.0005	0.0001	0.0000003

are shown two sets of numbers with very different interpretations, resulting from different assumptions. They are based on one observation $T=t$, where $T\sim N(\mu ,1)$ and $\mu \ge 0$. If one assumes the boundary hypothesis $\mu =0$, the second row of p-values gives the correct “degree of surprise” at having observed $T=t$; the smaller the p-value, the more surprised one is with the outcome. However, if one only assumes $\mu \ge 0$, the first row gives an estimate of the expected evidence $\mathrm{E}\left[T\right]=\mu$ for the alternative $\mu >0$. This estimate has an additive standard normal error. When one interprets p-values, one must be careful not to interpret the smallness of their magnitudes as though they were evidence for the alternative on a linear scale. The first row in Table 1 gives a much more reasonable estimate of “evidence for the alternative” together with an easily understood standard error. The compatibility of this calibration scale for evidence with Bayesian calibration scales for p-values and Bayes factors is discussed in [10] (Section 4.3).

The generality of this definition of evidence for the alternative stems from the fact that in many situations the natural test statistics X can be transformed to T, which has approximately a normal distribution with unit variance. Also, it is a more basic concept of the test than level and power. For a normally distributed test statistic T with unit variance, the expected evidence for one-sided alternatives $\mu >{\mu }_0$ is related to the level α and power $1-\beta (\mu )$ through the sum of the probits:

$${\mathrm{E}}_{\mu }\left[T\right]=z_{1-\alpha }+z_{1-\beta (\mu )}.$$

(1)

In the same testing problem the sample size n required to detect an alternative ${\mu }_1$ with power 0.8 at level 0.05 is the solution of $\sqrt{n}({\mu }_1-{\mu }_0)=z_{0.95}+z_{0.8}\approx 2.5;$ the expected evidence in such an experiment therefore lies between weak and moderate.

For negative T, $-T$ is interpreted as evidence for the null $\mu \le {\mu }_0$. Because of the symmetry of the problem, if we had begun with the null hypothesis $\mu \ge {\mu }_0$ against the alternative $\mu <{\mu }_0$ the evidence for the alternative would be defined as ${\mu }_0-X$.

1.3. Properties of Evidence in Two One-Sided Z-Tests (TOST)

Consider the simplest example with one observation $X\sim N(\mu ,1)$ for testing $H_0:\left|\mu \right|\ge {\mu }_0,$ where ${\mu }_0>0$ defines the equivalence alternative $H_1:\left|\mu \right|<{\mu }_0$. The null hypothesis consists of two possibilities: $\mu \le -{\mu }_0$ and $\mu \ge {\mu }_0$. The left hand part is rejected at level α if $X+{\mu }_0\ge z_{1-\alpha }$, and the evidence for its alternative is $T_{-}=X+{\mu }_0$, because $T_{-}\sim N(\mu +{\mu }_0,1)$ and $T_{-}$ has an expected value that increases in μ and is 0 at the boundary $\mu =-{\mu }_0$. The right hand part is rejected at level α if $X-{\mu }_0\le z_{\alpha }$, and the evidence for its alternative is $T_{+}={\mu }_0-X$, because $T_{+}\sim N({\mu }_0-\mu ,1)$, whose expected value is increasing with decreasing μ and is 0 at its null boundary. The two one-sided testing procedure (equivalence test) rejects in favor of equivalence only if both of the one-sided tests reject their respective null hypotheses, and this has level $\alpha ,$ because only one null hypothesis can hold.

The evidence for the alternative hypothesis of equivalence is logically the minimum of the evidences for the two one-sided tests:

$$T=\text{min}\{T_{-},T_{+}\}={\mu }_0+T_0,$$

(2)

where $T_0=\text{min}\{X,-X\}$ and $X\sim N(\mu ,1)$. Now $-T_0=\left|X\right|$ has a folded (to the right) normal distribution ([14] (p. 170), and [16]), with parameters ($\mu ,1$), so $T_0$ has a folded (to the left) normal distribution with the same parameters. The density of $T_0$ is given in terms of the standard normal density $\phi$ for $t<0$ by $f_{T_0}(t;\mu )=\phi (t-\mu )+\phi (t+\mu ),$ so the density of T is for $t<0$

$$f_T(t;\mu )=\phi (t-\mu -{\mu }_0)+\phi (t+\mu -{\mu }_0).$$

(3)

In Figure 1 are shown in black lines some examples of $f_T(t)$ for ${\mu }_0=4$ and several choices of μ. When $\mu =0$, (exact equivalence), the density is negative half-normal with upper bound ${\mu }_0=4,$ but as $\left|\mu \right|$ increases, the distribution rapidly approaches normality.

Also of interest are the mean and standard deviation of T as μ varies. The first two moments of $T_0$ are ${\text{E}}_{\mu }\left[T_0\right]=\mu \{1-2\Phi (\mu )\}-2\phi (\mu )$ and ${\text{E}}_{\mu }\left[T_0^2\right]=1+{\mu }^2$, so

$$\begin{array}{ccc}\hfill {\mathrm{E}}_{\mu }\left[T\right]& =& {\mu }_0+{\mathrm{E}}_{\mu }\left[T_0\right]={\mu }_0+\mu \{1-2\Phi (\mu )\}-2\phi (\mu )\hfill \end{array}$$

(4)

$$\begin{array}{ccc}\hfill {\text{Var}}_{\mu }\left[T\right]& =& 1+{\mu }^2-{\left\{\mu \left(1-2\Phi (\mu )\right)-2\phi (\mu )\right\}}^2.\hfill \end{array}$$

(5)

The mean and standard deviation of T are shown in Figure 2 as black lines. The top left-hand plot reveals that even in the case of perfect equivalence $\mu =0$, the equivalence test with ${\mu }_0=3$ will only yield, on average, weak evidence for it. With ${\mu }_0=4$, this average evidence for equivalence when $\mu =0$ becomes moderate.

The plots in Figure 2 suggest that, except for μ near 0, the distribution of T defined by (2) is approximately normal with mean near $\text{min}\{\mu +{\mu }_0,{\mu }_0-\mu \}$ and standard deviation near one. For μ near 0, which is of interest in the case of equivalence, it is not normal, but this is perhaps compensated for by having a smaller standard error. Its distribution approaches a negative half-normal as $\left|\mu \right|\to 0,$ with mean and standard deviation from (4) and (5) converging to ${\mu }_0-\sqrt{2/\pi }$ and $\sqrt{1-2/\pi }$, respectively.

1.3.1. How Evidence Grows with Sample Size in Two One-Sided Z-Tests

The evidence for a one-sided Z-test grows with the square root of the sample size $\sqrt{n}$ for if the sample mean ${\overline{X}}_n\sim N(\mu ,1/n)$ then the evidence for the alternative $\mu >-{\mu }_0$ to the null $\mu \le -{\mu }_0$ is $T_{n,-}=\sqrt{n}(\overline{X}+{\mu }_0)\sim N(\sqrt{n}(\mu +{\mu }_0),1)$, which is increasing in μ, has variance 1, and has expected value 0 at the boundary $\mu =-{\mu }_0$. Similarly, $T_{n,+}=\sqrt{n}({\mu }_0-\overline{X})\sim N(\sqrt{n}({\mu }_0-\mu ),1)$ the evidence for the alternative $\mu <{\mu }_0$ to the null $\mu \ge {\mu }_0$. Thus the evidence for equivalence $\left|\mu \right|<{\mu }_0$ based on n observations is:

$$T_n=\text{min}\{T_{n,-},T_{n,+}\}=\sqrt{n}{\mu }_0+T_{n,0},$$

(6)

where $T_{n,0}=\sqrt{n}\text{min}\{\overline{X},-\overline{X}\}$ and $\sqrt{n}\overline{X}\sim N(\sqrt{n}\mu ,1)$. Now $T_{n,0}$ has a folded to the left normal distribution with parameters $\sqrt{n}\mu ,1$ and $T_n$ is a shift by $\sqrt{n}{\mu }_0$ of $T_{n,0}$, so its mean and variance are:

$$\begin{array}{ccc}\hfill {\text{E}}_{\mu }\left[T_n\right]& =& \sqrt{n}{\mu }_0+\sqrt{n}\mu \{1-2\Phi (\sqrt{n}\mu )\}-2\phi (\sqrt{n}\mu )\hfill \\ \hfill {\text{Var}}_{\mu }\left[T_n\right]& =& 1+n{\mu }^2-{\left\{\sqrt{n}\mu \left(1-2\Phi (\sqrt{n}\mu )\right)-2\phi (\sqrt{n}\mu )\right\}}^2.\hfill \end{array}$$

A plot of the densities of $T_4$ compared to T defined by (3) are also shown in Figure 1 as red lines, and similarly for the mean and standard deviation of $T_4$ in Figure 2.

1.3.2. Sample Size Determination

For a one-sided Z-test based on n observations one can obtain a given expected evidence 2.5, say, for an alternative distant ${\mu }_0$ from the null, by taking n to satisfy $\sqrt{n}{\mu }_0=2.5$. So $n_{\text{1-sided}}=\lceil {(2.5/{\mu }_0)}^2\rceil ,$ where $\lceil r\rceil$ is the smallest integer greater than or equal to r. For a TOST Z-test with equivalence alternative $\left|\mu \right|<{\mu }_0$, to obtain the same expected evidence when in fact $\mu =0$ one needs by (7) to have $\sqrt{n}{\mu }_0-\sqrt{2/\pi }=2.5,$ or $n_{\text{TOST}}={\lceil (2.5+0.8)/{\mu }_0)}^2\rceil ,$ which is 74% larger than $n_{\text{1-sided}}$.

If one had asked for only weak expected evidence 1.645 instead of 2.5 in the above paragraph, the ratio of sample sizes required by TOST Z-tests to a one-sided test is 2.2, so the equivalence test would require 120% more observations than the one-sided test.

1.4. Connection of Evidence in TOST with a One-Sided Test

The evidence $T=T_{\text{TOST}}$ defined by (2) is based on $X\sim N(\mu ,1)$ for the null hypothesis $H_0:\left|\mu \right|\ge {\mu }_0>0$, composed of two disjoint sets, with the equivalence alternative $H_1:\left|\mu \right|<{\mu }_0$. One could equally study the evidence in the equivalent experiment $S=X^2\sim {\chi }_1^2(\lambda )$, where $\lambda ={\mu }^2$, for the hypotheses restated as $H_0:\lambda \ge {\lambda }_0$ against the equivalence alternative $H_1:\lambda <{\lambda }_0,$ where ${\lambda }_0={\mu }_0^2$. The evidence for equivalence in an experiment with $S\sim {\chi }_1^2(\lambda )$ is a special case of (13) found in Section 3:

$$T=T_{1,{\lambda }_0}(S)=\left\{\begin{array}{cc}\sqrt{{\lambda }_0}-S/\sqrt{2}\hfill & \text{for}S<1;\hfill \\ \sqrt{{\lambda }_0}-\sqrt{S-1/2}\hfill & \text{for}S\ge 1.\hfill \end{array}\right.$$

(7)

The top left plot of Figure 3 compares the graph of this T with that of $T_{\text{TOST}}$ when ${\mu }_0=2$. Further, its expected evidence is by (12) approximately $\sqrt{{\lambda }_0+1/2}-\sqrt{\lambda +1/2}.$ The top right plot of Figure 3 shows the graph of this approximate expected evidence for equivalence as a function of μ for the case ${\mu }_0=2$ as a dashed line, to be compared with the previously obtained expected evidence in the two one-sided test experiment (4), whose graph is shown as a solid line. The bottom plots are for ${\mu }_0=4$.

2. More Examples of Two One-Sided Tests (TOSTS)

2.1. Evidence for Equivalence in Two One-Sided Binomial Tests

Given $X\sim$ Binomial$(n,p)$, and let $\widehat{p}=X/n$. Let $p_1<p<p_2$ define the region of equivalence. (Often this region will also be of the form $|p-p_0|<{\Delta }_0$ for some $p_0,{\Delta }_0$.) We want to test at level α the null $H_0:p\le p_1$ or $p\ge p_2$ against the equivalence alternative $H_1:p_1<p<p_2$. The null hypothesis is two-sided and its right-hand part is rejected at level α if $\widehat{p}\le p_2-z_{\alpha }\sqrt{p_2(1-p_2)/n},$ whereas the left-hand part is rejected at level α if $\widehat{p}\ge p_1+z_{1-\alpha }\sqrt{p_1(1-p_1)/n}.$ Only one of these tests can reject the null, so the level of the combined tests is α. We have assumed that n is large enough so that normal critical points give accurate levels.

The VST of $\widehat{p}$ is the well-known arc-sine transformation $h(\widehat{p})=2\sqrt{n}arcsin(\sqrt{\widehat{p}})$, which is asymptotically normal with variance 1 and asymptotic mean $2\sqrt{n}arcsin(\sqrt{p})$. Large values of $h(\widehat{p})$ indicate evidence for large p. The evidence in the test of $p\le p_1$ for an alternative $p>p_1$ is therefore $T_{-}=h(\widehat{p})-h(p_1)$, while the evidence in the test of $p\ge p_2$ for an alternative $p<p_2$ is $T_{+}=h(p_2)-h(\widehat{p})$. For the combined two one-sided tests, the evidence for equivalence is the minimum evidence in these two one-sided tests; that is, $T=\text{min}\{T_{-},T_{+}\}$.

Example 1. 

(Intervention success of new treatment.) In Example 4.2 of [17] (p. 59) a highly toxic drug used in chemotherapy treatment for a tumor led to a 73% two-year progression-free survival period. A new combined and much more tolerable treatment was administered to 361 patients and it was deemed equivalent to the previous treatment if the success rate fell in the interval [0.65, 0.75]. In the new treatment 191 patients survived a two-year progression-free period, and Wellek used two one-sided binomial tests to find that non-equivalence was rejected at the 0.05 level with estimated power 0.12. For these data $T_{+}=2.84$, $T_{-}=0.793$ so the evidence for equivalence of the treatment effect is $T=0.793$, which is “weak”. The standard error of $T=0.793$ is known (see the comments at the end of Section 2), to satisfy $0.60\le \text{SE}\left[T\right]\le 1$, but is likely to near the smaller bound, because $\widehat{p}=191/273$ is close to the center of the equivalence interval. This result is consistent with the analysis of [17], and it is much simpler.

2.2. Evidence for Equivalence of Risks

It is often the case that one wants to compare risks associated with new and standard treatments, with data often displayed in 2 by 2 tables; an example is given below after we introduce notation and explain how to find evidence for equivalence in this context.

Let $X_1$ and $X_2$ be two independent binomial random variables with parameters $(n_1,p_1)$ and $(n_2,p_2)$, respectively. Letting ${\widehat{p}}_i=(X_i+0.5)/(n_i+1)$ for $i=1,2$ the unknown risk difference $\Delta =p_1-p_2$ is estimated by $\widehat{\Delta }={\widehat{p}}_1-{\widehat{p}}_2$. We want the evidence for equivalence hypothesis ${\Delta }_1<\Delta <{\Delta }_2$, where ${\Delta }_1,{\Delta }_2$ are specified bounds, usually of the form $-{\Delta }_0,{\Delta }_0$. This can be achieved by combining the results of two one-sided tests: $\Delta \le {\Delta }_1$ versus $\Delta >{\Delta }_1$ and $\Delta \ge {\Delta }_2$ versus $\Delta <{\Delta }_2$. To find the evidence for the alternative in the first test, we use the VST of $\widehat{\Delta }$ derived in Kulinskaya et al. [11]. This is a family of VSTs indexed by a parameter $0<A<1$. For the choice $A=1/2$ the nuisance parameter is $\psi =\overline{p}=(p_1+p_2)/2$, and we also require $N=n_1+n_2$, $v=(1-2\overline{p})(1/2-n2/N)$ and $w=\sqrt{\overline{p}(1-\overline{p})+v^2}.$ Then Equation 2.3 of [11], can be written:

$$T(\widehat{\Delta },\overline{p},{\Delta }_1)=\sqrt{\frac{4n_1{n}_2}{N}}\left(arcsin\left(\frac{\widehat{\Delta }/2+v}{w}\right)-arcsin\left(\frac{{\Delta }_1/2+v}{w}\right)\right).$$

(8)

Reference [11] show that the statistic $T(\widehat{\Delta },\widehat{\overline{p}},{\Delta }_1)$ obtained by replacing $\overline{p}$, v and w by their plug-in estimates, is for large $n_1,n_2$ normally distributed with mean that is monotone increasing in Δ from 0 at the null $\Delta ={\Delta }_1$. Further, this statistic has variance 1 at the null, which allows them to derived large-sample confidence intervals for Δ; these intervals are shown to be quite competitive for even small to moderate sample sizes in [13]. Next define

$$\begin{array}{ccc}\hfill T_{-}& =& T(\widehat{\Delta },\widehat{\overline{p}},{\Delta }_1)\hfill \\ \hfill T_{+}& =& -T(\widehat{\Delta },\widehat{\overline{p}},{\Delta }_2)\hfill \\ \hfill T& =& \text{min}\{T_{-},T_{+}\}.\hfill \end{array}$$

(9)

$T_{-}$ gives the putative evidence for the alternative $\Delta >{\Delta }_1$ for the null $\Delta \le {\Delta }_1$, while $T_{+}$ gives the putative evidence for the alternative $\Delta <{\Delta }_2$ for the null $\Delta \ge {\Delta }_2$. We say “putative” because, as [11] point out, the variances of these statistics can stray far from 1 if Δ is not near the null. However, the evidence T for equivalence ${\Delta }_1<\Delta <{\Delta }_2$ is better behaved, and has standard error similar to that for the two one-sided Z-test evidence discussed in Section 2. R scripts for computing (9) are in Appendix B.

Example 2. 

(Comparing methods of patient care.) As described in [18], the objective of a randomized trial was to determine whether a standard method of care for patients by doctors was comparable to nurse-practitioner care. For the first group, there were $n_1=225$ patients and of these $X_1=148$ were found to have adequate care. For the second group, of $n_2=167$ patients, $X_2=115$ were found to have adequate care. Letting $p_1,p_2$ be the probability of adequate care for the first, second methods and $\Delta =p_1-p_2$, it was desired to test for “equivalence of treatments” defined by $|\Delta |\le {\Delta }_0=0.1$. For these data, $\widehat{\Delta }=-0.03$, $T_{-}=1.461,$ $T_{+}=2.719$ and $T=1.461,$ which is close to $1.65$ with a standard error less than 1. That is, the evidence for equivalence is positive but weak. By way of comparison, Reference [18] found the p-value for the equivalence alternative $|\Delta |\le {\Delta }_0=0.1$ to be 0.005, but in a later corrected analysis in [19] calculated it to be 0.07.

In order to obtain expected moderate evidence 3.3 ± 1 for equivalence $|\Delta |\le {\Delta }_0=0.1$ in this setting when in fact there is near equivalence, one would need sample sizes in each group near 1000.

2.3. Evidence for Equivalence in Two One-Sided t-Tests

For the t-test, the equivalence test of $H_0:\left|\mu \right|\ge {\mu }_0$ against the alternative $H_1:\left|\mu \right|<{\mu }_0$ is based on n measurements of the differential effect. The t-statistic is a function of the estimated mean $\overline{y}$ and standard deviation s obtained from the sample. The null hypothesis is two-sided and its right-hand part is rejected if $S_{+}=\sqrt{n}(\overline{y}-{\mu }_0)/s<q{t}_{n-1,\alpha },$ whereas the left-hand part is rejected if $S_{-}=\sqrt{n}(\overline{y}+{\mu }_0)/s>q{t}_{n-1,1-\alpha }.$ In both of these expressions, $q{t}_{n-1,p}$ denotes the p-quantile of the corresponding t-distribution. Both parts of the null hypothesis must be rejected in order to get significant evidence for equivalence. This holds if the confidence interval $[\overline{y}\pm q{t}_{n-1,1-\alpha }s/\sqrt{n}]$ is contained within $[\pm {\mu }_0]$.

The t-statistic S, if the true mean is μ, has a non-central t-distribution with $n-1$ degrees of freedom and noncentrality parameter $\lambda =\sqrt{n}(\mu -{\mu }_0)/\sigma$. We are interested in evidence in favor of small $\left|\mu \right|$. For the left-hand part, the non-centrality parameter is $\lambda =\sqrt{n}(\mu +{\mu }_0)/\sigma$ and we are interested in evidence in favor of large μ. The VST is derived in [9,20] and is defined by $h(S)=\sqrt{2n}{sinh}^{-1}\left(S/\sqrt{2n}\right),$ where ${sinh}^{-1}(x)=ln(x+\sqrt{x^2+1}).$ This is an increasing function and measures the evidence in favor of large μ. The evidence contained in the data in favor of equivalence for the right-hand part of the null hypothesis is thus, $-h(S_{+})$, while for the left-hand part it is $h(S_{-})$. Both need to be sufficiently large in order to conclude in favor of equivalence, that is, the empirical evidence

$$\widehat{E}=\text{min}\left\{-\sqrt{2n}{sinh}^{-1}\left(\frac{\overline{y}-{\mu }_0}{\sqrt{2}s}\right),\sqrt{2n}{sinh}^{-1}\left(\frac{\overline{y}+{\mu }_0}{\sqrt{2}s}\right)\right\}$$

must be at least 2 and better 3. Negative values of the empirical evidence can occur and they have to be interpreted as evidence in favor of non-equivalence.

Example 3. 

Figure 4 shows a plot of the empirical evidence as a function of the average of the measurements. The evidence from the t-statistic is nearly linear in $\overline{y}$ and largest if $\overline{y}$ is exactly halfway between the equivalence limits. The difference with the usual statistical tests is striking. There, the evidence will grow with the distance from the null hypothesis and can become arbitrarily large. Here, the maximal size of the evidence is limited by the equivalence limits.

The behavior of the evidence is as expected. If the sample size grows, so does the amount of evidence. If the standard deviation grows, there is less evidence if all other conditions are the same. The amount of evidence is bigger than a desired amount (2 or 4, for example), if $\overline{y}$ is within an interval centered at the halfway mark between the equivalence limits.

Approximate Normality of the Variance Stabilized t-Statistic

The VST is symmetric with regard to the origin, because $(x+\sqrt{x^2+1})=1/(-x+\sqrt{x^2+1})$, that is, $ln(x+\sqrt{x^2+1})=-ln(-x+\sqrt{x^2+1})$. The expansion

$$h(S)=S-\frac{1}{6}\frac{S^3}{2n}+\frac{3}{40}\frac{S^5}{4n^2}O(n^{-3}),$$

shows that for values of S up to order $O(n^{1/3})$, the deviation from the identity is small. Only for values of the t-statistic S further into the tail does the VST pull them towards zero, that is, $h(S)<S$. For very large values of S the function $h(S)$ is logarithmic. The tail of the t-density evaluated at x is $O(x^{-n})$ as $x\to \infty$ and thus has a tail index of $n-1$. The VST transforms this to an infinite tail index.

3. Evidence in Multivariate Equivalence Tests

Multivariate equivalence tests are often based on a test statistic S having an exact or approximate non-central chi-squared distribution, denoted $S\sim {\chi }_{\nu }^2(\lambda )$, where ν is the known degrees of freedom (df ), and the non-centrality parameter (ncp) $\lambda \ge 0$ is unknown. Others are based on the non-central F distribution, see Section 4. The null hypothesis postulates non-equivalence between the samples, $\lambda \ge {\lambda }_0$, whereas the alternatives postulate practical equivalence $\lambda <{\lambda }_0$. The limit ${\lambda }_0$ is a positive constant adapted to the context; examples are given in [17] and the following sections.

Wellek [8,20] looks at the case of possibly dependent measurements (K of them) that are done independently on n subjects. He then wants to test whether the K measurements have equal means. His first proposal is to pass to the $K-1$ differences between the K measurements and to use Hotelling’s $T^2$ test for 0 means. He then remarks on the elliptical shape of the equivalence region, which might be criticized as being arbitrary. Reference [20] then discusses rectangular regions, as we do, and comments on the difficulties of this approach. The material in Section 3.2 below proposes a possible compromise solution. A fully Bayesian approach to multivariate equivalence testing is found in [5].

3.1. A VST for the Non-Central Chi-Squared Statistic

Once the “equivalence limit” ${\lambda }_0$ is chosen, one can carry out a Neyman–Pearson test which rejects non-equivalence $\lambda \ge {\lambda }_0$ at level α in favor of equivalence when the test statistic is sufficiently small, that is, S less than the α-quantile of the ${\chi }_{\nu }^2({\lambda }_0)$ distribution ($c_{\alpha }={\chi }_{\nu ,\alpha }^2({\lambda }_0)$). The power function of this test is the probability of deciding in favor of equivalence

$$1-\beta (\lambda )=P_{\lambda }(S\le c_{\alpha }),0\le \lambda <{\lambda }_0,$$

(10)

where $\beta (\lambda )$ is the probability of falsely reaching a conclusion of non-equivalence.

The testing approach underlying (10) is easier to understand when the test statistic is variance stabilized. In this context a VST is a monotone decreasing function $h(S)$ of the test statistic S, which for all values of λ is approximately normal with variance one. Rather than summarizing the evidence by an accept/reject decision, by a p-value or by a confidence interval, we propose to use the statistic $T=h(S)-h({\text{E}}_{{\lambda }_0}\left[S\right])$ because it provides a more informed and interpretable measure of the evidence in favor of equivalence. The larger its value, the more evidence resides in the data in favor of equivalence. Since its variance remains close to one for all λ, it is only the value of T that matters. The expected evidence ${\text{E}}_{\lambda }\left[T\right]$ is

$${\mathcal{K}}_{{\lambda }_0}(\lambda )={\mathrm{E}}_{\lambda }\left[T\right]\approx h({\mathrm{E}}_{\lambda }\left[S\right])-h({\mathrm{E}}_{{\lambda }_0}\left[S\right]).$$

(11)

This is a quantity that increases monotonically as λ decreases to 0. By construction, it is 0 at the equivalence limit ${\lambda }_0$ and has a maximal value of ${\mathcal{K}}_{{\lambda }_0}(0)$. The observed evidence for equivalence can be reported as $T\pm 1$, indicating that evidence T for equivalence has a standard normal error.

One can derive an approximation for ${\mathcal{K}}_{{\lambda }_0}(\lambda )={\text{E}}_{\lambda }\left[T(S)\right]$ for the evidence in $S\sim {\chi }_{\nu }^2(\lambda )$ for testing $\lambda \ge {\lambda }_0$ against $\lambda <{\lambda }_0$ as follows. The variance ${\text{Var}}_{\lambda }\left[S\right]=2\nu +4\lambda$ is not constant (not stable) and in order to stabilize it asymptotically one can use the standard delta-method [9] (p. 242). Using the fact that ${\text{E}}_{\lambda }\left[S\right]=\nu +\lambda \ge \nu$, one has ${\text{Var}}_{\lambda }\left[S\right]=g({\text{E}}_{\lambda }\left[S\right])$, where $g(s)=4s-2\nu$ is defined for $s\ge \nu$. The transformation $h(s)$ that removes the dependence on λ is equal to an antiderivative of $-1/\sqrt{g(s)}$, which leads to $h(s)=-\sqrt{s-\nu /2}$, where the negative sign was chosen in order to obtain a decreasing function in s. This standard procedure fails to define a VST for all $s\ge 0$, because strictly speaking our function $h(s)$ should only be applied in the range $s\ge \nu$, and even if we tried to extend it towards $s=0$, its value is undefined for $s<\nu /2$

This problem is created by the zero crossing of $g(s)$ at $s=\nu /2$ and various ideas for extending $h(s)$ to the entire positive real line could be tried; see (13) below. For this or any other such choice agreeing with $h(s)$ for $s>\nu$, the transformed statistic $h^{*}(S)$ has approximate expected value ${\text{E}}_{\lambda }\left[h^{*}(S)\right]\doteq h^{*}({\text{E}}_{\lambda }\left[S\right])=h^{*}(\nu +\lambda )\doteq h(\nu +\lambda )=-\sqrt{\lambda +\nu /2}$. After centering $h^{*}(S)$ at the limit ${\lambda }_0$, one obtains the observed evidence $T=h^{*}(S)+\sqrt{{\lambda }_0+\nu /2}$. The expected evidence for equivalence, to first order, is thus

$${\mathcal{K}}_{{\lambda }_0}(\lambda )={\mathrm{E}}_{\lambda }\left[T\right]=\sqrt{{\lambda }_0+\nu /2}-\sqrt{\lambda +\nu /2}.$$

(12)

The expected evidence (12) in the experiment has a maximum at $\lambda =0$, namely ${\mathcal{K}}_{{\lambda }_0}(0)=\sqrt{{\lambda }_0+\nu /2}-\sqrt{\nu /2}$. The dotted line in Figure 5 shows what the equivalence values ${\lambda }_0$ must be as a function of ν, namely ${\lambda }_0={\mathcal{K}}_{{\lambda }_0}^2(0)+\sqrt{2\nu }{\mathcal{K}}_{{\lambda }_0}(0),$ where the maximal expected evidence is of varying strengths. For example, when $\nu =15$, for moderate maximal expected evidence, ${\lambda }_0$ must be at least 29. More frequently, the equivalence bound ${\lambda }_0$ is determined by context, and the degrees of freedom ν determined by $\nu \ge {({\lambda }_0-{\mathcal{K}}_{{\lambda }_0}^2(0))}^2/\{2{\mathcal{K}}_{{\lambda }_0}^2(0)\}$.

The evidence for $\lambda <{\lambda }_0$ in $S\sim {\chi }_{\nu }^2(\lambda )$ can be defined for certain $M>{\mathcal{K}}_{{\lambda }_0}(0)$:

$$T=T_{\nu ,{\lambda }_0}(S)=\left\{\begin{array}{cc}M-S/\sqrt{2\nu }\hfill & \text{for}S<\nu ;\hfill \\ M-\sqrt{S-\nu /2}\hfill & \text{for}S\ge \nu .\hfill \end{array}\right.$$

(13)

This T has a negative continuous derivative, and for the choice $M=\sqrt{{\lambda }_0}$ has $T\approx N(0,1)$ at the null $\lambda ={\lambda }_0$. Further, at perfect equivalence $\lambda =0$ it has $\mathrm{E}\left[T\right]\approx {\mathcal{K}}_{{\lambda }_0}(0)$ for ${\mathcal{K}}_{{\lambda }_0}$ given by (12). These claims are made based on simulation studies, using R scripts in Appendix B.

3.2. Evidence for Equal Means

Given independent $X_k\sim N({\mu }_k,1)$, $k=1,\cdots ,K$, we want to find evidence for equivalence in the sense that all means equal $\mu =({\sum }_k{\mu }_k)/K$ simultaneously. A test statistic is $S={\sum }_k{(X_k-\overline{X})}^2$, which has distribution $S\sim {\chi }_{K-1}^2(\lambda )$, where $\lambda ={\sum }_k{({\mu }_k-\mu )}^2$. In practice, the ${\mu }_k$ are considered “equal” if ${\text{max}}_k\left\{\right|{\mu }_k-\mu \left|\right\}\le ϵ$ for a given $ϵ>0$. How can this last notion of equivalence be translated into a value for the limit ${\lambda }_0$? To solve this problem, we note that $\{({\mu }_1-\mu ),\cdots ,({\mu }_K-\mu )\}$ defines a $K-1$ dimensional hyperplane of $R^K$, which when shifted to the origin is a $K-1$ dimensional subspace of $R^K$, and distances between points in the hyperplane are preserved under translation. Thus it suffices to solve the following problem for arbitrary $K\ge 2$: given ${\mu }_1,\cdots ,{\mu }_K$ with $\mu =({\sum }_k{\mu }_k)/K=0$ and ${\text{max}}_k\left\{\right|{\mu }_k\left|\right\}\le ϵ$, find an appropriate choice of ${\lambda }_0={\sum }_k{\mu }_k^2=r^2,$ which is the square of the Euclidean distance r of the point $({\mu }_1,\cdots ,{\mu }_K)$ from the origin in $R^K$. After a solution for the equivalence boundary ${\lambda }_0={\lambda }_0(ϵ,K)$ is found, it can be implemented in the case of unknown μ by replacing K by $K-1$.

The largest ball contained in the K-dimensional cube with edge $2ϵ$, both having the same center, has radius $ϵ,$ whereas the smallest ball containing the same cube has radius $\sqrt{K}ϵ$. The latter choice would allow for one ${\mu }_k$ to be as large as $\sqrt{K}ϵ$ (if all other ${\mu }_j=0$) and overall equivalence would be claimed even though it was violated in one case. To reduce this violation, the radius ought to be in-between the extremes. A “reasonable” compromise requires the volume of the $L_2$-ball to equal ${(2ϵ)}^K$, the volume of the cube with side $2ϵ$. The volume of a ball in $R^K$ with radius r is given by:

$$V_K(r)=\left\{\begin{array}{cc}\frac{{\pi }^m{r}^{2m}}{\Gamma (m+1)},\hfill & K=2m;\hfill \\ \frac{{\pi }^m{(2r)}^{2m+1}\Gamma (m+1)}{\Gamma (2m+2)},\hfill & K=2m+1.\hfill \end{array}\right.$$

(14)

To ensure equal volumes for the hypersphere and the hypercube for all K, one needs the radius of the ball to be $r_0=\sqrt{K{b}_K}ϵ$, where

$$b_K=\left\{\begin{array}{cc}\frac{4{\{\Gamma (1+K/2)\}}^{2/K}}{\pi K},\hfill & K=2m;\hfill \\ \frac{1}{{\pi }^{1-1/K}K}{\left\{\frac{\Gamma (K+1)}{\Gamma ((K+1)/2)}\right\}}^{2/K},\hfill & K=2m+1.\hfill \end{array}\right.$$

(15)

A good approximation to $b_K$ is given by $c_K=2/(\pi e)+{(1.7·K)}^{-5/6}$, see

Table 2. For selected values of K are shown the exact $b_K$ coefficients (15) to 3 decimal places so that the K-dimensional ball of radius $\sqrt{K{b}_K}ϵ$, has the same volume as the K-dimensional cube of side $2ϵ$. The approximate value is $c_K=2/(\pi e)+{(1.7·K)}^{-5/6}$.

K	2	3	4	5	6	7	8	10	50	100	∞
$b_K$	$2/\pi$	0.513	0.450	0.412	0.386	0.367	0.352	0.332	0.259	0.248	0.23420...
$c_K$	0.595	0.492	0.437	0.402	0.379	0.361	0.348	0.329	0.259	0.248	$2/(\pi e)$

and Figure 6. This makes it easier to choose the desired equivalence limit ${\lambda }_0=r_0^2=b_{K}K{ϵ}^2\sim 2K{ϵ}^2/(\pi e)$ for moderate and large K.

Example 4. 

To illustrate the use of this table, suppose $K=4$, μ is unknown, and we want each of ${\mu }_k$ to satisfy $|{\mu }_k-\mu |\le ϵ=1/2$, say, in order to claim “equivalence of all means”. As discussed above the corresponding problem with μ known to equal 0 in $K-1=3$ dimensions is to utilize ${\lambda }_0={\lambda }_0(ϵ,K-1)={\lambda }_0(1/2,3)=r_0^2=3b_3{ϵ}^2=3\times 0.513\times 0.25=0.3375$. From Equation (12) it then follows that for $K=4,$ $\nu =3$ and this ${\lambda }_0$ the maximum expected evidence possible for equivalence of all 4 means is about 0.13, which is almost negligible.

If we had begun this section with each $X_k$ replaced by ${\overline{X}}_k\sim N({\mu }_k,1/n)$, for some sample size $n\ge 2$, then the test statistic would be $S_n={\sum }_k{({\overline{X}}_k-\overline{\overline{X}})}^2$, which has distribution $S_n\sim {\chi }_{\nu }^2(\lambda )$, where $\nu =K-1$ and $\lambda =n{\sum }_k{({\mu }_k-\mu )}^2$. By imposing the same condition ${\text{max}}_k\left\{\right|{\mu }_k-\mu \left|\right\}\le ϵ$, where μ is unknown, the “equal-volume” solution ${\sum }_k{({\mu }_k-\mu )}^2\le =\nu b_{\nu }{ϵ}^2$, so the appropriate equivalence hypothesis is $\lambda \le {\lambda }_0=n\nu b_{\nu }{ϵ}^2$. The maximum expected evidence in $T_n=T(S_n)$ is attained when all means are equal, and this maximum is $\sqrt{{\lambda }_0+\nu /2}-\sqrt{\nu /2},$ or $\sqrt{n\nu b_{\nu }{ϵ}^2+\nu /2}-\sqrt{\nu /2}$ which is growing at rate $\sqrt{n}$. Continuing Example 4, $n=20$ will yield weak maximum expected evidence 1.65 for equivalence.

3.3. Application to between Group Sum of Squares

Given independent observations from K groups with different means $X_{ki}\sim N({\mu }_k,1),$ $i=1,\cdots ,n_k$, $k=1,\cdots ,K$, we denote the total sample size by $N={\sum }_k{n}_k$, and the group sample proportions by $q_k=n_k/N$. Let the kth sample mean be ${\overline{X}}_k$ and the weighted group sample mean by $\overline{X}={\sum }_k{q}_k{\overline{X}}_k$; it is an unbiased estimator of the weighted population mean $\mu ={\sum }_k{q}_k{\mu }_k$. Then the between group sum of squares $SS{B}_{\text{between}}=N{\sum }_k{q}_k{({\overline{X}}_k-\overline{X})}^2\sim {\chi }_{\nu }^2(\lambda )$, where $\nu =K-1$ and $\lambda =N{\sum }_k{q}_k{({\mu }_k-\mu )}^2$; see, for example, [9] (p. 184). The evidence in $S\sim {\chi }_{\nu }^2(\lambda )$ for equivalence $\lambda <{\lambda }_0$ was derived in Section 3.1.The practical problem is to choose ${\lambda }_0$. Once ${\lambda }_0$ is chosen one can compute the maximum evidence in the experiment; it is by (12) equal to $\sqrt{{\lambda }_0+(K-1)/2}-\sqrt{(K-1)/2}$. As explained in Section 3.1, this determines the maximum power for equivalence at any given level.

First we consider the argument of [17] (p. 164) for choosing ${\lambda }_0$. He introduces the parameter ${\psi }^2={\sum }_k\frac{n_k}{\overline{n}}{({\mu }_k-\mu )}^2$, where $\overline{n}=N/K,$ which he calls a generalized squared Euclidean distance between $({\mu }_1,\cdots ,{\mu }_K)$ and $(\mu ,\cdots ,\mu )$. He proposes to define homogeneity in terms of ${\psi }^2$ and the equivalence hypothesis by ${\psi }^2\le {ϵ}^2$, where ϵ is to be chosen. Note that our ncp $\lambda =\overline{n}{\psi }^2$, the same as his. For $K=2$ condition ${\psi }^2\le {ϵ}^2$ reduces to $|{\mu }_2-{\mu }_1|\le \sqrt{2}ϵ$. This leads [17] (p. 164) using conditions for comparing 2 normal populations, to suggest that, in general, one take ${\lambda }_{0,Wellek}=\overline{n}{ϵ}^2$, with ϵ ranging from $1/4$ to $1/2$ where $1/4$ yields a “strict” equivalence limit $\overline{n}/16$ while $ϵ=1/2$ leads to what he calls a “liberal” limit of $\overline{n}/4$. However, this approach assumes that the requirement ${\text{max}}_k\left\{\right|{\mu }_k-\mu \left|\right\}\le ϵ$ for a pre-specified ϵ holds for all K. From our point of view the choice of ${\lambda }_0$ should grow with $\sqrt{K}$ and ϵ should be determined by context.

In the balanced case $n_k\equiv n$, we can make direct comparisons between Wellek’s criterion and ours, derived near the end of Section 3.2, which yielded ${\lambda }_{0,MS}=n\nu b_{\nu }{ϵ}^2$. Assuming the same choice of ϵ, the ratio of ${\lambda }_{0,MS}/{\lambda }_{0,Wellek}=\nu b_{\nu }$. Using Table 2, this ratio varies with $\nu =1,2,3,\cdots$ considerably and equals, respectively, $0.64,1.03,1.35,1.65,1.93,\cdots ;$ further it grows with ν as $2\nu /(\pi e)$. Only for $\nu =2$ are the two criteria the same.

4. Testing for Equivalence of K Groups

Given independent observations from K groups with common unknown variance ${\sigma }^2>0$ and different means $X_{ki}\sim N({\mu }_k,{\sigma }^2),$ $i=1,\cdots ,n_k$, $k=1,\cdots ,K$. Let $N={\sum }_k{n}_k$, $q_k=n_k/N$ for all k and the overall mean $\mu ={\sum }_k{q}_k{\mu }_k$. We define equivalence of means for a given $ϵ>0$ if none of the standardized ${\mu }_k$ differs more than ϵ from zero:

$${\text{max}}_k\left\{\frac{|{\mu }_k-\mu |}{\sigma }\right\}\le ϵ.$$

(16)

Just as in Section 3.3 where σ was assumed known (and without loss of generaliity set equal to 1), we can define $\lambda =N{\sum }_k{q}_k{\{({\mu }_k-\mu )/\sigma \}}^2$. Given ${\lambda }_0$ we want evidence for the hypothesis of equivalence $\lambda <{\lambda }_0$. The arguments for choosing ${\lambda }_0={\lambda }_{0,MS}=n{\nu }_1{b}_{{\nu }_1}{ϵ}^2$ where ${\nu }_1=K-1$ carry over from Section 3.2 and Section 3.3, provided sampling is nearly balanced so that all $n_k\approx \overline{n}=n$.

The within group sum of squares is defined by

$$S{S}_{\text{within}}=\sum _{k-1}^K\sum _{i=1}^{n_i}{(X_{ki}-{\overline{X}}_k)}^2.$$

(17)

Standard theory [9] (p. 196) shows $S=(S_{\text{between}}/{\nu }_1)/\{S{S}_{\text{within}}/{\nu }_2\}$ has a non-central F distribution with df ${\nu }_1=K-1$, ${\nu }_2=N-K$ and ncp λ. A VST for the statistic S has been derived by [21] and also [9] (p. 197). It assumes ${\nu }_2>4$. Let $a^2=({\nu }_2-4)/2$ and $c^2={\nu }_2^2({\nu }_1+{\nu }_2-2)/\left\{{\nu }_1^2({\nu }_2-2)\right\}$. The VST is $h=h(S)$, defined by $h(s)=-a{cosh}^{-1}\left((s+{\nu }_2/{\nu }_1)/c\right)$. where the inverse hyperbolic cosine function is defined by ${cosh}^{-1}(x)=ln(x+\sqrt{x^2-1})$. Now ${cosh}^{-1}(x)$ is only defined for $\left|x\right|\le 1$ so the VST $h(s)$ is only defined for $s>c-{\nu }_2/{\mu }_1$.

The evidence for $\lambda <{\lambda }_0$ is defined by $T=h(S)-h({\text{E}}_{{\lambda }_0}\left[S\right])$. The expected value of the non-central F-statistic S is ${\text{E}}_{\lambda }\left[S\right]=({\nu }_1+\lambda ){\nu }_2/\left\{{\nu }_1({\nu }_2-2)\right\},$ so the expected evidence, to first order, is given by:

$${\mathcal{K}}_{{\lambda }_0}(\lambda )={\text{E}}_{\lambda }\left[h(S)\right]-h({\text{E}}_{{\lambda }_0}\left[S\right]).$$

(18)

Unfortunately the VST itself and hence the evidence $T=h(S)-h({\mathrm{E}}_{{\lambda }_0}\left[S\right])$ for equivalence is undefined for informative small values of S. To make it useful, we extend the evidence function to small values via monotone linearization in (19). Strictly speaking, the F-statistic VST was only derived for $s>b={\nu }_2/({\nu }_2-2)$, as mentioned in [9] (p. 197), for the same reason the chi-squared VST derivation had a limited domain, see Section 3.1. And, for the same reasons given there, we can extend it to $0\le s\le b={\nu }_2/({\nu }_2-2)$ without changing much the expectation (18). Evidence for equivalence $\lambda <{\lambda }_0$ based on $S\sim F_{{\nu }_1,{\nu }_2,\lambda }$ is defined for certain $M>{\mathcal{K}}_{{\lambda }_0}(0)$ by

$$T=T_{{\nu }_1,{\nu }_2,{\lambda }_0}(S)=\left\{\begin{array}{cc}M-aS{cosh}^{-1}\left((b+{\nu }_2/{\nu }_1)/c\right)/b\hfill & \text{for}S<b;\hfill \\ M-a{cosh}^{-1}\left((S+{\nu }_2/{\nu }_1)/c\right)\hfill & \text{for}S\ge b.\hfill \end{array}\right.$$

(19)

This T is continuous and increasing as S moves to 0. For the choice $M=\sqrt{{\lambda }_0-{\nu }_1/({\nu }_1+{\nu }_2)}$ it has $T\approx N(0,1)$ at the null $\lambda ={\lambda }_0;$ and, at perfect equivalence $\lambda =0$ it has $\mathrm{E}\left[T\right]\approx {\mathcal{K}}_{{\lambda }_0}(0)$ for the ${\mathcal{K}}_{{\lambda }_0}$ of (18).

Example 5. 

Example 7.1 of [17] (p. 165) considers four treatments for hypertension with measurements taken on diastolic blood pressure averaged over an interval. To test for equivalence of the treatments, the following summary data (sample size, mean, standard deviation) were recorded: $n_1=10$, ${\overline{x}}_1=99.8120$, $s_1=7.56391$; $n_2=12$, ${\overline{x}}_2=99.2903$, $s_2=5.9968$; $n_3=13$, ${\overline{x}}_3=100.0024$, $s_3=10.4809$; and $n_4=15$, ${\overline{x}}_4=98.6407$, $s_4=4.5309$. This yields $N=50$, $\overline{n}=12.5$ and $\overline{\overline{x}}=99.3849$. Continuing, $S{S}_{between}=15.196$ and $S{S}_{within}=2516.135$ so the F-test statistic $S=(S{S}_{between}/(K-1))/(S{S}_{within}/(N-K)=0.0926$. For $ϵ=0.5$ and ${\lambda }_{0,MS}=\overline{n}{\nu }_1{b}_{{\nu }_1}{ϵ}^2=4.219$ we have by (19) the evidence for equivalence $T=1.934$, which is slightly more than weak.

By way of comparison, [17] uses traditional hypothesis testing with the smaller equivalence boundary ${\lambda }_{0,Wellek}=\overline{n}{ϵ}^2=3.125$ and finds the non-central F-test significant at level 0.05 with an estimated power 0.18 for detecting perfect equivalence.

5. Summary and Discussion

For the test statistic $S\sim N(\mu ,1)$ of a null hypothesis $\mu \le {\mu }_0$ and alternative $\mu >{\mu }_0$ Neyman–Pearson methods help one make a decision; while a p-value can provide a measure of surprise regarding the boundary hypothesis $\mu ={\mu }_0$. But what one often wants from S is a measure of evidence for the alternative $\mu >{\mu }_0$. In this simplest of statistical tests, $T=S-{\mu }_0$ is such a measure of evidence for the alternative, because T estimates the unknown expected evidence $\mathrm{E}\left[T\right]=\mu -{\mu }_0$ which is linearly increasing with μ and comes with an easily understood standard normal error. Values of T near 1.645, 3.3 and 5 are interpreted as weak, moderate and strong evidence for the alternative $\mu >{\mu }_0$. In addition the expected evidence can be written as the sum of the probits of level and power.

The vast majority of routine statistical tests can be transformed into the above setting through variance stabilization. And the mean of a variance stabilized test statistic $T=\text{VST}(S)$, after centering at ${\mu }_0$, is very close to the signed square root of the Kullback–Leibler symmetrized divergence between the null and alternative distributions. This result gives more theoretical support for calling T the evidence for the alternative; references are listed in Section 1.

What we have done here is to extend the above ideas to the more complicated hypotheses of the form $\left|\mu \right|\ge {\mu }_0$ versus the equivalence alternative $\left|\mu \right|<{\mu }_0$, with applications for TOST based on one- and two-sample binomial experiments and two one-sided t-tests. Then, in the multivariate setting, we have found modifications of the classical VSTs for non-central chi-squared and non-central F-tests which make it practicable to find evidence for the equivalence alternative $\lambda <{\lambda }_0$ to the null hypothesis $\lambda \ge {\lambda }_0$ of non-equivalence.

The practical choice of equivalence limit ${\lambda }_0$ is also an important ingredient, and we have provided a new approach to assist in its choice. In particular, when testing for equivalence of means in K arms of a study, we found the value of the radius required so that the K-ball has the same volume as the K-cube of edge $2ϵ$. This leads to a proposal for converting the condition ${\text{max}}_k|{\mu }_k-\mu |\le ϵ$ into an approximate equivalence condition $\lambda <{\lambda }_0$.

The new extensions of classical VSTs for non-central chi-squared and F statistics require more work in the choice of M to center them properly at the null ${\lambda }_0$. Simulation studies show that adjusting M so that the mean of the VST is 0 when $\lambda ={\lambda }_0$ automatically ensures that the mean of the VST statistic when $\lambda =0$ is near its expected maximum value. The choice $M\approx \sqrt{{\lambda }_0}$ works adequately, but simple formulae for M that depend on the degree(s) of freedom as well as ${\lambda }_0$ would be useful.

Finally, we note that finding simple expressions for the expected evidence in a VST greatly assists one in finding minimal sample sizes when planning an experiment for determining equivalence; by knowing the maximum expected evidence for equivalence, one also learns of the power to detect perfect equivalence at any given level.

Another application is in goodness-of-fit tests, where instead of “backing into” a model by not rejecting it a liberal level such as 0.1, one could find the evidence for the model. Chapter 8 of [17] is a good starting point for solving this problem. Further research will also take into account recent results on bio-equivalence defined in terms of the Kullback–Leibler divergences, see [4,7].