New Equivalence Tests for Approximate Independence in Contingency Tables

Abstract

We introduce new equivalence tests for approximate independence in two-way contingency tables. The critical values are calculated asymptotically. The finite sample performance of the tests is improved by means of the bootstrap. An estimator of boundary points is developed to make the bootstrap based tests statistically efficient and computationally feasible. We compare the performance of the proposed tests for different table sizes by simulation. Then we apply the tests to real data sets.

1. Introduction

Testing for approximate row-column independence in two-way contingency tables is a common task in statistical practice. The first publications on this topic go back to Hodges and Lehmann [1], Diaconis and Efron [2]. More recently, Liu and Lindsay [3] applied the semi-parametric tubular tolerance regions to the row-column independence model in two-way contingency tables. The method relies on the analytical properties of the LRT statistic to obtain a closed form estimator of boundary points. Wellek [4] develops a test for independence in multi-way contingency tables in Section 9.2. For this purpose, he applies a test for consistency with a fully specified multinomial distribution as follows. First, the marginal distributions of the contingency table are calculated. The test statistic is the Euclidean distance between the product measure of the marginal distributions and the contingency table. The critical value is calculated asymptotically.

Ostrovski [5] proposes a general method to test equivalence to families of multinomial distributions, which is based on the minimum distance

$$d\left(p,\mathcal{M}\right)=\underset{q\in \mathcal{M}}{inf}d\left(p,q\right)$$

(1)

to a family $\mathcal{M}$ of multinomial distributions. If d is Euclidean distance and $\mathcal{M}$ is the independence model then the calculation of minimum distance Equation (1) requires numerical optimization. Generally, the method relies on the existence of a continuous minimizer of Equation (1). Unfortunately, it could not be shown if a continuous minimizer exists for the independence model. Instead, Ostrovski [5] assumes the existence of a continuous minimizer at all points and then applies the method to test for approximate independence. Additionally, numerical calculation of minimum distance Equation (1) makes the bootstrap test computationally intensive.

We follow the lines of [5], but avoid the numerical valuation of the minimum distance Equation (1) in the special case of independence testing. We also propose an efficient bootstrap test, which is based on the randomized estimator of the boundary points.

Any two-way contingency table of the size $k_1\times k_2$ corresponds to a probability matrix from ${\mathbb{R}}^{k_1}\times {\mathbb{R}}^{k_2}$. Let $p=\left(p_{ij}\right)$ denote the probability matrix. Let $\mathcal{M}$ be the independence model, which contains all product measures of the corresponding dimensions. The approximate row-column independence can be shown by testing

$$H_0=\left\{d\left(p,\mathcal{M}\right)\ge \epsilon \right\}\mathrm{against}{H}_1=\left\{d\left(p,\mathcal{M}\right)<\epsilon \right\},$$

(2)

where $\epsilon >0$ is a tolerance parameter.

Let r and c denote the probability vectors of the marginal distributions, which are defined by $r_i={\sum }_{j=1}^{k_2}{p}_{ij}$ and $c_j={\sum }_{i=1}^{k_1}{p}_{ij}$. A probability matrix p belongs to $\mathcal{M}$ iff the equality $p_{ij}=r_i{c}_j$ is fulfilled for all $p_{ij}$. We consider the transformations $h_a$ and $h_r$ of the matrix p, which are defined by $h_a\left(p\right)=\left(p_{ij}-r_i{c}_j\right)$ and $h_r\left(p\right)=\left(\frac{p_{ij}}{r_i{c}_j}\right).$

For any differentiable distance l on ${\mathbb{R}}^{k_1}\times {\mathbb{R}}^{k_2}$ we define two new distances $d_a\left(p,q\right)=l\left(h_a\left(p\right),h_a\left(q\right)\right)$ and $d_r\left(p,q\right)=l\left(h_r\left(p\right),h_r\left(q\right)\right)$. It should be noted that $d_a$ and $d_r$ are only pseudo-metrics because $d_r\left(p,q\right)=0$ or $d_a\left(p,q\right)=0$ does not imply $p=q$. We put these distances in Equation (1) and obtain

$$d_a\left(p,\mathcal{M}\right)=\underset{q\in \mathcal{M}}{inf}{d}_a\left(p,q\right)=\underset{q\in \mathcal{M}}{inf}l\left(h_a\left(p\right),0\right)=l\left(h_a\left(p\right),0\right)$$

and $d_r\left(p,\mathcal{M}\right)=l\left(h_r\left(p\right),1\right)$, where 0 denotes the zero matrix and 1 is the matrix of ones. The distances $d_a\left(p,\mathcal{M}\right)$ and $d_r\left(p,\mathcal{M}\right)$ can be interpreted respectively as the absolute deviation and the relative deviation between p and the product measure of the marginal distributions. The distances $d_a\left(p,\mathcal{M}\right)$ and $d_r\left(p,\mathcal{M}\right)$ are easy to calculate without optimization.

Therefore, $d_a$ and $d_r$ are good candidates for the general distance d in Definition (1) and we will use only these two specific distances in remainder of the paper.

We observe a contingency table $p_n$ of relative frequencies, where n is the sample size and p is the true underlying probability matrix. Then the test statistic for Equation (2) is $T_a\left(p_n\right)=\sqrt{n}\left(d_a\left(p_n,\mathcal{M}\right)-\epsilon \right)$ or $T_r\left(p_n\right)=\sqrt{n}\left(d_r\left(p_n,\mathcal{M}\right)-\epsilon \right)$ depending on user preference. Below we write $d_{*}$ instead of $d_a$ and $d_r$ if the statements are correct for both distances. We use the subscript * instead of a and r, if appropriate.

2. Asymptotic Tests

In this section, we derive the asymptotic distribution of the test statistic and give a detailed description of the asymptotic test.

Let $v:{\mathbb{R}}^{k_1}\times {\mathbb{R}}^{k_2}\to {\mathbb{R}}^{k_1+k_2}$ be the usual bijection $v\left(p\right)=\left(p_{11},p_{12},\dots ,p_{k_1{k}_2}\right)$. Let ${\mathring{d}}_{*}$ denote the derivative of the function $q\mapsto d_{*}\left(v^{-1}\left(q\right),\mathcal{M}\right)$, which can be easily calculated using the chain rule.

Proposition 1.

Let p be a boundary point of $H_0$ and $q=v\left(p\right)$. Let $D_q$ denote a square diagonal matrix, whose diagonal entries are $q_1,\dots ,q_{k_1+k_2}$. Then the asymptotic distribution of $T_{*}\left(p_n\right)$ is Gaussian with mean zero and variance ${\sigma }_{*}\left(p\right)={\mathring{d}}_{*}\left(q\right)\mathsf{\Sigma }\left(q\right){\mathring{d}}_{*}{\left(q\right)}^t$, where $\mathsf{\Sigma }\left(q\right)=D_q-q{q}^t$ is a covariance matrix.

Proof. 

Let $q_n=v\left(p_n\right)$. The normalized vector $\sqrt{n}\left(q_n-q\right)$ converges weakly to a random variable, which is Gaussian with mean zero and covariance matrix $\mathsf{\Sigma }\left(q\right)$, see [6], Theorem 14.3-4 for details. The assertion follows by the delta method, see [7], p. 26, Theorem 3.1. □

The asymptotic variance ${\sigma }_{*}\left(p\right)$ is unknown and can be estimated by ${\sigma }_{*}\left(p_n\right)$. The estimator ${\sigma }_{*}\left(p_n\right)$ is consistent by the continuous mapping theorem because $p\mapsto {\sigma }_{*}\left(p\right)$ is a continuous function. Let $l_{\alpha }$ denote the lower $\alpha$-quantile of the normal distribution. Then the critical value of the asymptotic test is $l_{\alpha }{\sigma }_{*}\left(p_n\right)$. Now we have all components of the asymptotic test, which can be carried out as follows:

• Given are the contingency table $p_n$ of relative frequencies, the tolerance parameter $\epsilon$ and the significance level $\alpha$.
• Calculate the test statistic $T_{*}\left(p_n\right)=\sqrt{n}\left(d_{*}\left(p_n,\mathcal{M}\right)-\epsilon \right)$.
• Calculate the asymptotic variance ${\sigma }_{*}\left(p_n\right)$.
• Reject $H_0$ if $T_{*}\left(p_n\right)\le l_{\alpha }{\sigma }_{*}\left(p_n\right)$.

The outlined test is locally asymptotically most powerful, see [8], Proposition 3.

Remark 1.

The minimum tolerance parameter ε, for which the asymptotic test rejects $H_0$, can be calculated as $d_{*}\left(p_n,\mathcal{M}\right)-\frac{1}{\sqrt{n}}{l}_{\alpha }{\sigma }_{*}\left(p_n\right)$.

Remark 2.

The asymptotic test can be straightforward generalized for the multi-way contingency tables.

3. Bootstrap Tests

The parametric bootstrap is an efficient method to improve the finite sample performance of the proposed tests. Let $\partial H_0$ denote the boundary of $H_0$. Let ${\tilde{p}}_n$ denote an estimator of p, which fulfills the condition ${\tilde{p}}_n\in \partial H_0$. The critical value $c\left(\alpha ,p\right)$ can be estimated by $c\left(\alpha ,{\tilde{p}}_n\right)={sup}_{x\in \mathbb{R}}\left\{x:P\left(T_{*}\left({\tilde{p}}_n\right)\le x\right)\le \alpha \right\}$ because the critical value should be estimated so as if $H_0$ were true. The estimator $c\left(\alpha ,{\tilde{p}}_n\right)$ can be computed by the Monte Carlo method to any degree of accuracy.

The minimum distance estimator of p would be difficult to compute because the boundary $\partial H_0$ cannot be parameterized to apply common optimization techniques. Therefore, we propose a computationally feasible estimator of p, which is based on the randomized approximation to the minimum distance estimator.

Let q be some probability matrix such that $d_{*}\left(q,\mathcal{M}\right)>\epsilon$. If $d_{*}\left(p_n,\mathcal{M}\right)\le \epsilon$, then let $a_n$ be the largest number from $\left[0,1\right]$ such that $d_{*}\left(a_n{p}_n+\left(1-a_n\right)q,\mathcal{M}\right)=\epsilon$. Otherwise let $a_n=1$. The linear combination $c\left(p_n,q\right)=a_n{p}_n+\left(1-a_n\right)q$ is a consistent estimator of the boundary point p under additional requirements as shown below.

Proposition 2.

Assume that $d_{*}\left(ap+\left(1-a\right)q,\mathcal{M}\right)>\epsilon$ for all $a\in \left[0,1\right)$. Then $c\left(p_n,q\right)\to p$ a.e. for $n\to \infty$.

Proof. 

We show that $a_n\to 1$ for $n\to \infty$. Let $d_{*}\left(p_n,\mathcal{M}\right)<\epsilon$ because $a_n=1$ otherwise. The function $f:a\mapsto d_{*}\left(a{p}_n+\left(1-a\right)q,\mathcal{M}\right)$ is continuous on $\left[0,1\right]$ and $f\left(0\right)=d_{*}\left(q,\mathcal{M}\right)>\epsilon$ as well as $f\left(1\right)=d_{*}\left(p_n,\mathcal{M}\right)\le \epsilon$. Therefore, there exists a largest number $a_n\in \left[0,1\right]$ such that $f\left(a_n\right)=\epsilon$. It is worth mentioning that $a_n$ is a function of $p_n$.

Let $E=\left\{{lim}_{n\to \infty }{p}_n=p\right\}$. By the strong law of large numbers, $p_n$ converges to p a.e. for $n\to \infty$ and therefore $P\left(E\right)=1$. Let $\omega \in E$ be an arbitrary point and let $a_n\left(\omega \right)$ denote $a_n\left(f_n\left(\omega \right)\right)$. The sequence of $a_n\left(\omega \right)$ is bounded. Hence, there exists a convergent sub-sequence $a_{n_j}\left(\omega \right)\to a_0\left(\omega \right)$ for $j\to \infty$. We obtain

$$a_{n_j}\left(\omega \right)p_{n_j}\left(\omega \right)+\left(1-a_{n_j}\left(\omega \right)\right)q\to a_0\left(\omega \right)p+\left(1-a_0\left(\omega \right)\right)q$$

for $j\to \infty$ and consequently ${}_{*}\left(a_0\left(\omega \right)p+\left(1-a_0\left(\omega \right)\right)q,\mathcal{M}\right)=\epsilon$. We conclude $a_0\left(\omega \right)=1$ due to the assumption $d_{*}\left(ap+\left(1-a\right)q,\mathcal{M}\right)>\epsilon$ for all $a\in \left[0,1\right)$. Overall, we have shown that $a_n\left(\omega \right)\to 1$ for all $\omega \in E$. □

Let $Q=\left\{q_1,\dots ,q_m\right\}$ be a finite set of probability matrices, such that any $q_i$ fulfills $d_{*}\left(q_i,\mathcal{M}\right)>\epsilon$. We define the estimator ${\tilde{p}}_n$ as a minimum distance estimator among the linear combinations $c\left(p_n,q\right)$ for all $q\in Q$. Formally, the estimator ${\tilde{p}}_n$ equals $c\left(p_n,q_i\right)$, which fulfills the conditions $q_i\in Q$ and $l\left(c\left(p_n,q_i\right),p_n\right)={min}_{q\in Q}l\left(c\left(p_n,q\right),p_n\right)$. Note that the distance l is used to define the estimator ${\tilde{p}}_n$ because $d_{*}$ is a pseudo-metric only.

Corollary 1.

Let at least one $q\in Q$ satisfy $d_{*}\left(ap+\left(1-a\right)q,\mathcal{M}\right)>\epsilon$ for all $a\in \left[0,1\right)$. Then ${\tilde{p}}_n\to p$ a.e. for $n\to \infty$.

Proof. 

By definition of ${\tilde{p}}_n$, we obtain

$$l\left({\tilde{p}}_n,p_n\right)\le l\left(c\left(p_n,q\right),p_n\right)\le l\left(c\left(p_n,q\right),p\right)+l\left(p,p_n\right),$$

where $l\left(c\left(p_n,q\right),p\right)\to 0$ a.e. by Proposition 2 and $l\left(p,p_n\right)\to 0$ a.e. by the strong law of large numbers. □

The bootstrap test can be carried out as follows:

• Given are the contingency table $p_n$ of relative frequencies, the tolerance parameter $\epsilon$, the number of exterior points m and the significance level $\alpha$.
• Calculate the test statistic $T_{*}\left(p_n\right)=\sqrt{n}\left(d_{*}\left(p_n,\mathcal{M}\right)-\epsilon \right)$.
• If $T_{*}\left(p_n\right)\ge 0$ then set ${\tilde{p}}_n=p_n$ and go to step 7.
• Find m different points $q\in H_0$ such that $d_{*}\left(q,\mathcal{M}\right)>\epsilon$. The following rejection algorithm can be applied for the search:

  (a)

  Simulate a random matrix w whose entries are independently uniformly distributed on $\left[0,1\right]$.

  (b)

  Normalize w to a probability matrix q.

  (c)

  Add q to Q if $d_{*}\left(q,\mathcal{M}\right)>\epsilon$ or reject q otherwise.

  (d)

  Repeat previous steps until all exterior points are found.

• Solve the equation $d_{*}\left(a_n{p}_n+(1-a_n)q,\mathcal{M}\right)=\epsilon$ for $a_n$ using some root finding method. Repeat for all $q\in Q$.
• Find the minimum distance estimator ${\tilde{p}}_n$ among all linear combinations $c\left(p_n,q\right)$, where $q\in Q$.
• Estimate the critical value $c\left(\alpha ,{\tilde{p}}_n\right)$ using Monte Carlo simulation.
• Reject $H_0$ if $T_{*}\left(p_n\right)\le c\left(\alpha ,{\tilde{p}}_n\right)$.

Remark 3.

The bootstrap test is asymptotically consistent, see [9], Theorem 15.6.1. Consequently, the test is also locally asymptotically most powerful see [8], Proposition 3.

Remark 4.

The appropriate number of exterior points m can be found empirically. We found that $m=\left(k_1+k_2\right)\ast 50$ is sufficient and scales well with the table size.

Remark 5.

The minimum tolerance parameter ε, for which the bootstrap test rejects $H_0$, can be found numerically. For this purpose, the equation $T_{*}\left(p_n\right)=c\left(\alpha ,{\tilde{p}}_n\right)$ should be solved for the tolerance parameter ε using some root finding algorithm. The exterior points and bootstrap samples should remain unchanged during optimization.

4. Simulation Study of Finite Sample Performance

We study the finite sample performance of the proposed tests by the Monte Carlo simulation for different sample sizes and table sizes. The tests are implemented in VB.NET and available online, see https://github.com/TestingEquivalence/TestingApproximateIndependence.

The distance l is scaled Euclidean distance $l_2$, where the scale factor is necessary to obtain comparable test results for different table sizes. We use $l=\frac{1}{\sqrt{k_1{k}_2}}{l}_2$ in case of $d_r$ and $l=\sqrt{k_1{k}_2}{l}_2$ in case of $d_a$. Alternatively the smoothed total variation distance would be a good choice, see [8].

The minimum $\epsilon$, for which the test power equals $0.9$, is calculated for different table sizes and sample sizes at the uniform probability matrices for the purpose of throwing some light on the appropriate values of $\epsilon$ and the effective sample sizes.

Table 1. Minimum tolerance parameter $\epsilon$, for which the test power equals $0.9$ at nominal level $\alpha =0.05$, is calculated at the uniform probability matrices using the distance $d_r$. The sample size is $100,\dots ,10,000$.

Asymptotic Test
table size	100	200	500	1000	2000	5000	10,000
2 × 4	0.390	0.272	0.171	0.120	0.085	0.054	0.038
3 × 3	0.418	0.295	0.185	0.130	0.092	0.058	0.041
3 × 4	0.474	0.331	0.208	0.146	0.104	0.066	0.046
4 × 4	0.540	0.375	0.236	0.166	0.117	0.074	0.052
4 × 5	0.593	0.413	0.258	0.181	0.128	0.081	0.057
5 × 5	0.655	0.453	0.283	0.200	0.141	0.089	0.063
Bootstrap Test
2 × 4	0.382	0.271	0.171	0.121	0.085	0.054	0.038
3 × 3	0.400	0.284	0.179	0.126	0.089	0.056	0.040
3 × 4	0.431	0.302	0.190	0.134	0.095	0.060	0.042
4 × 4	0.464	0.322	0.203	0.143	0.101	0.064	0.045
4 × 5	0.488	0.339	0.212	0.149	0.106	0.067	0.047
5 × 5	0.519	0.356	0.223	0.158	0.111	0.071	0.050

shows the minimum $\epsilon$ for the distance $d_r$. The minimum $\epsilon$ for $d_a$ can be found in

Table A1. Minimum tolerance parameter $\epsilon$, for which the test power equals $0.9$ at nominal level $\alpha =0.05$, is calculated at the uniform probability matrices using the distance $d_a$. The sample size is $100,\dots ,10,000$.

Asymptotic Test
table size	100	200	500	1000	2000	5000	10,000
2 × 4	0.377	0.267	0.169	0.120	0.085	0.054	0.038
3 × 3	0.405	0.290	0.183	0.130	0.092	0.058	0.041
3 × 4	0.458	0.327	0.207	0.146	0.104	0.065	0.046
4 × 4	0.520	0.368	0.234	0.165	0.117	0.074	0.052
4 × 5	0.567	0.403	0.255	0.180	0.128	0.081	0.057
5 × 5	0.623	0.444	0.281	0.199	0.141	0.089	0.063
Bootstrap Test
2 × 4	0.378	0.268	0.170	0.120	0.085	0.054	0.038
3 × 3	0.396	0.282	0.178	0.125	0.089	0.056	0.040
3 × 4	0.423	0.302	0.190	0.134	0.095	0.060	0.042
4 × 4	0.457	0.320	0.203	0.143	0.101	0.064	0.045
4 × 5	0.478	0.337	0.212	0.149	0.106	0.067	0.047
5 × 5	0.501	0.354	0.222	0.158	0.111	0.070	0.050

because the results are very similar for $d_a$ and $d_r$. The minimum $\epsilon$ decreases with the increasing sample size at the rate $n^{-\frac{1}{2}}$. The minimum parameter $\epsilon$ climbs with the increasing table size at the rate $k_1+k_2$. Thus, the test power falls slowly with the increasing table size. The bootstrap tests have a smaller minimum $\epsilon$ than the asymptotic tests and the difference increases considerably with the table size.

We study the type I error rates at 100 randomly selected points from $\partial H_0$ because the boundary of $H_0$ is a very complex set and it is difficult to identify particularly interesting boundary points. The points are found using steps 4 and 5 of the algorithm at the end of Section 3. The sample size n equals $100\ast \left(k_1+k_2\right)$ to maintain similar test power for different table sizes because test power falls with increasing table size. The simulation results are summarized in

Table 2. Summary of the simulated exact rejection probability of the equivalence tests at nominal level $\alpha =0.05$ and tolerance parameter $\epsilon =0.2$. The rejection probability is simulated at 100 randomly selected boundary points. The sample size is $\left(k_1+k_2\right)\ast 100$ and the number of replications is 10.000 for each experiment.

	Asymptotic Test Based on ${\mathit{d}}_{\mathit{r}}$						Asymptotic Test Based on ${\mathit{d}}_{\mathit{a}}$
	2 × 4	3 × 3	3 × 4	4 × 4	4 × 5	5 × 5	2 × 4	3 × 3	3 × 4	4 × 4	4 × 5	5 × 5
Minimum	0.02	0.01	0.01	0.00	0.00	0.00	0.04	0.03	0.02	0.01	0.00	0.00
Maximum	0.12	0.10	0.06	0.05	0.03	0.01	0.08	0.08	0.06	0.03	0.02	0.01
Average	0.05	0.04	0.02	0.01	0.01	0.00	0.06	0.05	0.03	0.02	0.01	0.00
Deviation	0.02	0.01	0.01	0.01	0.00	0.00	0.01	0.01	0.01	0.00	0.00	0.00
	Bootstrap Test Based on${\mathit{d}}_{\mathit{r}}$						Bootstrap Test Based on${\mathit{d}}_{\mathit{a}}$
Minimum	0.00	0.00	0.01	0.03	0.03	0.04	0.04	0.03	0.04	0.04	0.04	0.04
Maximum	0.07	0.08	0.10	0.13	0.10	0.11	0.09	0.08	0.07	0.07	0.07	0.06
Average	0.04	0.05	0.05	0.05	0.05	0.06	0.05	0.05	0.05	0.05	0.05	0.05
Deviation	0.01	0.01	0.01	0.01	0.01	0.01	0.01	0.01	0.01	0.00	0.00	0.00

. The power of all tests varies considerably from point to point. The averaged power of the asymptotic tests decreases quickly with the table size. The asymptotic tests are not conservative for the small tables and become very conservative for the larger tables. The averaged power of the bootstrap tests is very close to the nominal level for all table sizes. However, the bootstrap tests are not conservative for all table sizes. Particularly, the $d_r$ based bootstrap test shows strong anti-conservative tendency.

A detailed analysis of the boundary points shows that the test power is far above the nominal level at the points, where $r_i{c}_j$ is close to zero for some i and j. Therefore, the test results should be treated with caution, if the marginal probability is close to zero for at least one category.

The conservative tests can be obtained by shrinking the tolerance parameter $\epsilon$.

Table A5. Summary of the simulated exact rejection probability of the equivalence tests at nominal level $\alpha =0.05$ and shrunk tolerance parameter $\epsilon =0.18$. The rejection probability is simulated at 100 randomly selected boundary points of $H_0=\left\{d_{*}\left(p,\mathcal{M}\right)\ge 0.2\right\}$. The sample size is $\left(k_1+k_2\right)\ast 100$ and the number of replications is 10.000 for each experiment.

	Asymptotic Test Based on ${\mathit{d}}_{\mathit{r}}$						Asymptotic Test Based on ${\mathit{d}}_{\mathit{a}}$
	2 × 4	3 × 3	3 × 4	4 × 4	4 × 5	5 × 5	2 × 4	3 × 3	3 × 4	4 × 4	4 × 5	5 × 5
Minimum	0.01	0.00	0.00	0.00	0.00	0.00	0.01	0.01	0.00	0.00	0.00	0.00
Maximum	0.08	0.06	0.03	0.02	0.01	0.00	0.04	0.04	0.02	0.01	0.00	0.00
Average	0.02	0.02	0.01	0.00	0.00	0.00	0.02	0.02	0.01	0.00	0.00	0.00
Deviation	0.01	0.01	0.00	0.00	0.00	0.00	0.01	0.01	0.00	0.00	0.00	0.00
	Bootstrap Test Based on${\mathit{d}}_{\mathit{r}}$						Bootstrap Test Based on${\mathit{d}}_{\mathit{a}}$
Minimum	0.00	0.00	0.00	0.01	0.01	0.01	0.01	0.01	0.01	0.01	0.01	0.01
Maximum	0.05	0.06	0.09	0.12	0.06	0.07	0.04	0.04	0.03	0.03	0.02	0.02
Average	0.02	0.02	0.02	0.02	0.02	0.02	0.02	0.02	0.02	0.02	0.02	0.01
Deviation	0.01	0.01	0.01	0.01	0.01	0.01	0.01	0.01	0.00	0.00	0.00	0.00

summarizes the simulation results for $\epsilon =0.18$, where the test power is calculated at the same points as in Table 2, i.e., $d_{*}\left(p,\mathcal{M}\right)=0.2$ at all considered points. Then the $d_a$ based tests are conservative at all points and the $d_r$ based tests are still non conservative at some points.

The type II error rates are studied at 100 randomly selected product measures for each table size, see

Table 3. Summary of the simulated test power at nominal level $\alpha =0.05$ and tolerance parameter $\epsilon =0.2$. The rejection probability is simulated at 100 randomly selected product measures. The sample size is $\left(k_1+k_2\right)\ast 100$ and the number of replications is 10.000 for each experiment.

	Asymptotic Test Based on ${\mathit{d}}_{\mathit{r}}$						Asymptotic Test Based on ${\mathit{d}}_{\mathit{a}}$
	2 × 4	3 × 3	3 × 4	4 × 4	4 × 5	5 × 5	2 × 4	3 × 3	3 × 4	4 × 4	4 × 5	5 × 5
Minimum	0.20	0.09	0.13	0.10	0.16	0.26	0.99	0.98	0.96	0.94	0.93	0.90
Maximum	0.99	0.98	0.97	0.95	0.92	0.87	1.00	1.00	0.99	0.97	0.96	0.92
Average	0.84	0.80	0.80	0.77	0.70	0.64	0.99	0.99	0.98	0.96	0.95	0.91
Deviation	0.17	0.21	0.18	0.16	0.18	0.15	0.00	0.00	0.00	0.00	0.01	0.00
	Bootstrap Test Based on${\mathit{d}}_{\mathit{r}}$						Bootstrap Test Based on${\mathit{d}}_{\mathit{a}}$
Minimum	0.13	0.13	0.22	0.28	0.56	0.73	0.98	0.97	0.97	0.98	0.98	0.99
Maximum	0.99	0.99	0.99	0.99	0.99	0.99	1.00	1.00	0.99	0.99	1.00	1.00
Average	0.82	0.80	0.86	0.89	0.90	0.94	0.99	0.99	0.99	0.99	0.99	0.99
Deviation	0.20	0.22	0.17	0.11	0.10	0.06	0.00	0.01	0.00	0.00	0.00	0.00

. It should be noted that Table 3 contains test power and the type II error rate equals 1 minus test power. The sample size equals $100\ast \left(k_1+k_2\right)$ to be comparable to the type I error analysis. The power of $d_r$-based tests changes very strongly from point to point. Given the fixed table size, the power of the $d_a$-based tests is almost constant at all considered points. The averaged power of the asymptotic tests decreases slightly with the increasing table size. The averaged power of the bootstrap tests does not change with the table size.

5. Real Data Sets

To demonstrate the application of the proposed tests, three examples with real data sets are considered: gender and nitrendipine therapy (Nitrendipine); eye color and hair color (Color); children number and income (Children). The corresponding two way contingency tables are given in Appendix A,

Table A2. Contingency table relating gender and treatment outcome on nitrendipine mono-therapy in patients suffering from mild arterial hypertension.

Gender	Outcome Category
	1	2	3	4
female	9	13	13	48
male	24	18	20	72

,

Table A3. Cross-classification of eye color and hair color.

Eye Color	Hair Color
	Black	Brunette	Red	Blonde
Brown	68	119	26	7
Blue	20	84	17	94
Hazel	15	54	14	10
Green	5	29	14	16

and

Table A4. Cross-classification of number of children by annual income.

No. of Children	Annual Income
	0–1	1–2	2–3	3+
0	2161	3577	2184	1636
1	2755	5081	2222	1052
2	936	1753	640	306
3	225	419	96	38
4+	39	98	31	14

.

Table 4. Minimum tolerance parameter $\epsilon$, for which $H_0$ can be rejected at the nominal level $\alpha =0.05$.

Data Set	n	Tests Based on ${\mathit{d}}_{\mathit{r}}$		Tests Based on ${\mathit{d}}_{\mathit{a}}$
		Asymptotic	Bootstrap	Asymptotic	Bootstrap
Nitrendipine	217	0.23	0.22	0.15	0.18
Color	592	0.59	0.58	0.59	0.59
Children	25263	0.28	0.31	0.18	0.18

displays the minimum tolerance parameter $\epsilon$, for which $H_0$ can be rejected at the nominal level $\alpha =0.05$. The three examples are also used in [5], such that a direct comparison is possible. The results for distance $d_a$ are similar to those presented in [5] after appropriate re-scaling. However, we avoid the unproven assumptions and the extensive use of the numeric optimization, which are necessary in [5].

The first example concerns with the question if the treatment outcome on nitrendipine mono-therapy in patients suffering from mild arterial hypertension depends on gender. The data set is also an example for approximate independence in [4]. The asymptotic and bootstrap test results for $d_r$ are very close to each other. The results for $d_a$ differ considerably for the asymptotic and bootstrap test. Given the small sample size, the treatment outcome and gender can be considered approximately independent.

A common example for independence testing is the cross-classification of eye color and hair color, see [2,3]. The test results in Table 4 reflect the well known fact that eye color and hair color are not independently distributed. All tests behave very similarly and can reject $H_0$ only for very large values of $\epsilon$.

The cross-classification of the number of children by the annual income has a large sample size. However, the category, where the number of children is larger than or equal 4, is sparsely populated. Therefore, the $d_r$ based tests can reject $H_0$ only for comparatively large values of $\epsilon$ and the test power is low. The $d_a$ based tests show that the number of children and annual income may be considered approximately independent, but the approximation is very inaccurate.