A Class of Association Measures for Categorical Variables Based on Weighted Minkowski Distance

Abstract

Measuring and testing association between categorical variables is one of the long-standing problems in multivariate statistics. In this paper, I define a broad class of association measures for categorical variables based on weighted Minkowski distance. The proposed framework subsumes some important measures including Cramér’s V, distance covariance, total variation distance and a slightly modified mean variance index. In addition, I establish the strong consistency of the defined measures for testing independence in two-way contingency tables, and derive the scaled forms of unweighted measures.

1. Introduction

Measuring and testing the association between categorical variables from observed data is one of the long-standing problems in multivariate statistics. The observed frequencies of two categorical variables are often displayed in a two-way contingency table, and a multinomial distribution can be used to model the cell counts. To be specific, let X and Y be two categorical random variables with finite sampling spaces $\mathcal{X}$ and $\mathcal{Y}$ ($\left|\mathcal{X}\right|<\infty ,\left|\mathcal{Y}\right|<\infty$, where $|·|$ stands for the cardinality of a set), and a simple random sample of size N can be summarized in a $\left|\mathcal{X}\right|\times \left|\mathcal{Y}\right|$ table with count $N_{xy}$ in cell $(x,y)$. Let $f(x,y)$, $f\left(x\right)$, and $f\left(y\right)$ be the joint and marginal probabilities of X and Y, i.e., $f(x,y)=P(X=x,Y=y),f\left(x\right)=P(X=x),f\left(y\right)=P(Y=y)$, then the statistical independence between X and Y can be defined as $f(x,y)=f\left(x\right)f\left(y\right)$ for any $(x,y)\in \mathcal{X}\times \mathcal{Y}$, i.e., all joint probabilities equal the product of their marginal probabilities. Pearson’s chi-squared statistic,

$$X^2=\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}\frac{{(f_N(x,y)-f_N\left(x\right)f_N\left(y\right))}^2}{f_N\left(x\right)f_N\left(y\right)/N},$$

where $f_N(x,y)=N_{xy}/N$, $f_N\left(x\right)={\sum }_{y\in \mathcal{Y}}{N}_{xy}/N$, and $f_N\left(y\right)={\sum }_{x\in \mathcal{X}}{N}_{xy}/N$, has been widely used to test independence in two-way contingency tables. Under independence and sufficient sample size, $X^2$ approximately follows a chi-squared distribution with $df=\left(\right|\mathcal{X}|-1)\left(\right|\mathcal{Y}|-1)$. However, for insufficient sample size (e.g., ${min}_{x,y}{N}_{x+}{N}_{+y}/N<5$, where $N_{x+}={\sum }_{y\in \mathcal{Y}}{N}_{xy},N_{+y}={\sum }_{x\in \mathcal{X}}{N}_{xy}$), the chi-squared test tends to be conservative. Zhang (2019) suggested a random permutation test based on the test statistic

$${\mathcal{D}}^2=\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}{(f_N(x,y)-f_N\left(x\right)f_N\left(y\right))}^2,$$

which is derived from the squared distance covariance, a measure of the dependence between two random vectors of any type (discrete or continuous) [1,2]. The ${\mathcal{D}}^2$ statistic is closely related to Pearson’s chi-squared statistic, both measuring the squared distance between $f(x,y)$ and $f\left(x\right)f\left(y\right)$, $(x,y)\in \mathcal{X}\times \mathcal{Y}$. In the numerical study of Zhang (2019), the distance covariance test was evaluated in terms of the statistical power and type I error rate under various settings (see Figures 1–3 in [1]). It is found that for relatively large sample sizes, the distance covariance test performs similarly well as Pearson’s chi-squared test. However, for relatively small sample sizes, the distance covariance test is substantially more powerful and it controls the type I error rate at the nominal level. For small sample, Pearson’s chi-squared test exhibits substantial conservativeness, in the sense that the type I error rate is much lower than the nominal level and it fails to reject many false hypotheses. For instance, in a simulation setting with 20 by 20 table and only 50 samples, the statistical power and type I error rate are both close to zero by Pearson’s chi-squared test, indicating an extreme conservativeness.

Although the distance covariance test has better empirical performance than Pearson’s chi-squared test, especially for small sample size, its theoretical properties have not been investigated. In addition, Zhang (2019) only studied two alternative measures, including distance covariance and projection correlation, but there are many other association measures in the literature remaining unexplored. To name a few, Goodman and Kruskal (1954) introduced two association measures for categorical variables, namely the concentration coefficient and the $\lambda$ coefficient [3]. Cui et al. (2015) developed a generic association measure based on a mean-variance index [4]. Theil (1970) proposed measuring the association between two categorical variables by the uncertainty coefficient [5]. McCane and Albert (2008) introduced the symbolic covariance, which expresses the covariance between categorical variables in terms of symbolic expressions [6]. In addition, Reshef et al. (2011) proposed a pairwise dependence measure called maximal information coefficient (MIC) based on the grid that maximizes the mutual information gained from the data [7].

The purpose of this paper is to extend my previous work [1] to a broad class of association measures using a general weighted Minkowski distance, and numerically evaluate some selected measures from the proposed class. The proposed class unifies many existing measures including $\varphi$ coefficient, Cramér’s V, distance covariance, total variation distance and a slightly modified mean variance index. Furthermore, the strong consistency of the independence tests based on these measures was established, and the scaled forms of unweighted measures were derived. The proposed class provides a rich set of alternatives to the prevailing chi-squared statistic, and it has many potential applications. For instance, it can be applied to the correlation-based modeling, such as correlation-based deep learning [8]. As enlightened by a reviewer, the proposed method may also be applied to the pseudorandom number generator tests, and may improve some existing chi-squared based tests including the poker test and gap test [9].

The remainder of this paper is structured as follows: In Section 2, I introduce the defined class of association measures, and study some important special cases. The scaled forms of unweighted measures are also derived. Section 3 compares the performance of selected measures using simulated data. Section 4 discusses some extensions including the application to ordinal data and conditional independence test for three-way tables.

2. Methods

2.1. A Class of Association Measures for Categorical Variables

As the strength of association between two categorical variables can be reflected by the distance between $f(x,y)$ and $f\left(x\right)f\left(y\right)$, here I define a class of measures based on the weighted Minkowski distance

$${\mathcal{L}}_{r,\omega }(X,Y)={\left\{\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}{|f(x,y)-f\left(x\right)f\left(y\right)|}^r{\omega }^r(x,y)\right\}}^{\frac{1}{r}},$$

(1)

where $r\ge 1$, $\omega (x,y)>0$, and $\omega (x,y)$ only depends on the marginal distributions of X and Y. For $0<r<1$, the defined distance violates the triangle inequality therefore it is not a metric. However, $r=\infty$ is allowed, and I denote by ${\mathcal{L}}_{\infty ,\omega }(X,Y)$ the maximum norm. It can be proved that ${\mathcal{L}}_{1,\omega }(X,Y)\ge {\mathcal{L}}_{2,\omega }(X,Y)\ge \dots \ge {\mathcal{L}}_{\infty ,\omega }(X,Y)$ for a given weight $\omega (x,y)$. Throughout this paper, I denote by ${\mathcal{L}}_r(X,Y)$ the unweighted measures, i.e., $\omega (x,y)=1$. The defined class is quite broad and I begin with some important special cases.

Firstly, most of the chi-squared-type measures belong to the defined class. For instance, the $\varphi$ coefficient for $2\times 2$ tables, i.e., $\left|\mathcal{X}\right|=\left|\mathcal{Y}\right|=2$,

$$\varphi (X,Y)={\left\{\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}\frac{{|f(x,y)-f\left(x\right)f\left(y\right)|}^2}{f\left(x\right)f\left(y\right)}\right\}}^{\frac{1}{2}},$$

is a special case of ${\mathcal{L}}_{2,\omega }(X,Y)$, where $\omega (x,y)={\left\{f\left(x\right)f\left(y\right)\right\}}^{-1/2}$. Extensions of $\varphi (X,Y)$ to $I\times J$ tables including Cramér’s V and Tschuprow’s T [10,11],

$$\begin{array}{cc}\hfill V(X,Y)& ={\left\{\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}\frac{{|f(x,y)-f\left(x\right)f\left(y\right)|}^2}{f\left(x\right)f\left(y\right)}\right\}}^{\frac{1}{2}}{\left\{\frac{1}{min\left(\right|\mathcal{X}|-1,|\mathcal{Y}|-1)}\right\}}^{\frac{1}{2}},\hfill \\ \hfill T(X,Y)& ={\left\{\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}\frac{{|f(x,y)-f\left(x\right)f\left(y\right)|}^2}{f\left(x\right)f\left(y\right)}\right\}}^{\frac{1}{2}}{\left\{\frac{1}{\sqrt{\left(\right|\mathcal{X}|-1)\left(\right|\mathcal{Y}|-1)}}\right\}}^{\frac{1}{2}},\hfill \end{array}$$

are also special cases of ${\mathcal{L}}_{2,\omega }(X,Y)$, where $\omega (x,y)=\{f\left(x\right)f\left(y\right)min(\left|\mathcal{X}\right|-1,\left|\mathcal{Y}\right|-{1\left)\right\}}^{-1/2}$ for Cramér’s V, and $\omega (x,y)={\left\{f\left(x\right)f\left(y\right)\sqrt{\left(\right|\mathcal{X}|-1)\left(\right|\mathcal{Y}|-1)}\right\}}^{-1/2}$ for Tschuprow’s T.

Distance covariance for categorical variables also belongs to the defined class. Distance covariance is a measure of statistical dependence between two random vectors X and Y. It is a special case of Hilbert-Schmidt independence criterion (HSIC) [12]. Let $(X_1,Y_1)$, $(X_2,Y_2)$ and $(X_3,Y_3)$ be three independent copies of $(X,Y)$, the distance covariance between X and Y is defined as the square root of

$${\mathrm{dCov}}^2(X,Y)=\mathrm{cov}(\parallel X_1-X_2\parallel ,\parallel Y_1-Y_2\parallel )-2\mathrm{cov}(\parallel X_1-X_2\parallel ,\parallel Y_1-Y_3\parallel ),$$

(2)

where $\parallel ·\parallel$ represents distance between vectors, e.g., Euclidean distance. An alternative definition of distance covariance is given in Sejdinovic et al. (2013) [12], which only uses two independent copies of $(X,Y)$. A proof of the equivalency between the two definitions is provided in Appendix A.1. One property of distance covariance is that ${\mathrm{dCov}}^2(X,Y)=0$ if and only if X and Y are statistically independent, indicating its potential of measuring nonlinear dependence. Zhang (2019) studied the distance covariance for categorical variables under multinomial model. Define $\parallel X_1-X_2\parallel =0$ if $X_1=X_2$ and 1 otherwise, one can show that

$$\mathrm{dCov}(X,Y)={\left\{\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}{|f(x,y)-f\left(x\right)f\left(y\right)|}^2\right\}}^{\frac{1}{2}},$$

(3)

and it is easy to see that $\mathrm{dCov}(X,Y)={\mathcal{L}}_2(X,Y)$. A detailed proof of Equation (3) is provided in Appendix A.2.

Another special case is total variation distance, which is defined as the largest difference between two probability measures [13]. Let ${\mu }_0(·)$ and ${\mu }_{\alpha }(·)$ be the measures under independence and dependence respectively, the total variation distance between ${\mu }_0$ and ${\mu }_{\alpha }$ can be used to measure the dependence between variables X and Y

$$\delta ({\mu }_0,{\mu }_{\alpha })=\underset{S\subset \mathcal{X}\times \mathcal{Y}}{max}|{\mu }_0\left(S\right)-{\mu }_{\alpha }\left(S\right)|.$$

(4)

In the case of discrete sampling spaces, let $S^{+}=\{(x,y),s.t.,f(x,y)>f\left(x\right)f\left(y\right)\}$ and $S^{-}=\{(x,y),s.t.,f(x,y)<f\left(x\right)f\left(y\right)\}$, then we have

$$\delta ({\mu }_0,{\mu }_{\alpha })= |{\mu }_0\left(S^{+}\right)-{\mu }_{\alpha }\left(S^{+}\right)| = |{\mu }_0\left(S^{-}\right)-{\mu }_{\alpha }\left(S^{-}\right)| =\frac{1}{2}\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}|f(x,y)-f\left(x\right)f\left(y\right)|,$$

(5)

therefore $\delta ({\mu }_0,{\mu }_{\alpha })={\mathcal{L}}_{1,\omega }(X,Y)$, where $\omega (X,Y)=\frac{1}{2}$.

In addition, I pointed out that the mean variance index (MV) recently developed by Cui et al. [4] also belongs to our defined class, subject to some slight modifications. The MV between two variables X and Y is defined as $MV\left(X\right|Y)=E_X\left(V_Y\left(F\left(X\right|Y)\right)\right)$, where $F\left(x\right|y)$ stands for conditional distribution function. It can be proved that $MV\left(X\right|Y)=0$ if and only if X and Y are independent. The MV measure is originally developed for continuous variables. To make it suitable for categorical variables while maintaining the main theoretical property, I slightly modified the definition of MV. First, I replaced the conditional c.d.f. $F\left(x\right|y)$ with conditional p.m.f. $f\left(x\right|y)$. Second, as the MV measure is generally asymmetric, i.e., $MV\left(X\right|Y)\ne MV(Y\left|X\right)$, I considered a symmetric version of the index, $MV(X,Y)=\frac{1}{2}(MV\left(X\right|Y)+MV\left(Y\right|X))$. With the two modifications, one can prove the following result (a detailed proof is provided in Appendix A.3)

$$\sqrt{\mathrm{MV}(X,Y)}={\left\{\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}\frac{1}{2}{|f(x,y)-f\left(x\right)f\left(y\right)|}^2\left(\frac{f\left(x\right)}{f\left(y\right)}+\frac{f\left(y\right)}{f\left(x\right)}\right)\right\}}^{\frac{1}{2}},$$

therefore $\sqrt{\mathrm{MV}(X,Y)}={\mathcal{L}}_{2,\omega }(X,Y)$, where $\omega (x,y)=\sqrt{\frac{1}{2}\left(\frac{f\left(x\right)}{f\left(y\right)}+\frac{f\left(y\right)}{f\left(x\right)}\right)}$. As $\frac{1}{2}\left(\frac{f\left(x\right)}{f\left(y\right)}+\frac{f\left(y\right)}{f\left(x\right)}\right)\ge 1$, we also have $\sqrt{\mathrm{MV}(X,Y)}\ge {\mathcal{L}}_2(X,Y)$.

Similar as the MV index, the symmetric version of some other directional association measures (e.g., the concentration coefficient [3]), are also the special cases of ${\mathcal{L}}_{r,\omega }$.

2.2. Sample Estimate and Independence Test

Given a simple random sample of size N, one can estimate ${\mathcal{L}}_{r,\omega ,N}(X,Y)$ using sample quantities

$${\mathcal{L}}_{r,\omega ,N}(X,Y)={\left\{\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}{|f_N(x,y)-f_N\left(x\right)f_N\left(y\right)|}^r{\omega }_N^r(x,y)\right\}}^{\frac{1}{r}},$$

(6)

where $f_N(x,y)$, $f_N\left(x\right)$ and $f_N\left(y\right)$ represent the maximum likelihood estimates of joint and marginal probabilities, respectively, i.e., $f_N(x,y)=N_{xy}/N$, $f_N\left(x\right)={\sum }_{y\in \mathcal{Y}}{N}_{xy}/N$, and $f_N\left(y\right)={\sum }_{x\in \mathcal{X}}{N}_{xy}/N$. The following theorem establishes the strong consistency of the independence test based on ${\mathcal{L}}_{r,\omega ,N}(X,Y)$ (a detailed proof is provided in Appendix A.4):

Theorem 1.

Assume that the estimated weights are bounded above by a constant $C>0$, i.e., ${sup}_{x,y}{\omega }_N(x,y)=C$, then for any $r\ge 1$ and $ϵ>0$, we have $P\left({\mathcal{L}}_{r,\omega ,N}(X,Y)>ϵ\right)<(2^{\left|\mathcal{X}\right|\left|\mathcal{Y}\right|}+2^{\left|\mathcal{Y}\right|}+2^{\left|\mathcal{X}\right|})exp(-N{ϵ}^2/18C^2)$ under independence. The inequality also holds for maximum norm ${\mathcal{L}}_{\infty ,\omega ,N}(X,Y)$.

It is noteworthy that the asymptotic null distribution of ${\mathcal{L}}_{r,\omega ,N}(X,Y)$ is impratical to derive. The theorem above provides a simple way to compute the upper bound of p-value, however, the bound $(2^{\left|\mathcal{X}\right|\left|\mathcal{Y}\right|}+2^{\left|\mathcal{Y}\right|}+2^{\left|\mathcal{X}\right|})exp(-N{ϵ}^2/18C^2)$ is generally not tight, thus the p-value could be largely overestimated. Here, I suggest a simple permutation procedure to evaluate the significance. One can randomly shuffle the observations of X (or equivalently, the observations of Y) for M times, and compute the test statistic ${\mathcal{L}}_{r,\omega ,N}(X_{perm},Y)$ for each permuted dataset. The permutation p-value can be computed as the proportion of ${\mathcal{L}}_{r,\omega ,N}(X_{perm},Y)$’s that exceed the actually observed one. I used the permutation p-value to evaluate statistical significant in our simulation studies.

2.3. Scaled Forms of Unweighted Measures

Motivated by the classic correlation coefficient, I define the following scaled form for unweighted measure ${\mathcal{L}}_r(X,Y)$:

$${\mathcal{L}}_r^{*}(X,Y)=\frac{{\mathcal{L}}_r(X,Y)}{\sqrt{{\mathcal{L}}_r(X,X)}\sqrt{{\mathcal{L}}_r(Y,Y)}},$$

(7)

where ${\mathcal{L}}_r(X,X)={\left\{{\sum }_{x\in \mathcal{X}}{\sum }_{x^{\prime }\in \mathcal{X}}{|f(x,x^{\prime })-f\left(x\right)f\left(x^{\prime }\right)|}^r\right\}}^{\frac{1}{r}}$, $f(x,x^{\prime })=f\left(x\right)$ if $x=x^{\prime }$ and $f(x,x^{\prime })=0$ otherwise.

The term ${\mathcal{L}}_r(X,X)$ can be written as

$${\mathcal{L}}_r(X,X)={\left\{\sum _{x\in \mathcal{X}}|f\left(x\right)-f^2{\left(x\right)|}^r+\sum _{x\in \mathcal{X}}\sum _{x^{\prime }\in \mathcal{X}\setminus x}{\left|f\left(x\right)f\left(x^{\prime }\right)\right|}^r\right\}}^{\frac{1}{r}},$$

and as examples, the explicit expressions for ${\mathcal{L}}_1(X,X)$, ${\mathcal{L}}_2(X,X)$, and ${\mathcal{L}}_{\infty }(X,X)$ are given below

• ${\mathcal{L}}_1(X,X)={\sum }_{x\in \mathcal{X}}\left[f\left(x\right)-f^2\left(x\right)\right]+{\sum }_{x\in \mathcal{X}}{\sum }_{x^{\prime }\in \mathcal{X}\setminus x}\left[f\left(x\right)f\left(x^{\prime }\right)\right]=2\left[1-{\sum }_{x\in \mathcal{X}}{f}^2\left(x\right)\right]$
• ${\mathcal{L}}_2(X,X)={\left\{{\sum }_{x\in \mathcal{X}}{f}^2\left(x\right)\left[{\sum }_{x\in \mathcal{X}}{f}^2\left(x\right)+1\right]-2{\sum }_{x\in \mathcal{X}}{f}^3\left(x\right)\right\}}^{\frac{1}{2}}$
• ${\mathcal{L}}_{\infty }(X,X)={max}_{x\in \mathcal{X}}\left[f\left(x\right)-f^2\left(x\right)\right]\vee {max}_{x\in \mathcal{X},x^{\prime }\in \mathcal{X}\setminus x}\left[f\left(x\right)f\left(x^{\prime }\right)\right]={max}_{x\in \mathcal{X}}\left[f\left(x\right)-f^2\left(x\right)\right]$

It can be seen that ${\mathcal{L}}_2^{*}(X,Y)$ is same as the distance correlation between X and Y [1], therefore $0\le {\mathcal{L}}_2^{*}(X,Y)\le 1$, where ${\mathcal{L}}_r^{*}(X,Y)=0$ if and only if X and Y are independent. In fact, for any $1\le r<\infty$, if $f\left(x\right)>0,f\left(y\right)>0$ for $x\in \mathcal{X},y\in \mathcal{Y}$, it can be proved that $0\le {\mathcal{L}}_r^{*}(X,Y)\le 1$, where ${\mathcal{L}}_r^{*}(X,Y)=0$ if and only if X and Y are independent, and ${\mathcal{L}}_r^{*}(X,Y)=1$ if and only if X and Y have perfect association, i.e., $\left|\mathcal{X}\right|=\left|\mathcal{Y}\right|$ and for any $x\in \mathcal{X}$, there exists a unique $y\in \mathcal{Y}$, such that $f(x,y)=f\left(x\right)=f\left(y\right)$.

For ${\mathcal{L}}_{\infty }^{*}(X,Y)$, by Cauchy-Schwarz inequality,

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\infty }(X,Y)& =\underset{x\in \mathcal{X},y\in \mathcal{Y}}{max}|f(x,y)-f\left(x\right)f\left(y\right)|\hfill \\ \hfill & =\underset{x\in \mathcal{X},y\in \mathcal{Y}}{max}\left|cov(I\{X=x\},I\{Y=y\})\right|\hfill \\ \hfill & \le \underset{x\in \mathcal{X},y\in \mathcal{Y}}{max}\sqrt{V\left(I\right\{X=x\left\}\right)V\left(I\right\{Y=y\left\}\right)}\hfill \\ \hfill & =\underset{x\in \mathcal{X}}{max}\sqrt{V\left(I\right\{X=x\left\}\right)}\underset{y\in \mathcal{Y}}{max}\sqrt{V\left(I\right\{Y=y\left\}\right)}\hfill \\ \hfill & =\sqrt{\underset{x\in \mathcal{X}}{max}\left[f\left(x\right)-f^2\left(x\right)\right]}\sqrt{\underset{y\in \mathcal{Y}}{max}\left[f\left(y\right)-f^2\left(y\right)\right]}\hfill \\ \hfill & =\sqrt{{\mathcal{L}}_{\infty }(X,X){\mathcal{L}}_{\infty }(Y,Y)},\hfill \end{array}$$

therefore $0\le {\mathcal{L}}_{\infty }^{*}\le 1$. However, in general, ${\mathcal{L}}_{\infty }^{*}(X,Y)=1$ does not imply that X and Y are perfectly associated. I gave an example in Table 1, where ${\mathcal{L}}_{\infty }^{*}(X,Y)=1$ but X and Y are not perfectly associated.

Table 1. An example that X and Y are not perfectly associated, but ${\mathcal{L}}_{\infty }^{*}(X,Y)=1$.

	Y = 1	Y = 2	Y = 3
X = 1	1/2	0	0
X = 2	0	1/8	1/8
X = 3	0	1/8	1/8

3. Numerical Study

Two simulation studies were conducted to compare the performance of some selected measures from our defined class. In both simulations, I set $\left|\mathcal{X}\right|=\left|\mathcal{Y}\right|=10$ and varied the sample size from 25 to 500, so that the simulated contingency tables were relatively large and sparse (average count $N/\left|\mathcal{X}\right|\left|\mathcal{Y}\right|$ is between $0.25$ and 5).

In the first simulation study, I considered the independence test based on different unweighted measures, including ${\mathcal{L}}_1$, ${\mathcal{L}}_2$, ${\mathcal{L}}_4$ and ${\mathcal{L}}_{\infty }$, under the following multinomial settings:

• Setting 1: $f(x,y)=0.05$ for 10 randomly selected cells and $f(x,y)=\frac{0.5}{90}$ for the remaining 90 cells
• Setting 2: $f(x,y)=0.08$ for 10 randomly selected cells and $f(x,y)=\frac{0.2}{90}$ for the remaining 90 cells
• Setting 3: $f(x,y)=0.1$ for one randomly selected cell and $f(x,y)=\frac{0.9}{99}$ for the remaining 99 cells
• Setting 4: $f(x,y)=0.2$ for one randomly selected cell and $f(x,y)=\frac{0.8}{99}$ for the remaining 99 cells

For each test, the p-values were computed based on 2000 random permutations. Figure 1 summarizes the empirical statistical power of the four tests under significance level 0.05. It could be seen that, in settings 1 and 2, the ${\mathcal{L}}_2$ measure (Euclidean distance) performed consistently better than the other three (comparable to ${\mathcal{L}}_4$). The maximum norm ${\mathcal{L}}_{\infty }$ performs the worst in these two settings. In settings 3 and 4, where a single cell accounts for most deviation from independence, the maximum norm performs the best, while the ${\mathcal{L}}_1$ measure (Manhattan distance) gives the lowest power. Figure 2 summarizes the type I error rate, where it can be seen that all the four tests control the type I error rates at the nominal level of 0.05.

Figure 1. Empirical statistical power of four different measures including ${\mathcal{L}}_1$ (blue), ${\mathcal{L}}_2$ (red), ${\mathcal{L}}_4$ (black) and ${\mathcal{L}}_{\infty }$ (green), under settings 1–4. In each setting, sample sizes are $n=25,50,75,100,200,500$, and all results were based on 1000 replications.

Figure 2. Empirical Type I error rate of four different measures including ${\mathcal{L}}_1$ (blue), ${\mathcal{L}}_2$ (red), ${\mathcal{L}}_4$ (black) and ${\mathcal{L}}_{\infty }$ (green), under settings 1–4. In each setting, sample sizes are $n=25,50,75,100,200,500$, and all results were based on 1000 replications.

In the second simulation study, I focused on ${\mathcal{L}}_{2,\omega }(X,Y)$ as it subsumes many popular measures. In particular, I compared three different weight functions, including $\omega (x,y)=1$ (distance covariance), $\omega (x,y)={\left\{f\left(x\right)f\left(y\right)\right\}}^{-1/2}$ (Pearson’s chi-squared), and $\omega (x,y)=\sqrt{\frac{1}{2}(\frac{f\left(x\right)}{f\left(y\right)}+\frac{f\left(y\right)}{f\left(x\right)})}$ (modified mean variance index). Figure 3 shows the empirical statistical power of the three measures under settings 1 and 2, where it can be seen that the unweighted ${\mathcal{L}}_2$ compares favorably to the weighted ones.

Figure 3. Empirical statistical power of three ${\mathcal{L}}_{2,\omega }$ measures including $\omega (x,y)=1$ (distance covariance), $\omega (x,y)={\left\{f\left(x\right)f\left(y\right)\right\}}^{-1/2}$ (chi-squared), and $\omega (x,y)=\sqrt{\frac{1}{2}(\frac{f\left(x\right)}{f\left(y\right)}+\frac{f\left(y\right)}{f\left(x\right)})}$ (symmetric mean variance index), under settings 1 and 2. In each setting, sample sizes are $n=25,50,75,100,200,500$, and all results were based on 1000 replications.

Based on the simulation studies, I recommend to the unweighted ${\mathcal{L}}_r$ measures with a moderate choice of r, for instance, $r=2,3,4$ for large sparse tables, because they could give satisfactory and stable statistical power in general scenarios. The maximum norm ${\mathcal{L}}_{\infty }$ is not recommended, unless one is very confident that there exist a very small number of cells that account for most deviation from independence.

4. Discussion

In this work, I proposed a rich class of dependence measures for categorical variables based on weighted Minkowski distance. The defined class unifies a number of existing measures including Cramér’s V, distance covariance, total variation distance and a slightly modified mean variance index. I provided the scaled forms of unweighted measures, which range from 0 (independence) to 1 (perfect association). Further, I established the strong consistency of the defined measures and suggested a simple permutation test for evaluating significance. Although I have used nominal and univariate categorical variables for illustrations, the proposed framework can be extended to other data types and problems:

First, the proposed measures can be used to detect ordinal association by assigning proper weights. Similar as Pearson’s correlation coefficient, one may assign larger weights to more extreme categories of X and Y. To be specific, let $d(x,x^{\prime })$ be the predefined distance between categories $X=x$ and $X=x^{\prime }$, and $d(y,y^{\prime })$ be the distance between y and $y^{\prime }$, and one could apply the following weight function

$$\omega (x,y)=E\left(d(x,X)d(y,Y)\right)=\sum _{x^{\prime }\in \mathcal{X}\setminus x,y^{\prime }\in \mathcal{Y}\setminus y}d(x,x^{\prime })d(y,y^{\prime })f\left(x^{\prime }\right)f\left(y^{\prime }\right),$$

which assigns larger weights to cells in the corners but smaller weights to cells in the center of the table.

Second, my framework can be generalized to random vectors and multi-way tables. In the case of three-way table $(X,Y,Z)$, one can define the following Minkowski distance between $f(x,y,z)$ and $f(x,y)f\left(z\right)$

$${\mathcal{L}}_{r,\omega }((X,Y),Z)={\left\{\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}\sum _{z\in \mathcal{Z}}{|f(x,y,z)-f(x,y)f\left(z\right)|}^r{\omega }^r(x,y,z)\right\}}^{\frac{1}{r}},$$

which can be used to test the joint independence between $(X,Y)$ and Z, or equivalently, to test the homogeneity of the joint distribution of $(X,Y)$ at different levels of Z. A similar permutation procedure can be applied to evaluate the statistical significance. One can also define the distance between $f(x,y,z)$ and $f\left(x\right)f\left(y\right)f\left(z\right)$ to test the mutual independence of $(X,Y,Z)$

$${\mathcal{L}}_{r,\omega }(X,Y,Z)={\left\{\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}\sum _{z\in \mathcal{Z}}{|f(x,y,z)-f\left(x\right)f\left(y\right)f\left(z\right)|}^r{\omega }^r(x,y,z)\right\}}^{\frac{1}{r}},$$

Furthermore, the framework can be extended to conditional independence test in three-way tables [14], by defining distance between conditional joint probabilities $f(x,y|z)$ and the product of conditional marginal probabilities $f\left(x\right|z\left)f\right(y\left|z\right)$

$${\mathcal{L}}_{r,\omega }(X,Y|Z)={\left\{\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}\sum _{z\in \mathcal{Z}}{|f(x,y|z)-f\left(x\right|z)f\left(y\right|z)|}^r{\omega }^r(x,y,z)\right\}}^{\frac{1}{r}}.$$

Appendix A.1. Equivalency between Two Definitions of Distance Covariance

• Definition by Szekely et al. (2007): ${\mathrm{dCov}}^2(X,Y)=E_{X_1{X}_2{Y}_1{Y}_2}(\parallel X_1-X_2\parallel \parallel Y_1-Y_2\parallel )+E_{X_1{X}_2}(\parallel X_1-X_2\parallel )E_{Y_1{Y}_2}(\parallel Y_1-Y_2\parallel )-2E_{X_1{X}_2{Y}_1{Y}_3}(\parallel X_1-X_2\parallel \parallel Y_1-Y_3\parallel )$
• Definition by Sejdinovic et al. (2013): ${\mathrm{dCov}}^2(X,Y)=E_{X_1{X}_2{Y}_1{Y}_2}(\parallel X_1-X_2\parallel \parallel Y_1-Y_2\parallel )+E_{X_1{X}_2}(\parallel X_1-X_2\parallel )E_{Y_1{Y}_2}(\parallel Y_1-Y_2\parallel )-2E_{X_1{Y}_1}(E_{X_2}(\parallel X_1-X_2\parallel )E_{Y_2}(\parallel Y_1-Y_2\parallel \left)\right)$

The first two terms are the same, and the equivalency between the two definitions can be showed as follows:

$$\begin{array}{cc}\hfill & E_{X_1{Y}_1}(E_{X_2}(\parallel X_1-X_2\parallel )E_{Y_2}(\parallel Y_1-Y_2\parallel \left)\right)\hfill \\ \hfill & ={\int }_{x_1}{\int }_{y_1}\left[{\int }_{x_2}\parallel x_1-x_2\parallel f\left(x_2\right)d{x}_2{\int }_{y_2}\parallel y_1-y_2\parallel f\left(y_2\right)d{y}_2\right]f(x_1,y_1)d{x}_{1}d{y}_1\hfill \\ \hfill & ={\int }_{x_1}{\int }_{y_1}\left[{\int }_{x_2}\parallel x_1-x_2\parallel f\left(x_2\right)d{x}_2{\int }_{y_3}\parallel y_1-y_3\parallel f\left(y_3\right)d{y}_3\right]f(x_1,y_1)d{x}_{1}d{y}_1\hfill \\ \hfill & ={\int }_{x_1}{\int }_{x_2}{\int }_{y_1}{\int }_{y_3}\parallel x_1-x_2\parallel \parallel y_1-y_3\parallel f(x_1,y_1)f\left(x_2\right)f\left(y_3\right)d{x}_{1}d{x}_{2}d{y}_{1}d{y}_3\hfill \\ \hfill & =E_{X_1{X}_2{Y}_1{Y}_3}(\parallel X_1-X_2\parallel \parallel Y_1-Y_3\parallel )\hfill \end{array}$$

Appendix A.2. Derivation of Equation (3)

Following Zhang (2019), I rewrite categorical variables X and Y as two random vectors of dimensions $\left|\mathcal{X}\right|$ and $\left|\mathcal{Y}\right|$, $\mathit{X}={\left\{I(X=x)\right\}}_{x\in \mathcal{X}}$ and $\mathit{Y}={\left\{I(Y=y)\right\}}_{y\in \mathcal{Y}}$, where $I(·)$ stands for the indicator function. Let $\parallel {\mathit{X}}_1-{\mathit{X}}_2\parallel$ equal 0 if ${\mathit{X}}_1={\mathit{X}}_2$ and 1 otherwise. Let $({\mathit{X}}_1,{\mathit{Y}}_2)$, $({\mathit{X}}_2,{\mathit{Y}}_2)$, $({\mathit{X}}_3,{\mathit{Y}}_3)$ be three independent copies of $(\mathit{X},\mathit{Y})$. By Equation (2), the squared distance covariance can be also expressed as

$${\mathrm{dCov}}^2(\mathit{X},\mathit{Y})=E(\parallel {\mathit{X}}_1-{\mathit{X}}_2\parallel \parallel {\mathit{Y}}_1-{\mathit{Y}}_2\parallel )+E(\parallel {\mathit{X}}_1-{\mathit{X}}_2\parallel \left)E\right(\parallel {\mathit{Y}}_1-{\mathit{Y}}_2\parallel )-2E(\parallel {\mathit{X}}_1-{\mathit{X}}_2\parallel \parallel {\mathit{Y}}_1-{\mathit{Y}}_3\parallel ).$$

Under multinomial sampling scheme, it is straightforward to show that

$$\begin{array}{cc}\hfill E(\parallel {\mathit{X}}_1-{\mathit{X}}_2\parallel \parallel {\mathit{Y}}_1-{\mathit{Y}}_2\parallel )& =P({\mathit{X}}_1\ne {\mathit{X}}_2,{\mathit{Y}}_1\ne {\mathit{Y}}_2)=\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}f(x,y)[1-f\left(x\right)-f\left(y\right)+f(x,y)],\hfill \\ \hfill E(\parallel {\mathit{X}}_1-{\mathit{X}}_2\parallel \left)E\right(\parallel {\mathit{Y}}_1-{\mathit{Y}}_2\parallel )& =P({\mathit{X}}_1\ne {\mathit{X}}_2)P({\mathit{Y}}_1\ne {\mathit{Y}}_2)=[1-\sum _{x\in \mathcal{X}}{f}^2\left(x\right)][1-\sum _{y\in \mathcal{Y}}{f}^2\left(y\right)],\hfill \\ \hfill E(\parallel {\mathit{X}}_1-{\mathit{X}}_2\parallel \parallel {\mathit{Y}}_1-{\mathit{Y}}_3\parallel )& =P({\mathit{X}}_1\ne {\mathit{X}}_2,{\mathit{Y}}_1\ne {\mathit{Y}}_3)=\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}f(x,y)[1-f\left(x\right)][1-f\left(y\right)].\hfill \end{array}$$

Summarizing the results above, I have

$$\mathrm{dCov}(X,Y)={\left\{\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}{|f(x,y)-f\left(x\right)f\left(y\right)|}^2\right\}}^{\frac{1}{2}}.$$

Appendix A.3. Derivation of the Modified Mean Variance Index

The symmetric mean variance index is defined as

$$MV(X,Y)=\frac{1}{2}(MV\left(X\right|Y)+MV\left(Y\right|X))=\frac{1}{2}(E_X\left(V_Y\left(f\left(X\right|Y)\right)\right)+E_Y\left(V_X\left(f\left(Y\right|X)\right)\right)).$$

I first derived the explicit formula for $E_X\left(V_Y\left(f\left(X\right|Y)\right)\right)$:

$$\begin{array}{cc}\hfill E_X\left(V_Y\left(f\left(X\right|Y)\right)\right)& =E_X{E}_Y{\left(f\left(X\right|Y)\right)}^2-E_X{E}_Y^2\left(f\left(X\right|Y)\right)\hfill \\ \hfill & =\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}{f}^2(x,y)\frac{f\left(x\right)}{f\left(y\right)}-\sum _{x\in \mathcal{X}}{f}^3\left(x\right)\hfill \\ \hfill & =\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}\frac{f\left(x\right)}{f\left(y\right)}(f^2(x,y)-f^2\left(x\right)f^2\left(y\right))\hfill \\ \hfill & =\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}\frac{f\left(x\right)}{f\left(y\right)}\{{(f(x,y)-f\left(x\right)f\left(y\right))}^2-2f^2\left(x\right)f^2\left(y\right)+2f(x,y)f\left(x\right)f\left(y\right)\}\hfill \\ \hfill & =\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}\frac{f\left(x\right)}{f\left(y\right)}{(f(x,y)-f\left(x\right)f\left(y\right))}^2\hfill \end{array}$$

Similarly, it can seen that $E_Y\left(V_X\left(f\left(Y\right|X)\right)\right)={\sum }_{x\in \mathcal{X}}{\sum }_{y\in \mathcal{Y}}\frac{f\left(y\right)}{f\left(x\right)}{(f(x,y)-f\left(x\right)f\left(y\right))}^2$, therefore

$$MV(X,Y)=\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}\frac{1}{2}\left(\frac{f\left(y\right)}{f\left(x\right)}+\frac{f\left(x\right)}{f\left(y\right)}\right){(f(x,y)-f\left(x\right)f\left(y\right))}^2.$$

Appendix A.4. Proof of Theorem 1

Because ${\mathcal{L}}_{1,\omega ,N}\ge {\mathcal{L}}_{2,\omega ,N}\ge \dots \ge {\mathcal{L}}_{\infty ,\omega ,N}$, I only need prove the strong consistency for ${\mathcal{L}}_{1,\omega ,N}$. For categorical variable X, let $f{\left(x\right)}_{x\in \mathcal{X}}$ be the probability mass function, N be the sample size, and $f_N\left(x\right)$ be the sample estimate, Biau and Gyorfi (2005) [15] proved the following result

Lemma A1.

For any $ϵ>0$, $P({\sum }_{x\in \mathcal{X}}|f_N\left(x\right)-f\left(x\right)|>ϵ)<2^{\left|\mathcal{X}\right|}{e}^{-\frac{N{ϵ}^2}{2}}$.

As ${sup}_{x,y}\omega (x,y)=C>0$, I have

$${\mathcal{L}}_{1,\omega ,N}(X,Y)\le C(\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}|f_N(x,y)-f(x,y)|+\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}|f(x,y)-f\left(x\right)f\left(y\right)|+\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}|f_N\left(x\right)f_N\left(y\right)-f\left(x\right)f\left(y\right)\left|\right).$$

Under independence, I have ${\sum }_{x\in \mathcal{X}}{\sum }_{y\in \mathcal{Y}}|f(x,y)-f\left(x\right)f\left(y\right)|=0$. By Lemma A1, the first term ${\sum }_{x\in \mathcal{X}}{\sum }_{y\in \mathcal{Y}}|f_N(x,y)-f(x,y)|$ satisfies that

$$P\left(C\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}|f_N(x,y)-f(x,y)|>\frac{ϵ}{3}\right)<2^{\left|\mathcal{X}\right|\left|\mathcal{Y}\right|}{e}^{-\frac{N{ϵ}^2}{18C^2}},$$

The third term can be bounded as follows:

$$\begin{array}{cc}\hfill \sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}|f_N\left(x\right)f_N\left(y\right)-f\left(x\right)f\left(y\right)|& \le \sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}|f\left(x\right)f_N\left(y\right)-f\left(x\right)f\left(y\right)|+\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}|f_N\left(x\right)f_N\left(y\right)-f\left(x\right)f_N\left(y\right)|\hfill \\ \hfill & =\sum _{y\in \mathcal{Y}}|f_N\left(y\right)-f\left(y\right)|+\sum _{x\in \mathcal{X}}|f_N\left(x\right)-f\left(x\right)|\hfill \end{array}$$

By Lemma A1, I have

$$P\left(C\sum _{y\in \mathcal{Y}}|f_N\left(y\right)-f\left(y\right)|>\frac{ϵ}{3}\right)<2^{\left|\mathcal{Y}\right|}{e}^{-\frac{N{ϵ}^2}{18C^2}},$$

$$P\left(C\sum _{x\in \mathcal{X}}|f_N\left(x\right)-f\left(x\right)|>\frac{ϵ}{3}\right)<2^{\left|\mathcal{X}\right|}{e}^{-\frac{N{ϵ}^2}{18C^2}},$$

and summarizing the results above, I have

$$\begin{array}{cc}\hfill P\left({\mathcal{L}}_{1,\omega ,N}(X,Y)>ϵ\right)& \le P\left(C\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}|f_N(x,y)-f(x,y)|>\frac{ϵ}{3}\right)\hfill \\ \hfill & +P\left(C\sum _{y\in \mathcal{Y}}|f_N\left(y\right)-f\left(y\right)|>\frac{ϵ}{3}\right)+P\left(C\sum _{x\in \mathcal{X}}|f_N\left(x\right)-f\left(x\right)|>\frac{ϵ}{3}\right)\hfill \\ \hfill & <2^{\left|\mathcal{X}\right|\left|\mathcal{Y}\right|}{e}^{-\frac{N{ϵ}^2}{18C^2}}+2^{\left|\mathcal{Y}\right|}{e}^{-\frac{N{ϵ}^2}{18C^2}}+2^{\left|\mathcal{X}\right|}{e}^{-\frac{N{ϵ}^2}{18C^2}}\hfill \\ \hfill & =(2^{\left|\mathcal{X}\right|\left|\mathcal{Y}\right|}+2^{\left|\mathcal{X}\right|}+2^{\left|\mathcal{Y}\right|})e^{-\frac{N{ϵ}^2}{18C^2}}.\hfill \end{array}$$