Geometric Estimation of Multivariate Dependency

Abstract

This paper proposes a geometric estimator of dependency between a pair of multivariate random variables. The proposed estimator of dependency is based on a randomly permuted geometric graph (the minimal spanning tree) over the two multivariate samples. This estimator converges to a quantity that we call the geometric mutual information (GMI), which is equivalent to the Henze–Penrose divergence. between the joint distribution of the multivariate samples and the product of the marginals. The GMI has many of the same properties as standard MI but can be estimated from empirical data without density estimation; making it scalable to large datasets. The proposed empirical estimator of GMI is simple to implement, involving the construction of an minimal spanning tree (MST) spanning over both the original data and a randomly permuted version of this data. We establish asymptotic convergence of the estimator and convergence rates of the bias and variance for smooth multivariate density functions belonging to a Hölder class. We demonstrate the advantages of our proposed geometric dependency estimator in a series of experiments.

1. Introduction

Estimation of multivariate dependency has many applications in fields such as information theory, clustering, structure learning, data processing, feature selection, time series prediction, and reinforcement learning, see [1,2,3,4,5,6,7,8,9,10], respectively. It is difficult to accurately estimate the mutual information in high-dimensional settings, specially where the data is multivariate with an absolutely continuous density with respect to Lebesgue measure—the setting considered in this paper. An important and regular measure of dependency is the Shannon mutual information (MI), which has seen extensive use across many application domains. However, the estimation of mutual information can often be challenging. In this paper, we focus on a measure of MI that we call the Geometric MI (GMI). This MI measure is defined as the asymptotic large sample limit of a randomized minimal spanning tree (MST) statistic spanning the multivariate sample realizations. The GMI is related to a divergence measure called the Henze–Penrose divergence [11,12], and related to the multivariate runs test [13]. In [14,15], it was shown that this divergence measure can be used to specify a tighter bound for the Bayes error rate for testing if a random sample comes from one of two distributions the bound in [14,15] is tighter than previous divergence-type bounds such as the Bhattacharrya bound [16]. Furthermore, the authors of [17] proposed a non-parametric bound on multi-class classification Bayes error rate using a global MST graph.

Let $\mathbf{X}$ and $\mathbf{Y}$ be random variables with unknown joint density $f_{XY}$ and marginal densities $f_X$ and $f_Y$, respectively, and consider two hypotheses: $H_0$, $\mathbf{X}$ and $\mathbf{Y}$ are independent and $H_1$, $\mathbf{X}$ and $\mathbf{Y}$ are dependent,

$$H_0:f_{XY}=f_X{f}_Y,\mathrm{versus}{H}_1:f_{XY}\ne f_X{f}_Y.$$

The GMI is defined as the Henze–Penrose divergence between $f_{XY}$ and $f_X{f}_Y$ which can be used as a dependency measure. In this paper, we prove that for large sample size n the randomized MST statistic spanning the original multivariate sample realizations and a randomly shuffled data set converges almost surely to the GMI measure. A direct implication of [14,15] is that the GMI provides a tighter bound on the Bayes misclassification rate for the optimal test of independence. In this paper, we propose an estimator based on a random permutation modification of the Friedman–Rafsky multivariate test statistic and show that under certain conditions the GMI estimator achieves the parametric mean square error (MSE) rate when the joint density is bounded and smooth. Importantly unlike other measures of MI, our proposed GMI estimator does not require explicit estimation of the joint and marginal densities.

Computational complexity is an important challenge in machine learning and data science. Most plug-in-based estimators, such as the kernel density estimator (KDE) or the K-nearest-neighbor (KNN) estimator with known convergence rate, require runtime complexity of $O(n^2)$, which is not suitable for large scale applications. Noshad et al. proposed a graph theoretic direct estimation method based on nearest-neighbor ratios (NNR) [18]. The NNR estimator is based on k-NN graph and computationally more tractable than other competing estimators with complexity $O(knlogn)$. The construction of the minimal spanning tree lies at the heart of the GMI estimator proposed in this paper. Since the GMI estimator is based on the Euclidean MST the dual-tree algorithm by March et al. [19] can be applied. This algorithm is based on the construction of Borůvka [20] and implements the Euclidean MST in approximately $O(nlogn)$ time. In this paper, we experimentally show that for large sample size the proposed GMI estimator has faster runtime than the KDE plug-in method.

1.1. Related Work

Estimation of mutual information has a rich history. The most common estimators of MI are based on plug-in density estimation, e.g., using the histogram, kernel density or kNN density estimators [21,22]. Motivated by ensemble methods applied to divergence estimation [23,24], in [22] an ensemble method for combining multiple KDE bandwidths was proposed for estimating MI. Under certain smoothness conditions this ensemble MI estimator was shown to achieve parametric convergence rates.

Another class of estimators of multivariate dependency bypasses the difficult density estimation task. This class includes the statistically consistent estimators of Rényi-$\alpha$ and KL mutual information which are motivated by the asymptotic limit of the length of the KNN graph, [25,26] when joint density is smooth. The estimator of [27] builds on KNN methods for Rényi entropy estimation. The authors of [26], showed that when MI is large the KNN and KDE approaches are ill-suited for estimating MI since the joint density may be insufficiently smooth when there are strong dependencies. To overcome this issue an assumption on the smoothness of the density is required, see [28,29], and [23,24]. For all these methods, the optimal parametric rate of MSE convergence is achieved when the densities are either d, $(d+1)/2$ or $d/2$ times differentiable [30]. In this paper, we assume that joint and marginal densities are smooth in the sense that they belong to Hölder continuous classes of densities ${\mathsf{\Sigma }}_d(\eta ,K)$, where the smoothness parameter $\eta \in (0,1]$ and the Lipschitz constant $K>0$.

A MI measure based on the Pearson chi-square divergence was considered in [31] that is computational efficient and numerically stable. The authors of [27,32] used nearest-neighbor graph and minimal spanning tree approaches, respectively, to estimate Rényi mutual information. In [22], a non-parametric mutual information estimator was proposed using a weighted ensemble method with $O(1/n)$ parametric convergence rate. This estimator was based on plug-in density estimation, which is challenging in high dimension.

Our proposed dependency estimator differs from previous methods in the following ways. First, it estimates a different measure of mutual information, the GMI. Second, instead of using the KNN graph the estimator of GMI uses a randomized minimal spanning tree that spans the multivariate realizations. The proposed GMI estimator is motivated by the multivariate runs test of Friedman and Rafsky (FR) [33] which is a multivariate generalization of the univariate Smirnov maximum deviation test [34] and the Wald-Wolfowitz [35] runs test in one dimension. We also emphasize that the proposed GMI estimator does not require boundary correction, in contrast to other graph-based estimators, such as, the NNR estimator [18], scalable MI estimator [36], or cross match statistic [37].

1.2. Contribution

The contribution of this paper has three components

(1)

We propose a novel non-parametric multivariate dependency measure, referred to as geometric mutual information (GMI), which is based on graph-based divergence estimation. The geometric mutual information is constructed using a minimal spanning tree and is a function of the Friedman–Rafsky multivariate test statistic.

(2)

We establish properties of the proposed dependency measure analogous to those of Shannon mutual information, such as, convexity, concavity, chain rule, and a type of data-processing inequality.

(3)

We derive a bound on the MSE rate for the proposed geometric estimator. An advantage of the estimator is that it achieves the optimal MSE rate without the need for boundary correction, which is required for most plug-in estimators.

1.3. Organization

The rest of the paper is organized as follows. In Section 2, we define the geometric mutual information and establish some of its mathematical properties. In Section 2.2 and Section 2.3, we introduce a statistically consistent GMI estimator and derive a bound on its mean square error convergence rate. In Section 3 we verify the theory through experiments.

Throughout the paper, we denote statistical expectation by $\mathbb{E}$ and the variance by abbreviation $\mathrm{Var}$. Bold face type indicates random vectors. All densities are assumed to be absolutely continuous with respect to non-atomic Lebesgue measure.

2. The Geometric Mutual Information (GMI)

In this section, we first review the definition of the Henze–Penrose (HP) divergence measure defined by Berisha and Hero in [13,14]. The Henze–Penrose divergence between densities f and g with domain ${\mathbb{R}}^d$ for parameter $p\in (0,1)$ is defined as follows (see [13,14,15]):

$${\displaystyle D_p(f,g)=\frac{1}{4pq}\left[\int \frac{{\left(pf(\mathbf{x})-qg(\mathbf{x})\right)}^2}{pf(\mathbf{x})+qg(\mathbf{x})}\mathrm{d}\mathbf{x}-{(p-q)}^2\right],}$$

(1)

where $q=1-p$. This functional is an f-divergence [38], equivalently, as an Ali-Silvey distance [39], i.e., it satisfies the properties of non-negativity, monotonicity, and joint convexity [15]. The measure (1) takes values in $[0,1]$ and $D_p(f,g)=0$ if and only if $f=g$ almost surely.

The mutual information measure is defined as follows. Let $f_X$, $f_Y$, and $f_{XY}$ be the marginal and joint distributions, respectively, of random vectors $\mathbf{X}\in {\mathbb{R}}^{d_x}$, $\mathbf{Y}\in {\mathbb{R}}^{d_y}$ where $d_x$ and $d_y$ are positive integers. Then by using (1), a Henze–Penrose generalization of the mutual information between $\mathbf{X}$ and $\mathbf{Y}$, is defined by

$$\begin{array}{cc}\hfill I_p(\mathbf{X};\mathbf{Y})& =D_p(f_{XY},f_X{f}_Y)\hfill \\ & {\displaystyle =\frac{1}{4pq}\left[\int \int \frac{{\left(p{f}_{XY}(\mathbf{x},\mathbf{y})-q{f}_X(\mathbf{y})f_Y(\mathbf{y})\right)}^2}{p{f}_{XY}(\mathbf{x},\mathbf{y})+q{f}_X(\mathbf{x})f_Y(\mathbf{y})}\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}-{(p-q)}^2\right].}\hfill \end{array}$$

(2)

We will show below that $I_p(\mathbf{X};\mathbf{Y})$ has a geometric interpretation in terms of the large sample limit of a minimal spanning tree spanning n sample realizations of the merged labeled samples $\mathbf{X}\cup \mathbf{Y}$. Thus, we call $I_p(\mathbf{X};\mathbf{Y})$ the GMI between $\mathbf{X}$ and $\mathbf{Y}$. The GMI satisfies similar properties to other definitions of mutual information, such as Shannon and Rényi mutual information. Recalling (3) in [14], an alternative form of $I_p$ is given by

$$\begin{array}{c}\hfill {\displaystyle I_p(\mathbf{X};\mathbf{Y})=1-A_p(\mathbf{X};\mathbf{Y})=\frac{u_p(\mathbf{X};\mathbf{Y})}{4pq}-\frac{{(p-q)}^2}{4pq},}\end{array}$$

(3)

where

$$\begin{array}{c}{\displaystyle A_p(\mathbf{X};\mathbf{Y})=\int \int \frac{f_{XY}(\mathbf{x},\mathbf{y})f_X(\mathbf{x})f_Y(\mathbf{y})}{p{f}_{XY}(\mathbf{x},\mathbf{y})+q{f}_X(\mathbf{x})f_Y(\mathbf{y})}\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}={\mathbb{E}}_{XY}\left[{\left(p\frac{f_{XY}(\mathbf{X},\mathbf{Y})}{f_X(\mathbf{X})f_Y(\mathbf{Y})}+q\right)}^{-1}\right],\mathrm{and}}\\ {\displaystyle u_p(\mathbf{X};\mathbf{Y})={\displaystyle \int \int \frac{{\left(p{f}_{XY}(\mathbf{x},\mathbf{y})-q{f}_X(\mathbf{x})f_Y(\mathbf{y})\right)}^2}{p{f}_{XY}(\mathbf{x},\mathbf{y})+q{f}_X(\mathbf{x})f_Y(\mathbf{y})}\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}}=1-4pq{A}_p(\mathbf{X};\mathbf{Y}).}\end{array}$$

(4)

The function $A_p(\mathbf{X};\mathbf{Y})$ was defined in [13] and is called the geometric affinity between $\mathbf{X}$ and $\mathbf{Y}$. The next subsection of the paper is dedicated to the basic inequalities and properties of the proposed GMI measure (2).

2.1. Properties of the Geometric Mutual Information

In this subsection we establish basic inequalities and properties of the GMI, $I_p$, given in (2). The following theorem shows that $I_p(\mathbf{X};\mathbf{Y})$ is a concave function in $f_X$ and a convex function in $f_{Y|X}$. The proof is given in Appendix A.1.

Theorem 1.

Denote by ${\tilde{I}}_p(f_{XY})$ the GMI $I_p(\mathbf{X};\mathbf{Y})$ when $\mathbf{X}\in {\mathbb{R}}^{d_x}$ and $\mathbf{Y}\in {\mathbb{R}}^{d_y}$ have joint density $f_{XY}$. Then the GMI satisfies

(i)

Concavity in $f_X$: Let $f_{Y|X}$ be conditional density of $\mathbf{Y}$ given $\mathbf{X}$ and let $g_X$ and $h_X$ be densities on ${\mathbb{R}}^{d_x}$. Then for ${\lambda }_1,{\lambda }_2\in [0,1]$, ${\lambda }_1+{\lambda }_2=1$

$${\tilde{I}}_p\left({\lambda }_1{f}_{Y|X}{g}_X+{\lambda }_2{f}_{Y|X}{h}_X\right)\ge {\lambda }_1{\tilde{I}}_p(f_{Y|X}{g}_X)+{\lambda }_2{\tilde{I}}_p(f_{Y|X}{h}_X).$$

(5)

The inequality is strict unless either ${\lambda }_1$ or ${\lambda }_2$ are zero or $h_X=g_X$.

(ii)

Convexity in $f_{Y|X}$: Let $g_{Y|X}$ and $h_{Y|X}$ be conditional densities of $\mathbf{Y}$ given $\mathbf{X}$ and let $f_X$ be marginal density. Then for ${\lambda }_1,{\lambda }_2\in [0,1]$, ${\lambda }_1+{\lambda }_2=1$

$${\tilde{I}}_p\left({\lambda }_1{g}_{Y|X}{f}_X+{\lambda }_2{h}_{Y|X}{f}_X\right)\le {\lambda }_1{\tilde{I}}_p(g_{Y|X}{f}_X)+{\lambda }_2{\tilde{I}}_p(h_{Y|X}{f}_X).$$

(6)

The inequality is strict unless either ${\lambda }_1$ or ${\lambda }_2$ are zero or $h_{Y|X}=g_{Y|X}$.

The GMI, $I_p(\mathbf{X};\mathbf{Y})$, satisfies properties analogous to the standard chain rule and the data-processing inequality [40]. For random variables $\mathbf{X}\in {\mathbb{R}}^{d_x},\mathbf{Y}\in {\mathbb{R}}^{d_y}$, and $\mathbf{Z}\in {\mathbb{R}}^{d_z}$ with conditional density $f_{XY|Z}$ we define the conditional GMI

$$\begin{array}{c}{\displaystyle I_p(\mathbf{X};\mathbf{Y}|\mathbf{Z})={\mathbb{E}}_{\mathbf{Z}}\left[{\tilde{I}}_p(f_{XY|Z})\right],\mathrm{where}}\hfill \\ {\displaystyle {\tilde{I}}_p(f_{XY|Z})=1-\int \int \frac{f_{XY|Z}(\mathbf{x},\mathbf{y}|\mathbf{z})f_{X|Z}(\mathbf{x}|\mathbf{z})f_{Y|Z}(\mathbf{y}|\mathbf{z})}{p{f}_{XY|Z}(\mathbf{x},\mathbf{y}|\mathbf{z})+q{f}_{X|Z}(\mathbf{x}|\mathbf{z})f_{Y|Z}(\mathbf{y}|\mathbf{z})}\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}.}\hfill \end{array}$$

(7)

The next theorem establishes a relation between the joint and conditional GMI.

Theorem 2.

For given d-dimensional random vector $\mathbf{X}$ with components $X_1,X_2,\cdots ,X_d$ and random variable Y,

$$I_p(\mathbf{X};Y)\ge I_p(X_1;Y)-{\displaystyle \sum _{i=1}^{d-1}\left(1-I_p(X_i;Y|{\mathbf{X}}^{i-1})\right)},$$

(8)

where ${\mathbf{X}}^i:=X_1,X_2,\cdots ,X_i$ and the conditional GMI $I_p(X_i;Y|{\mathbf{X}}^{i-1})$ is defined in (7).

For $d=2$ Theorem 2 reduces to

$$I_p(X_1,X_2;Y)\ge I_p(X_1;Y)-\left(1-I_p(X_2;Y|X_1)\right),$$

(9)

Please note that when ${\displaystyle {\displaystyle \sum _{i=1}^{d-1}\left(1-I_p(X_i;Y|{\mathbf{X}}^{i-1})\right)}\ge 1}$, the inequality (8) is trivial since $0\le I_p(X_1;Y)\le 1$. The proof of Theorem 2 is given in Appendix A.2. Theorem 2 is next applied to the case where $\mathbf{X}$ and $\mathbf{Y}$ form a Markov chain. The proof of the following “leany” data-processing inequality (Proposition 1) is provided in Appendices section, Appendix A.3.

Proposition 1.

Suppose random vectors $\mathbf{X},\mathbf{Y},\mathbf{Z}$ form a Markov chain denoted, $\mathbf{X}\to \mathbf{Y}\to \mathbf{Z}$, in the sense that $f_{XYZ}=f_{X|Y}{f}_{Y|Z}{f}_Z$. Then for $p\in (0,1)$

$$I_p(\mathbf{Y};\mathbf{X})\ge I_p(\mathbf{Z};\mathbf{X})-{\left(p{\mathbb{E}}_{XY}\left[{\delta }_{X,Y}\right]+(1-p)\right)}^{-1},$$

(10)

where

$${\displaystyle {\delta }_{X,Y}=\int \frac{f_{X|Y}(\mathbf{X}|\mathbf{Y})f_{Z|Y}(\mathbf{z}|\mathbf{Y})}{f_{X|Z}(\mathbf{X}|\mathbf{z})}\mathrm{d}\mathbf{z}.}$$

Furthermore, if both $\mathbf{X}\to \mathbf{Y}\to \mathbf{Z}$ and $\mathbf{X}\to \mathbf{Z}\to \mathbf{Y}$ together hold true, we have $I_p(\mathbf{Y};\mathbf{X})=I_p(\mathbf{Z};\mathbf{X})$.

The inequality in (10) becomes interpretable as the standard data-processing inequality $I_p(\mathbf{Y};\mathbf{X})\ge I_p(\mathbf{Z};\mathbf{X})$, when

$${\mathbb{E}}_Z\left[\frac{f(\mathbf{Z}|\mathbf{Y})}{f(\mathbf{Z}|\mathbf{X})}\right]=\infty ,$$

since

$${\mathbb{E}}_{XY}\left[{\delta }_{X,Y}\right]={\mathbb{E}}_{XY}\left({\displaystyle \frac{f(\mathbf{X}|\mathbf{Y})}{f(\mathbf{X})}{\mathbb{E}}_Z\left[\frac{f(\mathbf{Z}|\mathbf{Y})}{f(\mathbf{Z}|\mathbf{X})}\right]}\right).$$

2.2. The Friedman–Rafsky Estimator

Let a random sample ${\{{\mathbf{x}}_i,{\mathbf{y}}_i\}}_{i=1}^n$ from $f_{XY}(\mathbf{x},\mathbf{y})$ be available. Here we show that the GMI $I_p(\mathbf{X};\mathbf{Y})$ can be directly estimated without estimating the densities. The estimator is inspired by the MST construction of [33] that provides a consistent estimate of the Henze–Penrose divergence [14,15]. We denote by ${\mathbf{z}}_i$ the i-th joint sample ${\mathbf{x}}_i,{\mathbf{y}}_i$ and by ${\mathcal{Z}}_n$ the sample set ${\left\{{\mathbf{z}}_i\right\}}_{i=1}^n$. Divide the sample set ${\mathcal{Z}}_n$ into two subsets ${\mathcal{Z}}_{n^{\prime }}^{\prime }$ and ${\mathcal{Z}}_{n^{″}}^{″}$ with the proportion $\alpha =n^{\prime }/n$ and $\beta =n^{″}/n$, where $\alpha +\beta =1$.

Denote by ${\tilde{\mathcal{Z}}}_{n^{″}}$ the set

$$\left\{({\mathbf{x}}_{i_k},{\mathbf{y}}_{j_k}),k=1,\cdots ,n^{″},\mathrm{selected}\mathrm{at}\mathrm{random}\mathrm{from}{\mathcal{Z}}_{n^{″}}^{″}\right\}:$$

This means that for each ${\mathbf{z}}_{ik}=({\mathbf{x}}_{ik},{\mathbf{y}}_{ik})\in {\mathcal{Z}}_{n^{″}}^{″}$ given the first element ${\mathbf{x}}_{ik}$ the second element ${\mathbf{y}}_{ik}$ is replaced by a randomly selected $y\in {\left\{y_{jk}\right\}}_{j=1}^{n^{″}}$. This results in a random shuffling of the binary relation relating $y_{ik}$ in $y_{jk}$. The estimator of $I_p(\mathbf{X};\mathbf{Y})$ is derived based on the Friedman–Rafsky (FR) multivariate runs test statistic [33] on the concatenated data set, ${\mathcal{Z}}_{n^{\prime }}^{\prime }\cup {\tilde{\mathcal{Z}}}_{n^{″}}$. The FR test statistic is defined as the number of edges in the MST spanning the merged data set that connect a point in ${\mathcal{Z}}_{n^{\prime }}^{\prime }$ to a point in ${\tilde{\mathcal{Z}}}_{n^{″}}$. This test statistic is denoted by ${\mathfrak{R}}_{n^{\prime },n^{″}}:={\mathfrak{R}}_{n^{\prime },n^{″}}({\mathcal{Z}}_{n^{\prime }}^{\prime },{\tilde{\mathcal{Z}}}_{n^{″}})$. Please note that since the MST is unique with probability one (under the assumption that all density functions are Lebesgue continuous) then all inter point distances between nodes are distinct. This estimator converges to $I_p(\mathbf{X};\mathbf{Y})$ almost surely as $n\to \infty$. The procedure is summarized in Algorithm 1.

Algorithm 1: MST-based estimator of GMI

Input: Data set ${\mathcal{Z}}_n:=\left\{{({\mathbf{x}}_i,{\mathbf{y}}_i)}_{i=1}^n\right\}$

1: Find $\tilde{\alpha }$ using arguments in Section 2.4

2: $n^{\prime }\leftarrow \tilde{\alpha }n$, $n^{″}\leftarrow (1-\tilde{\alpha })n$

3: Divide ${\mathcal{Z}}_n$ into two subsets ${\mathcal{Z}}_{n^{\prime }}^{\prime }$ and ${\mathcal{Z}}_{n^{″}}^{″}$

4: ${\tilde{\mathcal{Z}}}_{n^{″}}\leftarrow \{{({\mathbf{x}}_{ik},{\mathbf{y}}_{jk})}_{k=1}^{n^{″}}$: shuffle first and second elements of pairs in ${\mathcal{Z}}_{n^{″}}^{″}\}$

5: $\widehat{\mathcal{Z}}\leftarrow {\mathcal{Z}}_{n^{\prime }}^{\prime }\cup {\tilde{\mathcal{Z}}}_{n^{″}}^{″}$

6: Construct MST on $\widehat{\mathcal{Z}}$

7: ${\mathfrak{R}}_{n^{\prime },n^{″}}\leftarrow \#$ edges connecting a node in ${\mathcal{Z}}_{n^{\prime }}^{\prime }$ to a node of ${\tilde{\mathcal{Z}}}_{n^{″}}$

8: ${\displaystyle {\widehat{I}}_p\leftarrow 1-{\mathfrak{R}}_{n^{\prime },n^{″}}\frac{n^{\prime }+n^{″}}{2n^{\prime }{n}^{″}}}$

Output: ${\widehat{I}}_p$, where $p=\tilde{\alpha }$

Theorem 3 shows that the output in Algorithm 1 estimates the GMI with parameter $p=\alpha$. The proof is provided in Appendix A.4.

Theorem 3.

For given proportionality parameter $\alpha \in (0,1)$, choose $n^{\prime }$, $n^{″}$ such that $n^{\prime }+n^{″}=n$ and, as $n\to \infty$, we have $n^{\prime }/n\to \alpha$ and $n^{″}/n\to \beta =1-\alpha$. Then

$${\displaystyle 1-{\mathfrak{R}}_{n^{\prime },n^{″}}({\mathcal{Z}}_{n^{\prime }}^{\prime },{\tilde{\mathcal{Z}}}_{n^{″}})\frac{n}{2n^{\prime }{n}^{″}}\to I_{\alpha }(\mathbf{X};\mathbf{Y}),a.s.}$$

(11)

Please note that the asymptotic limit in (11) depends on the proportionality parameter $\alpha$. Later in Section 2.4, we discuss the choice of an optimal parameter $\tilde{\alpha }$. In Figure 1, we illustrate the MST constructed over merged independent ($\rho =0$) and highly dependent ($\rho =0.9$) data sets drawn from two-dimensional normal distributions with correlation coefficients $\rho$. Notice that the edges of the MST connecting samples with different colors, corresponding to independent and dependent samples, respectively, are indicated in green. The total number of green edges is the FR test statistic ${\mathfrak{R}}_{n^{\prime },n^{″}}({\mathcal{Z}}_{n^{\prime }}^{\prime },{\tilde{\mathcal{Z}}}_{n^{″}})$.

Figure 1. The MST and FR statistic of spanning the merged set of normal points when $\mathbf{X}$ and $\mathbf{Y}$ are independent (denoted in blue points) and when $\mathbf{X}$ and $\mathbf{Y}$ are highly dependent (denoted in red points). The FR test statistic is the number of edges in the MST that connect samples from different color nodes (denoted in green) and it is used to estimate the GMI $I_p$.

2.3. Convergence Rates

In this subsection we characterize the MSE convergence rates of the GMI estimator of Section 2.2 in the form of upper bounds on the bias and the variance. This MSE bound is given in terms of the sample size n, the dimension d, and the proportionality parameter $\alpha$. Deriving convergence rates for mutual information estimators has been of interest in information theory and machine learning [22,27]. The rates are typically derived in terms of a smoothness condition on the densities, such as the Hölder condition [41]. Here we assume $f_X$, $f_Y$ and $f_{XY}$ have support sets ${\mathbb{S}}_X$, ${\mathbb{S}}_Y$, and ${\mathbb{S}}_{XY}:={\mathbb{S}}_X\times {\mathbb{S}}_Y$, respectively, and are smooth in the sense that they belong to Hölder continuous classes of densities ${\mathsf{\Sigma }}_d^s(\eta ,$ K), $0<\eta \le 1$ [42,43]:

Definition 1.

(Hölder class): Let $\mathcal{X}\subset {\mathbb{R}}^d$ be a compact space. The Hölder class of functions ${\mathsf{\Sigma }}_d(\eta ,K)$, with Hölder parameters η and K, consists of functions g that satisfy

$$\begin{array}{c}\left\{g:\parallel g(\mathbf{z})-p_{\mathbf{x}}^{\lfloor \eta \rfloor }(\mathbf{z}){\parallel }_d\le K\parallel \mathbf{x}-\mathbf{z}{\parallel }_d^{\eta },\mathbf{x},\mathbf{z}\in \mathcal{X}\right\},\hfill \end{array}$$

(12)

where $p_{\mathbf{x}}^k(\mathbf{z})$ is the Taylor polynomial (multinomial) of g of order k expanded about the point $\mathbf{x}$ and $\lfloor \eta \rfloor$ is defined as the greatest integer strictly less than η.

To explore the optimal choice of parameter $\alpha$ we require bounds on the bias and variance bounds, provided in Appendix A.5. To obtain such bounds, we will make several assumptions on the absolutely continuous densities $f_X$, $f_Y$, $f_{XY}$ and support sets ${\mathbb{S}}_X$, ${\mathbb{S}}_Y$, ${\mathbb{S}}_{XY}$:

(A.1)

Each of the densities belong to ${\displaystyle {\mathsf{\Sigma }}_d(\eta ,K)}$ with smoothness parameters $\eta$ and Lipschitz constant K.

(A.2)

The volumes of the support sets are finite, i.e., $0<\mathbb{V}({\mathbb{S}}_X)<\infty ,0<\mathbb{V}({\mathbb{S}}_Y)<\infty$.

(A.3)

All densities are bounded i.e., there exist two sets of constants $C_X^L,C_Y^L,C_{XY}^L$ and $C_X^U,C_Y^U,C_{XY}^U$ such that $0<C_X^L\le f_X\le C_X^U<\infty$, $0<C_Y^L\le f_Y\le C_Y^U<\infty$ and $0<C_{XY}^L\le f_{XY}\le C_{XY}^U<\infty$.

The following theorem on the bias follows under assumptions (A.1) and (A.3):

Theorem 4.

For given $\alpha \in (0,1)$, $\beta =1-\alpha$, $d\ge 2$, and $0<\eta \le 1$ the bias of the ${\mathfrak{R}}_{n^{\prime },n^{″}}:={\mathfrak{R}}_{n^{\prime },n^{″}}({\mathcal{Z}}_{n^{\prime }}^{\prime },{\tilde{\mathcal{Z}}}_{n^{″}})$ satisfies

$$\begin{array}{c}{\displaystyle |\frac{\mathbb{E}\left[{\mathfrak{R}}_{n^{\prime },n^{″}}\right]}{n}-2\alpha \beta \int \int \frac{f_{\mathbf{X}\mathbf{Y}}(\mathbf{x},\mathbf{y})f_{\mathbf{X}}(\mathbf{x})f_{\mathbf{Y}}(\mathbf{y})}{\alpha f_{\mathbf{X}\mathbf{Y}}(\mathbf{x},\mathbf{y})+\beta f_{\mathbf{X}}(\mathbf{x})f_{\mathbf{Y}}(\mathbf{y})}\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}|}\hfill \\ \le O\left(max\left\{n^{-{\eta }^2/(d(1+\eta ))},{(\beta n)}^{-\eta /(1+\eta )},c_d{n}^{-1}\right\}\right),\hfill \end{array}$$

(13)

where $c_d$ is the largest possible degree of any vertex of MST on ${\mathcal{Z}}_{n^{\prime }}^{\prime }\cup {\tilde{\mathcal{Z}}}_{n^{″}}$. The explicit form of (13) is provided in Appendix A.5.

Please note that according to Theorem 13 in [44], the constant $c_d$ is lower bounded by $\mathsf{\Omega }\left(\sqrt{d}{2}^{n(1-H(\gamma ))}\right)$, $\gamma =2^{-d}$ and $H(\gamma )$ is the binary entropy i.e.,

$$H(\gamma )=-\gamma log\gamma -(1-\gamma )log(1-\gamma ).$$

A proof of Theorem 4 is given in Appendix A.5. The next theorem gives an upper bound on the variance of the FR estimator ${\mathfrak{R}}_{n^{\prime },n^{″}}$. The proof of the variance result requires a different approach than the bias bound (the Efron–Stein inequality [45]). It is similar to arguments in ([46], Appendix A.3), and is omitted. In Theorem 5 we assume that the densities $f_X$, $f_Y$, and $f_{XY}$ are absolutely continuous and bounded (A.3).

Theorem 5.

Given $\alpha \in (0,1)$, the variance of the estimator ${\mathfrak{R}}_{n^{\prime },n^{″}}:={\mathfrak{R}}_{n^{\prime },n^{″}}({\mathcal{Z}}_{n^{\prime }}^{\prime },{\tilde{\mathcal{Z}}}_{n^{″}})$ is bounded by

$${\displaystyle Var\left({\displaystyle \frac{{\mathfrak{R}}_{n^{\prime },n^{″}}}{n}}\right)\le \frac{(1-\alpha )c_d}{n},\alpha =n^{\prime }/n,}$$

(14)

where $c_d$ is a constant depending only on the dimension d.

2.4. Minimax Parameter $\alpha$

Recall assumptions (A.1), (A.2), and (A.3) in Section 2.3. The constant $\alpha$ can be chosen to minimize the maximum the MSE converges rate where the maximum is taken over the space of Hölder smooth joint densities $f_{XY}$.

Throughout this subsection we use the following notations:

• ${ϵ}_{XY}:=f_{XY}/f_X{f}_Y$,
• $C_{ϵ}^L:=C_{XY}^L/C_X^U{C}_Y^U$ and $C_{ϵ}^U:=C_{XY}^U/C_X^L{C}_Y^L$,
• $C_n:=C_{XY}^{L}n/2$,
• ${\displaystyle {\alpha }_0^L:=\frac{2}{C_n}}$ and ${\alpha }_0^U:=min\left\{{\displaystyle \frac{1}{4},\frac{1+1/C_n}{4+2C_{ϵ}^U},1-n^{\eta /d-1}}\right\}$, where $\eta$ is the smoothness parameter,
• $l_n:=⌊n^{\eta /(d^2(1+\eta ))}⌋$.

Now define ${\tilde{G}}_{{ϵ}_{XY},n}^{\alpha ,\beta }(\mathbf{x},\mathbf{y})$ by

$${\displaystyle \frac{({ϵ}_{XY}(\mathbf{x},\mathbf{y})+1/(\beta C_n))(1+{ϵ}_{XY}(\mathbf{x}\mathbf{y})+1/(\beta C_n))}{{(\alpha +\beta {ϵ}_{XY}(\mathbf{x},\mathbf{y}))}^2},\beta =1-\alpha .}$$

(15)

Consider the following optimization problem:

$$\begin{array}{cc}\underset{\alpha }{\min }\underset{{ϵ}_{XY}}{\max }\hfill & \tilde{\Delta }(\alpha ,{ϵ}_{XY})+c_d(1-\alpha )n^{-1}\hfill \\ \mathrm{subject}\mathrm{to}\hfill & C_{ϵ}^L\le {ϵ}_{XY}\le C_{ϵ}^U,\hfill \\ & {\alpha }_0^L\le \alpha \le {\alpha }_0^U,\hfill \end{array}$$

(16)

where

$${\displaystyle \tilde{\Delta }(\alpha ,{ϵ}_{XY}):=D(n,l_n,d,\eta )+\tilde{D}(n,l_n,d)C_{XY}^U{\int \int }_{{\mathbb{S}}_{XY}}{\tilde{G}}_{{ϵ}_{XY},n}^{\alpha ,\beta }(\mathbf{x},\mathbf{y})\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y},}$$

(17)

and

$$\begin{array}{c}{\displaystyle D(n,l_n,d,\eta )=c_2{l}_n^d{n}^{-1}+c_d{2}^d{n}^{-1}+c^{\prime }{l}_n^d{n}^{-\eta /d}+c{l}_n^d{n}^{-1/d}+2c_1{l}_n^{d-1}{n}^{1/d-1}+c_3{l}_n^{-d\eta },}\hfill \end{array}$$

(18)

$$\begin{array}{c}{\displaystyle \tilde{D}(n,l_n,d)=2+n^{-1}2c^{″}\sum _{i=1}^M{l}_n{l}_n^d{a}_i^{-1}+n^{-3/2}2c_1^{\prime }\sum _{i=1}^M{l}_n{l}_n^{d/2}\sqrt{b_i}{a}_i^2}\hfill \\ {\displaystyle +n^{-1}\sum _{i=1}^{M}2n^{-3/2}{l}_n^{-d/2}\frac{\sqrt{b_i}}{a_i^2}{\left(n{a}_i{l}_n^d+n^2{a}_i^2\right)}^{1/2}{\left(n{b}_i{l}_n^d+n^2{b}_i^2\right)}^{1/2}.}\hfill \end{array}$$

(19)

Please note that in (18), $c,c^{\prime },c_1,c_2$ are constants, and $c_d$ only depends on the dimension d. Also, in (19), $a_i$ and $b_i$ are constants. Let ${ϵ}_{XY}^{*}$ be the optimal ${ϵ}_{XY}$ i.e., ${ϵ}_{XY}^{*}$ be the solution of the optimization problem (16). Set

$$\begin{array}{c}{\displaystyle \mathsf{\Xi }(\alpha ):=\frac{\mathrm{d}}{\mathrm{d}\alpha }\left(\tilde{\Delta }(\alpha ,{ϵ}_{XY}^{*})+c_d(1-\alpha )n^{-1}\right),}\hfill \end{array}$$

(20)

such that ${\displaystyle \tilde{\Delta }(\alpha ,{ϵ}_{XY}^{*})}$ is (17) when ${ϵ}_{XY}={ϵ}_{XY}^{*}$. For $\alpha \in [{\alpha }_0^L,{\alpha }_0^U]$, the optimal choice of ${ϵ}_{XY}$ in terms of maximizing the MSE is ${ϵ}_{XY}^{*}=C_{ϵ}^U$ and the saddle point for the parameter $\alpha$, denoted by $\tilde{\alpha }$, is given as follows:

• $\tilde{\alpha }={\alpha }_0^U$, if $\mathsf{\Xi }({\alpha }_0^U)<0$.
• $\tilde{\alpha }={\alpha }_0^L$, if $\mathsf{\Xi }({\alpha }_0^L)>0$.
• $\tilde{\alpha }={\mathsf{\Xi }}^{-1}(0)$, if ${\alpha }_0^L\le {\mathsf{\Xi }}^{-1}(0)\le {\alpha }_0^U$.

Further details are given in Appendix A.6.

3. Simulation Study

In this section, numerical simulations are presented that illustrate the theory in Section 2. We perform multiple experiments to demonstrate the utility of the proposed GMI estimator of the HP-divergence in terms of the dimension d and the sample size n. Our proposed MST-based estimator of the GMI is compared to density plug-in estimators of the GMI, in particular the standard KDE density plug-in estimator of [22], where the convergence rates of Theorems 4 and 5 are validated. We use multivariate normal simulated data in the experiments. In this section, we also discuss the choice of the proportionality parameter $\alpha$ and compare runtime of the proposed GMI estimator approach with KDE method.

Here we perform four sets of experiments to illustrate the proposed GMI estimator. For the first set of experiments the MSE of the GMI estimator in Algorithm 1 is shown in Figure 2-left. The samples were drawn from d-dimensional normal distribution, with various sample sizes and dimensions $d=6,10,12$. We selected the proportionality parameter $\alpha =0.3$ and computed the MSE in terms of the sample size n. We show the log–log plot of MSE when n varies in $[100,1500]$. Please note that the empirically optimal proportion $\alpha$ depends on n, so to avoid the computational complexity we fixed $\alpha$ for this experiment. The experimental result shown in Figure 2-left validates the theoretical MSE growth rates derived from (13) and (14), i.e., decreasing sub-linearly in n and increasing exponentially in d.

Figure 2. (left) Log–log plot of theoretical and experimental MSE of the proposed MST-based GMI estimator as a function of sample size n for $d=6,10,12$ and fixed smoothness parameter $\eta$. (right) The GMI estimator was implemented using two approaches, Algorithm 1 and KDE method where the KDE-GMI used KDE density estimators in the formula (2). In this experiment, samples are generated from the two-dimensional normal distribution with zero mean and covariance matrix (21) for various value of $\rho \in [0.1,0.9]$.

In Figure 2-right, we compare the proposed MST-based GMI estimator with the KDE-GMI estimator [22]. For the KDE approach, we estimated the joint and marginal densities and then plugged them into the proposed expression (2). The bandwidth h used for the KDE plug-in estimator was set as $h=n^{-1/(d+1)}$. The choice of h minimizes the bound on the MSE of the plug-in estimator. We generated data from the two-dimensional normal distribution with zero mean and covariance matrix

$$\left(\begin{array}{cc}1& \rho \\ \rho & 1\end{array}\right).$$

(21)

The coefficient $\rho$ is varied in range $[0.1,0.9]$. The true GMI was computed by the Monte Carlo approximation to the integral (2). Please note that as $\rho$ increases, the MST-GMI outperforms the KDE-GMI approach. In this set of experiments $\alpha =0.6$.

Figure 3 again compares the MST-GMI estimator with the KDE-GMI estimator. samples are drawn from the multivariate standard normal distribution with dimensions $d=4$ and $d=12$. In both cases the proportionality parameter $\alpha =0.5$. The left plots in Figure 3 show the MSE (100 trials) of the GMI estimator implemented with an KDE estimator (with bandwidth as in Figure 2 i.e., $h=n^{-1/(d+1)}$) for dimensions $d=4,12$ and various sample sizes. For all dimensions and sample sizes the MST-GMI estimator also outperforms the plug-in KDE-GMI estimator based on the estimated log–log MSE slope given in Figure 3 (left plots). The right plots in Figure 3 compares the MST-GMI with the KDE-GMI. In this experiment, the error bars denote standard deviations with 100 trials. We observe that for higher dimension $d=12$ and larger sample size n, the KDE-GMI approaches the true GMI at a slower rate than the MST-GMI estimator. This reflects the power of the proposed graph-based approach to estimating GMI.

Figure 3. MSE log–log plots as a function of sample size n (left) for the proposed MST-GMI estimator (“Estimated GMI”) and the standard KDE-GMI plug-in estimator of GMI. The right column of plots correspond to the GMI estimated for dimension $d=4$ (top) and $d=12$ (bottom). In both cases the proportionality parameter $\alpha$ is $0.5$. The MST-GMI estimator in both plots for sample size n in $[700,1600]$ outperforms the KDE-GMI estimator, especially for larger dimensions.

The comparison between MSEs for various dimension d is shown in Figure 4 (left). This experiment highlights the impact of higher dimension on the GMI estimators. As expected, for larger sample size n, MSE decreases while for higher dimension it increases. In this setting, we have generated samples from standard normal distribution with size $n\in [{10}^2,4\times {10}^3]$ and $\alpha =0.5$. From Figure 4 (left) we observe that for larger sample size, MSE curves are ordered based on their corresponding dimensions. Results in Section 2.4 strongly depend on the lower bounds $C_X^L,C_Y^L,C_{XY}^L$ and upper bounds $C_X^U,C_Y^U,C_{XY}^U$ and provide optimal parameter $\alpha$ in the range $[{\alpha }_0^L,{\alpha }_0^U]$, therefore in the experiment section we only analyze one case where the lower bounds $C_X^L,C_Y^L,C_{XY}^L$ and upper bounds $C_X^U,C_Y^U,C_{XY}^U$ are known and the optimal $\alpha$ becomes ${\alpha }_0^L$. Figure 4 (right) illustrates the MSE vs proportion parameter $\alpha$ when $n=500,{10}^4$ samples are generated from truncated normal distribution with $\rho =0.7,0.5$. First, following Section 2.4, we compute the bound $[{\alpha }_0^L,{\alpha }_0^U]$ and then derive the optimal $\alpha$ in this range. Therefore, each experiment with different sample size and $\rho$ provides different range $[{\alpha }_0^L,{\alpha }_0^U]$. We observe that the MSE does not appeared a monotonic function in $\alpha$ and its behavior strongly depends on sample size n, d, and density functions’ bounds. Additional study of the dependency is described in Appendix A.6. In this set of experiments $\mathsf{\Xi }({\alpha }_0^L)>0$, therefore following the results in Section 2.4, we have $\tilde{\alpha }={\alpha }_0^L$. In this experiment the optimal value of $\alpha$ is always the lower bound ${\alpha }_0^L$ and indicated in the Figure 4 (right).

Figure 4. MSE log–log plots as a function of sample size n for the proposed FR estimator. We compare the MSE of our proposed FR estimator for various dimensions $d=15,20,50$ (left). As d increases, the blue curve takes larger values than green and orange curves i.e., MSE increases as d grows. However, this is more evidential for large sample size n. The second experiment (right) focuses on optimal proportion $\alpha$ for $n=500,{10}^4$ and $\rho =0.7,0.5$. $\tilde{\alpha }$ is the optimal $\alpha$ for $\alpha \in [{\alpha }_0^L,{\alpha }_0^U]$.

The parameter $\alpha$ is studied further for three scenarios where the lower bounds $C_X^L,C_Y^L,C_{XY}^L$ and upper bounds $C_X^U,C_Y^U,C_{XY}^U$ are assumed unknown, therefore results in Section 2.4 are not applicable. In this set of experiments, we varied $\alpha$ in the range $(0,1)$ to divide our original sample. We generated sample from an isotropic multivariate standard normal distribution $(\rho =0)$ in all three scenarios (all features are independent). Therefore, the true GMI is zero and in all scenarios the GMI column, corresponding to the MST-GMI, is compared with zero. In each scenario we fixed dimension d and sample size n and varied $\alpha =0.2,0.5,0.8$. The dimension and sample size in Scenarios 1,2, and 3 are $d=6,8,10$ and $n=1000,1500,2000$, respectively. In Table 1 the last column ($\alpha$) stars the parameter $\alpha \in \{0.2,0.5,0.8\}$ with the minimum MSE and GMI $(I_{\alpha })$ in each scenario. Table 1 shows that in these sets of experiments when $\alpha =0.5$, the GMI estimator has less MSE (i.e., is more accurate) than when $\alpha =0.2$ or $\alpha =0.8$. This experimentally demonstrates that if we split our training data, the proposed Algorithm 1 performs better with $\alpha =0.5$.

Table 1. Comparison between different scenarios of various dimensions and sample sizes in terms of parameter $\alpha$. We applied the MST-GMI estimator to estimate the GMI ($I_{\alpha }$) with $\alpha =0.2,0.5,0.8$. We varied dimension $d=6,8,10$ and sample size $n=1000,1500,2000$ in each scenario. We observe that for $\alpha =\{0.2,0.5,0.8\}$, the MST-GMI estimator provides lowest MSE when $\alpha =0.5$ indicated by star (*).

Overview Table for Different d, n, and $\mathit{\alpha }$
Experiments	Dimension ($\mathit{d}$)	Sample Size ($\mathit{n}$)	GMI (${\mathit{I}}_{\mathit{\alpha }}$)	MSE ($\times {\mathbf{10}}^{-\mathbf{4}}$)	Parameter ($\mathit{\alpha }$)
Scenario 1–1	6	1000	0.0229	12	0.2
Scenario 1–2	6	1000	0.0143	4.7944	0.5 *
Scenario 1–3	6	1000	0.0176	6.3867	0.8
Scenario 2–1	8	1500	0.0246	11	0.2
Scenario 2–2	8	1500	0.0074	1.6053	0.5 *
Scenario 2–3	8	1500	0.0137	5.3863	0.8
Scenario 3–1	10	2000	0.0074	2.3604	0.2
Scenario 3–2	10	2000	0.0029	0.54180	0.5 *
Scenario 3–3	10	2000	0.0262	11	0.8

Finally, Figure 5 shows the runtime as a function of sample size n. We vary sample size in the range $[{10}^3,{10}^4]$. Observe that for smaller number of samples the KDE-GMI method is slightly faster but as n becomes large we see significant relative speedup of the proposed MST-GMI method.

Figure 5. Runtime of KDE approach and proposed MST-based estimator of GMI vs sample size. The proposed GMI estimator achieves significant speedup, while for small sample size, the KDE method becomes overly fast. Please note that in this experiment the sample is generated from the Gaussian distribution in dimension $d=2$.

4. Conclusions

In this paper, we have proposed a new measure of mutual information, called Geometric MI (GMI), which is related to the Henze–Penrose divergence. The GMI can be viewed as dependency measure that is the limit of the Friedman–Rafsky test statistic, which depends on the MST over all data points. We established some properties of the GMI in terms of convexity/concavity, chain rule, and a type of data-processing inequality. A direct estimator of the GMI, called the MST-GMI, was introduced that uses random permutations of observed relationships between variables in the multivariate samples. An explicit form for the MSE convergence rate bound was derived that depends on a free parameter called the proportionality parameter. An asymptotically optimal form for this free parameter was given that minimizes the MSE convergence rate. Simulation studies were performed that illustrate and verify the theory.

Appendix A.1. Theorem 1

Proof. 

The proof is similar to the result for standard (Shannon) mutual information. However, we require the following lemma, proven in analogous manner to the log-sum inequality:

Lemma A1.

For non-negative real numbers ${\alpha }_1,\cdots ,{\alpha }_n$ and ${\beta }_1,\cdots ,{\beta }_n$, given $p\in (0,1)$, $q=1-p$,

$${\displaystyle \sum _{i=1}^n{\alpha }_i{\left(p\left(\frac{{\beta }_i}{{\alpha }_i}\right)+q\right)}^{-1}\ge \sum _{i=1}^n{\alpha }_i{\left(p\left(\frac{{\displaystyle \sum _{i=1}^n}{\beta }_i}{{\displaystyle \sum _{i=1}^n}{\alpha }_i}\right)+q\right)}^{-1}.}$$

Notice this follows by using the convex function ${\displaystyle u(y)=y^2/(p+qy)}$ for any $p\in (0,1)$, $q=1-p$, and the Jensen inequality.

Define the shorthand $\underset{\mathbf{x}}{\int }$, $\underset{\mathbf{y}}{\int }$, and $\underset{\mathbf{x}\mathbf{y}}{\int }$ for $\int \mathrm{d}\mathbf{x}$, $\int \mathrm{d}\mathbf{y}$ and $\int \int \mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}$, respectively. To prove part (i) of Theorem 1, we represent the LHS of (5) as:

$$\begin{array}{c}{\displaystyle {\tilde{I}}_p\left({\lambda }_1{f}_{Y|X}{g}_X+{\lambda }_2{f}_{Y|X}{h}_X\right)=1-{\int }_{\mathbf{x}\mathbf{y}}\left({\lambda }_1{f}_{Y|X}{g}_X+{\lambda }_2{f}_{Y|X}{h}_X\right)\times }\hfill \\ {\displaystyle {\left[{\displaystyle p\frac{{\lambda }_1{f}_{Y|X}{g}_X+{\lambda }_2{f}_{Y|X}{h}_X}{\left({\int }_{\mathbf{x}}{\lambda }_1{f}_{Y|X}{g}_X+{\lambda }_2{f}_{Y|X}{h}_X\right)\left({\int }_{\mathbf{y}}{\lambda }_1{f}_{Y|X}{g}_X+{\lambda }_2{f}_{Y|X}{h}_X\right)}+q}\right]}^{-1}}\hfill \\ {\displaystyle =1-{\int }_{\mathbf{x}\mathbf{y}}\left({\lambda }_1{f}_{Y|X}{g}_X+{\lambda }_2{f}_{Y|X}{h}_X\right){\left[{\displaystyle p\frac{f_{Y|X}}{\left({\int }_{\mathbf{x}}{\lambda }_1{f}_{Y|X}{g}_X+{\lambda }_2{f}_{Y|X}{h}_X\right)}+q}\right]}^{-1}.}\hfill \end{array}$$

Furthermore, the RHS of (5) can be rewritten as

$$\begin{array}{c}{\lambda }_1{\tilde{I}}_p(f_{Y|X}{g}_X)+{\lambda }_2{\tilde{I}}_p(f_{Y|X}{h}_X)\hfill \\ {\displaystyle =1-\underset{\mathbf{x}\mathbf{y}}{\int }\left({\displaystyle {\lambda }_1{f}_{Y|X}{g}_X{\left[p\frac{f_{Y|X}{g}_X}{\left(\underset{\mathbf{x}}{\int }{f}_{Y|X}{g}_X\right)\left(\underset{\mathbf{y}}{\int }{f}_{Y|X}{g}_X\right)}+q\right]}^{-1}+{\lambda }_2{f}_{Y|X}{h}_X{\left[p\frac{f_{Y|X}{h}_X}{\left(\underset{\mathbf{x}}{\int }{f}_{Y|X}{h}_X\right)\left(\underset{\mathbf{y}}{\int }{f}_{Y|X}{h}_X\right)}+q\right]}^{-1}}\right)}\hfill \\ {\displaystyle =1-\underset{\mathbf{x}\mathbf{y}}{\int }\left({\displaystyle {\lambda }_1{f}_{Y|X}{g}_X{\left[p\frac{f_{Y|X}}{\underset{\mathbf{x}}{\int }{f}_{Y|X}{g}_X}+q\right]}^{-1}+{\lambda }_2{f}_{Y|X}{h}_X{\left[p\frac{f_{Y|X}}{\underset{\mathbf{x}}{\int }{f}_{Y|X}{h}_X}+q\right]}^{-1}}\right).}\hfill \end{array}$$

Thus, to prove $\mathrm{LHS}\ge \mathrm{RHS}$, we use the inequality below:

$$\begin{array}{c}{\displaystyle \left({\lambda }_1{f}_{Y|X}{g}_X+{\lambda }_2{f}_{Y|X}{h}_X\right){\left[p\frac{f_{Y|X}}{\underset{\mathbf{x}}{\int }{\lambda }_1{f}_{Y|X}{g}_X+{\lambda }_2{f}_{Y|X}{h}_X}+q\right]}^{-1}}\hfill \\ {\displaystyle \le \left({\lambda }_1{f}_{Y|X}{g}_X\right){\left[p\frac{\pi }{\underset{\mathbf{x}}{\int }{f}_{Y|X}{g}_X}+q\right]}^{-1}+\left({\lambda }_2{f}_{Y|X}{h}_X\right){\left[p\frac{\pi }{\underset{\mathbf{x}}{\int }{f}_{Y|X}{h}_X}+q\right]}^{-1}.}\hfill \end{array}$$

In Lemma A1, let

$$\begin{array}{c}{\displaystyle {\alpha }_1=\frac{{\lambda }_1\left(\underset{\mathbf{x}}{\int }{f}_{Y|X}{g}_X\right)\left({\lambda }_1{f}_{Y|X}{g}_X+{\lambda }_2{f}_{Y|X}{h}_X\right)}{\underset{\mathbf{x}}{\int }{\lambda }_1{f}_{Y|X}{g}_X+{\lambda }_2{f}_{Y|X}{h}_X},}\\ {\displaystyle {\alpha }_2=\frac{{\lambda }_2\left(\underset{\mathbf{x}}{\int }{f}_{Y|X}{h}_X\right)\left({\lambda }_1{f}_{Y|X}{g}_X+{\lambda }_2{f}_{Y|X}{h}_X\right)}{\underset{\mathbf{x}}{\int }{\lambda }_1{f}_{Y|X}{g}_X+{\lambda }_2{f}_{Y|X}{h}_X},}\end{array}$$

and for $i=1,2$,

$${\displaystyle {\beta }_i=\frac{{\lambda }_i{f}_{Y|X}\left({\lambda }_1{f}_{Y|X}{g}_X+{\lambda }_2{f}_{Y|X}{h}_X\right)}{\underset{\mathbf{x}}{\int }{\lambda }_1{f}_{Y|X}{g}_X+{\lambda }_2{f}_{Y|X}{h}_X}.}$$

Then the claimed assertion (i) is obtained. Part (ii) follows by convexity of $D_p$ and the following expression:

$$\begin{array}{c}{\tilde{I}}_p\left({\lambda }_1{g}_{Y|X}{f}_X+{\lambda }_2{h}_{Y|X}{f}_X\right)\hfill \\ {\displaystyle =D_p\left({\lambda }_1{f}_X{g}_{Y|X}+{\lambda }_2{f}_X{h}_{Y|X},\left(\underset{\mathbf{x}}{\int }{\lambda }_1{f}_X{g}_{Y|X}+{\lambda }_2{f}_X{h}_{Y|X}\right)\left(\underset{\mathbf{y}}{\int }{\lambda }_1\varphi {\pi }_1+{\lambda }_2\varphi {\pi }_2\right)\right)}\hfill \\ {\displaystyle =D_p({\lambda }_1{f}_X{g}_{Y|X}+{\lambda }_2{f}_X{h}_{Y|X},f_X\left(\underset{\mathbf{x}}{\int }{\lambda }_1{f}_X{g}_{Y|X}+{\lambda }_2{f}_X{h}_{Y|X}\right)}\hfill \\ {\displaystyle =D_p\left({\lambda }_1{f}_X{g}_{Y|X}+{\lambda }_2{f}_X{h}_{Y|X},{\lambda }_1\left(\underset{\mathbf{x}}{\int }{f}_X{g}_{Y|X}\right)\left(\underset{\mathbf{y}}{\int }{f}_X{g}_{Y|X}\right)+{\lambda }_2\left(\underset{\mathbf{x}}{\int }{f}_X{h}_{Y|X}\right)\left(\underset{\mathbf{y}}{\int }{f}_X{h}_{Y|X}\right)\right).}\hfill \end{array}$$

Therefore, the claim in (6) is proved. □

Appendix A.2. Theorem 2

Proof. 

We start with (9). Given $p\in (0,1)$ and $q=1-p$, we can easily check that for positive $t>q$, $s>q$, such that $t,s\ne 1$:

$$(t+s)\left[(ts)+q(1-t-s)\right]-p(ts)\ge 0.$$

This implies

$${\displaystyle p\left(\frac{t-q}{p}\right)\left(\frac{s-q}{p}\right)+q\ge \frac{ts}{t+s}.}$$

By substituting

$${\displaystyle \frac{f_{X_{1}Y}(x_1,y)}{f_{X_1}(x_1)f_Y(y)}=\frac{t-q}{p},\frac{f_{X_{2}Y|X_1}(x_2,y|x_1)}{f_{X_2|X_1}(x_2|x_1)f_{Y|X_1}(y|x_1)}=\frac{s-q}{p},}$$

we get

$$\begin{array}{c}{\displaystyle {\left(p\frac{f_{X_1{X}_{2}Y}(x_1,x_2,y)}{f_{X_1{X}_2}(x_1,x_2)f_Y(y)}+q\right)}^{-1}\le {\left(p\frac{f_{X_{1}Y}(x_1,y)}{f_{X_1}(x_1)f_Y(y)}+q\right)}^{-1}}\hfill \\ {\displaystyle +{\left(p\frac{f_{X_{2}Y|X_1}(x_2,y|x_1)}{f_{X_2|X_1}(x_2|x_1)f_{Y|X_1}(y|x_1)}+q\right)}^{-1}.}\hfill \end{array}$$

(A1)

Consequently

$$\begin{array}{c}{\displaystyle I_p(X_1,X_2;Y)\ge I_p(X_1;Y)-{\mathbb{E}}_f\left[{\left(p\frac{f_{X_{2}Y|X_1}(x_2,y|x_1)}{f_{X_2|X_1}(x_2|x_1)f_{Y|X_1}(y|x_1)}+q\right)}^{-1}\right].}\hfill \end{array}$$

(A2)

Here f is the joint PDF of random vector $(X_1,X_2,Y)$. From the conditional GMI definition in (7) the expectation term in (A2) is equivalent to $1-I_p(X_2;Y|X_1)$. This completes the proof. □

Appendix A.3. Proposition 1

Proof. 

Recall the Theorem 2, part (i). First from $\mathbf{X}\to \mathbf{Y}\to \mathbf{Z}$ we have $f_{XYZ}=f_{XY}{f}_{Z|Y}$ and then by applying the Jensen inequality,

$$\begin{array}{c}{I}_p(\mathbf{X};\mathbf{Y})=I_p(\mathbf{X};\mathbf{Y},\mathbf{Z})\mathrm{and}\hfill \\ {\displaystyle I_p(\mathbf{X};\mathbf{Y},\mathbf{Z})\ge I_p(\mathbf{Z};\mathbf{X})-\mathbb{E}\left[{\left(p\pi (\mathbf{X},\mathbf{Y},\mathbf{Z})+q\right)}^{-1}\right]}\hfill \\ {\displaystyle \ge I_p(\mathbf{Z};\mathbf{X})-{\left(p\mathbb{E}\left[\pi (\mathbf{X},\mathbf{Y},\mathbf{Z})\right]+q\right)}^{-1},}\hfill \end{array}$$

(A3)

where

$${\displaystyle \pi (\mathbf{x},\mathbf{y},\mathbf{z})=\frac{f_{YX|Z}(\mathbf{y},\mathbf{x}|\mathbf{z})}{f_{Y|Z}(\mathbf{y}|\mathbf{z})f_{X|Z}(\mathbf{x}|\mathbf{z})}.}$$

Now by Markovian property we can immediately simplify the RHS in (A3) to the RHS in (10).

Furthermore, we can easily show that if $\mathbf{X}\to \mathbf{Z}\to \mathbf{Y}$, we have $f_{XYZ}=f_{ZX}{f}_{Y|Z}$ and therefore $I_p(\mathbf{Z};\mathbf{X})=I_p(\mathbf{X};\mathbf{Y},\mathbf{Z})$. This together with (A3) proves that under both conditions $\mathbf{X}\to \mathbf{Y}\to \mathbf{Z}$ and $\mathbf{X}\to \mathbf{Z}\to \mathbf{Y}$, the equality $I_p(\mathbf{X};\mathbf{Y})=I_p(\mathbf{Z};\mathbf{X})$ holds true. □

Appendix A.4. Theorem 3

Proof. 

We first derive two required Lemmas A2 and A3 below:

Lemma A2.

Consider random vector $\mathbf{Z}=(\mathbf{X},\mathbf{Y})$ with joint probability density function (pdf) $f_{XY}$. Let ${\mathfrak{Z}}_n=\{{\mathbf{z}}_1,\dots ,{\mathbf{z}}_n\}={\left\{({\mathbf{x}}_i,{\mathbf{y}}_i)\right\}}_{i=1}^n$ be a set of samples with pdf $f_{XY}$. Let ${\mathfrak{Z}}_{n^{\prime }}^{\prime }$ and ${\mathfrak{Z}}_{n^{″}}^{″}$ be two distinct subsets of ${\mathfrak{Z}}_n$ such that $n^{\prime }+n^{″}=n$ and sample proportion is $\alpha =n^{\prime }/n$ and $\beta =1-\alpha$. Next, let ${\tilde{\mathfrak{Z}}}_{n^{″}}=\{{\tilde{\mathbf{z}}}_1,\dots ,{\tilde{\mathbf{z}}}_{n^{″}}\}$ be a set of pairs such that ${\tilde{\mathbf{z}}}_k=({\mathbf{x}}_{i_k},{\mathbf{y}}_{j_k})$, $k=1,\cdots ,n^{″}$ are selected at random from ${\mathfrak{Z}}_{n^{″}}^{\prime }$. Denote $\tilde{\mathbf{Z}}=(\tilde{\mathbf{X}},\tilde{\mathbf{Y}})$ as the random vector corresponding to samples in ${\tilde{\mathfrak{Z}}}_{n^{″}}$. Then as $n\to \infty$ such that $n^{″}$ also grows in a linked manner that $\beta \ne 0$ then the distribution of $\tilde{\mathbf{Z}}$ convergences to $f_X\times f_Y$ i.e., random vectors $\tilde{\mathbf{X}}$ and $\tilde{\mathbf{Y}}$ become mutually independent.

Proof. Consider two subsets $\mathbb{A},\mathbb{B}\subset {\mathbb{R}}^n$, then we have

$$\begin{array}{c}{\displaystyle P(\tilde{\mathbf{X}}\in \mathbb{A},\tilde{\mathbf{Y}}\in \mathbb{B})=\mathbb{E}\left[{\mathbf{I}}_{\mathbb{A}}(\tilde{\mathbf{X}}).{\mathbf{I}}_{\mathbb{B}}(\tilde{\mathbf{Y}})\right]=\mathbb{E}\left[{\displaystyle \sum _{i,j}{\mathbf{I}}_{\mathbb{A}}({\mathbf{X}}_i).{\mathbf{I}}_{\mathbb{B}}({\mathbf{Y}}_j).P\left((\tilde{\mathbf{X}},\tilde{\mathbf{Y}})=({\mathbf{X}}_i,{\mathbf{Y}}_j)|{\mathbf{Z}}_n\right)}\right].}\hfill \end{array}$$

Here ${\mathbf{I}}_{\mathbb{A}}$ stands for the indicator function. Please note that

$${\displaystyle P\left((\tilde{\mathbf{X}},\tilde{\mathbf{Y}})=({\mathbf{X}}_i,{\mathbf{Y}}_j)|{\mathbf{Z}}_n\right)=\frac{1}{{n^{″}}^2},}$$

and ${\mathbf{X}}_i$ and ${\mathbf{Y}}_j$, $i\ne j$ are independent, therefore

$$\begin{array}{c}{\displaystyle P(\tilde{\mathbf{X}}\in \mathbb{A},\tilde{\mathbf{Y}}\in \mathbb{B})=\frac{1}{{n^{″}}^2}\sum _{i\ne j}P({\mathbf{X}}_i\in \mathbb{A})P({\mathbf{Y}}_j\in \mathbb{B})+\frac{1}{{n^{″}}^2}\sum _{i=1}^{n}P({\mathbf{X}}_i\in \mathbb{A},{\mathbf{Y}}_i\in \mathbb{B})}\hfill \\ {\displaystyle =P({\mathbf{X}}_i\in \mathbb{A})P({\mathbf{Y}}_j\in \mathbb{B})+\frac{1}{n^{″}}\left\{P({\mathbf{X}}_i\in \mathbb{A},{\mathbf{Y}}_i\in \mathbb{B})-P({\mathbf{X}}_i\in \mathbb{A})P({\mathbf{Y}}_i\in \mathbb{B})\right\},}\hfill \end{array}$$

this implies that

$$\begin{array}{c}{\displaystyle |P(\tilde{\mathbf{X}}\in \mathbb{A},\tilde{\mathbf{Y}}\in \mathbb{B})-P(\tilde{\mathbf{X}}\in \mathbb{A})P(\tilde{\mathbf{Y}}\in \mathbb{B})|\le \frac{1}{n^{″}}\int \int |f_{XY}(\mathbf{x},\mathbf{y})-f_X(\mathbf{x})f_Y(\mathbf{y})|\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}.}\hfill \end{array}$$

(A4)

On the other hand, we know that $n^{″}=\beta n$, so we get

$$\begin{array}{c}{\displaystyle |P(\tilde{\mathbf{X}}\in \mathbb{A},\tilde{\mathbf{Y}}\in \mathbb{B})-P(\tilde{\mathbf{X}}\in \mathbb{A})P(\tilde{\mathbf{Y}}\in \mathbb{B})|\le \frac{1}{\beta n}\int \int |f_{XY}(\mathbf{x},\mathbf{y})-f_X(\mathbf{x})f_Y(\mathbf{y})|\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}.}\hfill \end{array}$$

(A5)

From (A5), we observe that when $\beta$ takes larger values the bound becomes tighter. So, if $n\to \infty$ such that $n^{″}$ also becomes large enough in a linked manner so that $\beta =\mathrm{constant}$ then the RHS in (A5) tends to zero. This implies that $\tilde{\mathbf{X}}$ and $\tilde{\mathbf{Y}}$ become independent when $n\to \infty$. □

An immediate result of Lemma A2 is the following:

Lemma A3.

For given random vector ${\mathbf{Z}}_n=({\mathbf{X}}_n,{\mathbf{Y}}_n)$ from joint density function $f_{XY}$ and with marginal density functions $f_X$ and $f_Y$ let ${\tilde{\mathfrak{Z}}}_{n^{″}}=\left\{{\tilde{\mathbf{z}}}_1,\cdots ,{\tilde{\mathbf{z}}}_{n^{″}}\right\}$ be realization of random vector $\tilde{\mathbf{Z}}$ as in Lemma A2 with parameter $\beta =n^{″}/n$. Then for given points of ${\tilde{\mathfrak{Z}}}_{n^{″}}$ at $\tilde{\mathbf{z}}=(\tilde{\mathbf{x}},\tilde{\mathbf{y}})$, we have

$$|f_{\tilde{Z}}(\tilde{\mathbf{x}},\tilde{\mathbf{y}})-f_X(\tilde{\mathbf{x}})f_Y(\tilde{\mathbf{y}})|=O\left({\displaystyle \frac{1}{\beta n}}\right).$$

(A6)

Now, we want to provide a proof of assertion (11). Consider two subsets ${\mathbf{Z}}_{n^{\prime }}^{\prime }$ and ${\tilde{\mathbf{Z}}}_{n^{″}}$ as described in Section 2.2. Assume that the components of sample ${\tilde{\mathbf{Z}}}_{n^{″}}$ follow density function ${\tilde{f}}_{\tilde{X}\tilde{Y}}$. Therefore by owing to Lemmas A2 and A3, when $n\to \infty$ then ${\tilde{f}}_{\tilde{X}\tilde{Y}}\to f_X{f}_Y$. Let $M_{n^{\prime }}$ and $N_{n^{″}}$ be Poisson variables with mean $n^{\prime }$ and $n^{″}$ independent of one another and $\left\{{\mathbf{Z}}_i^{\prime }\right\}$ and $\left\{{\tilde{\mathbf{Z}}}_j\right\}$. Assume two Poisson processes ${\mathfrak{Z}}_{n^{\prime }}^{\prime }=\left\{{\mathbf{Z}}_1^{\prime },\cdots ,{\mathbf{Z}}_{M_{n^{\prime }}}^{\prime }\right\}$ and ${\tilde{\mathfrak{Z}}}_{n^{″}}=\left\{{\tilde{\mathbf{Z}}}_1,\cdots ,{\tilde{\mathbf{Z}}}_{N_{n^{″}}}\right\}$, and denote the FR statistic ${\mathfrak{R}}_{n^{\prime },n^{″}}^{\prime }$ on these processes. Following the arguments in [13,46] we shall prove the following:

$${\displaystyle \frac{\mathbb{E}\left[{\mathfrak{R}}_{n^{\prime },n^{″}}^{\prime }\right]}{n^{\prime }+n^{″}}\to 2\alpha \beta \int \int \frac{f_{X,Y}(\mathbf{x},\mathbf{y})f_X(\mathbf{x})f_Y(\mathbf{y})}{\alpha f_{XY}(\mathbf{x},\mathbf{y})+\beta f_X(\mathbf{x})f_Y(\mathbf{y})}\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}.}$$

This follows due to $|{\mathfrak{R}}_{n^{\prime },n^{″}}^{\prime }-{\mathfrak{R}}_{n^{\prime },n^{″}}|\le K_d\left(|M_{n^{\prime }}-n^{\prime }|+|N_{n^{″}}-n^{″}|\right)$, where $K_d$ is a constant defined in Lemma 1, [13] and $n^{\prime }+n^{″}=n$. Thus, ${(n^{\prime }+n^{″})}^{-1}\mathbb{E}|{\mathfrak{R}}_{n^{\prime },n^{″}}^{\prime }-{\mathfrak{R}}_{n^{\prime },n^{″}}|\to 0$ as $n\to \infty$. Let ${\mathbf{W}}_1^{n^{\prime },n^{″}},{\mathbf{W}}_2^{n^{\prime },n^{″}},\cdots$ be independent variables with common density

$${\varphi }_{n^{\prime },n^{″}}(\mathbf{x},\mathbf{y})=\left(n^{\prime }{f}_{XY}(\mathbf{x},\mathbf{y})+n^{″}{\tilde{f}}_{\tilde{X}\tilde{Y}}(\mathbf{x},\mathbf{y})\right)/(n^{\prime }+n^{″}),$$

for $(\mathbf{x},\mathbf{y})\in {\mathbb{R}}^d\times {\mathbb{R}}^d$. Let $L_{n^{\prime },n^{″}}$ be an independent Poisson variable with mean $n^{\prime }+n^{″}$. Let ${\mathfrak{F}}_{n^{\prime },n^{″}}^{\prime }=\left\{{\mathbf{W}}_1^{n^{\prime },n^{″}},\cdots ,{\mathbf{W}}_{L_{n^{\prime },n^{″}}}^{n^{\prime },n^{″}}\right\}$ a non-homogeneous Poisson process of rate $n^{\prime }{f}_{XY}+n^{″}{\tilde{f}}_{\tilde{X}\tilde{Y}}$. Assign mark 1 to a point in ${\mathfrak{F}}_{n^{\prime },n^{″}}^{\prime }$ with probability

$$n^{\prime }{f}_{XY}(\mathbf{x},\mathbf{y})/\left(n^{\prime }{f}_{XY}(\mathbf{x},\mathbf{y})+n^{″}{\tilde{f}}_{\tilde{X}\tilde{Y}}(\mathbf{x},\mathbf{y})\right),$$

and mark 2 otherwise. By the marking theorem [13,47], the FR test statistic ${\tilde{\mathfrak{R}}}_{n^{\prime },n^{″}}$ has the same distribution as ${\mathfrak{R}}_{n^{\prime },n^{″}}^{\prime }$. Given points of ${\mathfrak{F}}_{n^{\prime },n^{″}}^{\prime }$ at ${\mathbf{z}}^{\prime }=({\mathbf{x}}^{\prime },{\mathbf{y}}^{\prime })$ and ${\mathbf{z}}^{″}=({\mathbf{x}}^{″},{\mathbf{y}}^{″})$, the probability that they have different marks is given by (A7).

$${\displaystyle g_{n^{\prime },n^{″}}({\mathbf{z}}^{\prime },{\mathbf{z}}^{″})=\frac{n^{\prime }{f}_{XY}({\mathbf{x}}^{\prime },{\mathbf{y}}^{\prime })n^{″}{\tilde{f}}_{\tilde{X},\tilde{Y}}({\mathbf{x}}^{″},{\mathbf{y}}^{″})+n^{″}{\tilde{f}}_{\tilde{X},\tilde{Y}}({\mathbf{x}}^{\prime },{\mathbf{y}}^{\prime })n^{\prime }{f}_{XY}({\mathbf{x}}^{″},{\mathbf{y}}^{″})}{\left(n^{\prime }{f}_{XY}({\mathbf{x}}^{\prime },{\mathbf{y}}^{\prime })+n^{″}{\tilde{f}}_{\tilde{X},\tilde{Y}}({\mathbf{x}}^{\prime },{\mathbf{y}}^{\prime })\right)\left(n^{\prime }{f}_{XY}({\mathbf{x}}^{″},{\mathbf{y}}^{″})+n^{″}{\tilde{f}}_{\tilde{X},\tilde{Y}}({\mathbf{x}}^{″},{\mathbf{y}}^{″})\right)},}$$

(A7)

define

$${\displaystyle g({\mathbf{z}}^{\prime },{\mathbf{z}}^{″})=\frac{\alpha \beta \left(f_{XY}({\mathbf{x}}^{\prime },{\mathbf{y}}^{\prime })f_X({\mathbf{x}}^{″})f_Y({\mathbf{y}}^{″})+f_X({\mathbf{x}}^{\prime })f_Y({\mathbf{y}}^{\prime })f_{XY}({\mathbf{x}}^{″},{\mathbf{y}}^{″})\right)}{\left(\alpha f_{XY}({\mathbf{x}}^{″},{\mathbf{y}}^{″})+\beta f_X({\mathbf{x}}^{″})f_Y({\mathbf{y}}^{″})\right)\left(\alpha f_{XY}({\mathbf{x}}^{\prime },{\mathbf{y}}^{\prime })+\beta f_X({\mathbf{x}}^{\prime })f_Y({\mathbf{y}}^{\prime })\right)},}$$

(A8)

then

$$\begin{array}{c}{\displaystyle \mathbb{E}\left[{\tilde{\mathfrak{R}}}_{n^{\prime },n^{″}}^{\prime }|{\mathfrak{F}}_{n^{\prime },n^{″}}^{\prime }\right]=\underset{i<j\le L_{n^{\prime },n^{″}}}{\sum \sum }{g}_n({\mathbf{W}}_i^{n^{\prime },n^{″}},{\mathbf{W}}_j^{n^{\prime },n^{″}}){\mathbf{I}}_{{\mathfrak{F}}_{n^{\prime },n^{″}}^{\prime }}({\mathbf{W}}_i^{n^{\prime },n^{″}},{\mathbf{W}}_j^{n^{\prime },n^{″}}).}\hfill \end{array}$$

(A9)

Now recall (A8). We observe that $g_{n^{\prime },n^{″}}({\mathbf{z}}^{\prime },{\mathbf{z}}^{″})\to g({\mathbf{z}}^{\prime },{\mathbf{z}}^{″})$. Going back to (A9), we can write

$$\begin{array}{c}{\displaystyle \mathbb{E}\left[{\tilde{\mathfrak{R}}}_{n^{\prime },n^{″}}^{\prime }\right]=\underset{i<j\le L_{n^{\prime },n^{″}}}{\sum \sum }{g}_{n^{\prime },n^{″}}({\mathbf{W}}_i^{n^{\prime },n^{″}},{\mathbf{W}}_j^{n^{\prime },n^{″}}){\mathbf{I}}_{{\mathfrak{F}}_{n^{\prime },n^{″}}^{\prime }}({\mathbf{W}}_i^{n^{\prime },n^{″}},{\mathbf{W}}_j^{n^{\prime },n^{″}})+o(n^{\prime }+n^{″}).}\hfill \end{array}$$

(A10)

For fixed $n^{\prime }$, $n^{″}$ consider the collection:

$${\mathfrak{F}}_{n^{\prime },n^{″}}=\left\{{\mathbf{W}}_1^{n^{\prime },n^{″}},\cdots ,{\mathbf{W}}_{n^{\prime },n^{″}}^{n^{\prime }+n^{″}}\right\}.$$

By the fact that $\mathbb{E}\left[M_{n^{\prime }}+N_{n^{″}}-(n^{\prime }+n^{″})\right]=o(n^{\prime }+n^{″})$, we have

$$\begin{array}{c}{\displaystyle \mathbb{E}\left[{\tilde{\mathfrak{R}}}_{n^{\prime },n^{″}}^{\prime }\right]=\underset{i<j\le n^{\prime }+n^{″}}{\sum \sum }{g}_{n^{\prime },n^{″}}({\mathbf{W}}_i^{n^{\prime },n^{″}},{\mathbf{W}}_j^{n^{\prime },n^{″}}){\mathbf{I}}_{{\mathfrak{F}}_{n^{\prime },n^{″}}}({\mathbf{W}}_i^{n^{\prime }},{\mathbf{W}}_j^{n^{″}})+o(n^{\prime }+n^{″}).}\hfill \end{array}$$

(A11)

Introduce

$$\varphi (\mathbf{x},\mathbf{y})=\alpha f_{XY}(\mathbf{x},\mathbf{y})+\beta f_X(\mathbf{x})f_Y(\mathbf{y}).$$

Then ${\varphi }_{n^{\prime },n^{″}}(\mathbf{x},\mathbf{y})\to \varphi (\mathbf{x},\mathbf{y})$ uniformly as ${\displaystyle n^{\prime }/n\to \alpha }$ and ${\displaystyle n^{″}/n\to \beta }$. Thus, using Proposition 1 in [13], we get

$$\begin{array}{c}{\displaystyle \frac{\mathbb{E}\left[{\tilde{\mathfrak{R}}}_{n^{\prime },n^{″}}^{\prime }\right]}{n}\to \int g(\mathbf{z},\mathbf{z})\varphi (\mathbf{z})\mathrm{d}\mathbf{z}=\int \int \frac{2\alpha \beta f_{XY}(\mathbf{x},\mathbf{y})f_X(\mathbf{x})f_Y(\mathbf{y})}{\alpha f_{XY}(\mathbf{x},\mathbf{y})+\beta f_X(\mathbf{x})f_Y(\mathbf{y})}\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}.}\hfill \end{array}$$

(A12)

 □

Appendix A.5. Theorem 4

Proof. 

We begin by providing a family of bias rate bounds for the FR test statistic ${\mathfrak{R}}_{n^{\prime },n^{″}}$ in terms of a parameter l. Assume $f_{XY}$, $f_X$, and $f_Y$ are in ${\mathsf{\Sigma }}_d(\eta ,K)$. Then by plugging the optimal l, we prove the bias rate bound given in (13).

Theorem A1.

Let ${\mathfrak{R}}_{n^{\prime },n^{″}}:=\mathfrak{R}({\mathfrak{Z}}_{n^{\prime }},{\mathfrak{Z}}_{n^{″}})$ be the FR test statistic. Then a bound on the bias rate of the ${\mathfrak{R}}_{n^{\prime },n^{″}}$ estimator for $0<\eta \le 1$, $d\ge 2$ is given by

$$\begin{array}{c}{\displaystyle |\frac{\mathbb{E}\left[{\mathfrak{R}}_{n^{\prime },n^{″}}\right]}{n}-2\alpha \beta \int \int \frac{f_{XY}(\mathbf{x},\mathbf{y})f_X(\mathbf{x})f_Y(\mathbf{y})}{\alpha f_{XY}(\mathbf{x},\mathbf{y})+\beta f_X(\mathbf{x})f_Y(\mathbf{y})}\mathrm{d}\mathbf{x}|}\hfill \\ \le O\left(l^d{(n)}^{-\eta /d}\right)+O\left(l^{-d\eta }\right)+O\left(l^d{\beta }^{-1}{n}^{-1}\right)+O\left(c_d{n}^{-1}\right),\hfill \end{array}$$

(A13)

where $0<\eta \le 1$ is the Hölder smoothness parameter and $c_d$ is the largest possible degree of any vertex of MST.

Set

$$\begin{array}{c}{\alpha }_i=\alpha n{a}_i{l}^d\left(1-a_i{l}^{-d}\right)+{\left(\alpha n\right)}^2{a}_i^2,\\ {\beta }_i=\beta n{b}_i{l}^d\left(1-b_i{l}^{-d}\right)+{\left(\beta n\right)}^2{b}_i^2.\end{array}$$

and

$${\displaystyle {\mathcal{A}}_{f,n}^{\beta ,\alpha }(\mathbf{x},\mathbf{y})=\frac{2f_{XY}(\mathbf{x},\mathbf{y})\left(f_X(\mathbf{x})f_Y(\mathbf{y})+{\delta }_f/(\beta n)\right)\left(f_{XY}(\mathbf{x},\mathbf{y})\sqrt{\alpha }+(f_X(\mathbf{x})f_Y(\mathbf{y})+{\delta }_f/(\beta n))\sqrt{\beta }\right)}{a_i^2{l}^{-d}{\left(\alpha f_{XY}(\mathbf{x},\mathbf{y})+\beta \left(f_X(\mathbf{x})f_Y(\mathbf{y})+{\delta }_f/(\beta n)\right)\right)}^2},}$$

(A14)

where

$${\displaystyle {\delta }_f=\int \int |f_{XY}(\mathbf{x},\mathbf{y})-f_X(\mathbf{x})f_Y(\mathbf{y})|\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y},}$$

(A15)

A more explicit form for the bound on the RHS is given below:

$$\begin{array}{c}{\displaystyle \Delta (\alpha ,f_{XY},f_X{f}_Y):=c_2{l}^d{(n)}^{-1}+c_d{2}^d{(n)}^{-1}+O\left(l^d{(n)}^{-\eta /d}\right)+O\left(l^d{(n)}^{-1/2}\right)}\\ {\displaystyle +O\left(c_d{(n)}^{-1/2}\right)+2c_1{l}^{d-1}{(n)}^{(1/d)-1}+{\delta }_f({(\beta n)}^{-1})\int \int \frac{2\alpha \beta f_{XY}(\mathbf{x},\mathbf{y})}{\alpha f_{XY}(\mathbf{x},\mathbf{y})+\beta f_X(\mathbf{x})f_Y(\mathbf{y})}\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}}\\ {\displaystyle +{(n)}^{-1}\sum _{i=1}^{M}2\int \int f_{XY}(\mathbf{x},\mathbf{y})\left(f_X(\mathbf{x})f_Y(\mathbf{y})+{\delta }_f/(\beta n)\right)({\alpha }_i{\beta }_i(\alpha n{a}_i{l}^{-d}{f}_{XY}^2(\mathbf{x},\mathbf{y})}\\ +\beta n{b}_i{l}^{-d}{\left(f_X(\mathbf{x})f_Y(\mathbf{y})+{\delta }_f/(\beta n)\right)}^2){)}^{1/2}/{\left(\alpha n{a}_i{f}_{XY}(\mathbf{x},\mathbf{y})+\beta n{b}_i{f}_X(\mathbf{x})f_Y(\mathbf{y})\right)}^2\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}\\ {\displaystyle +{(n)}^{-1}\sum _{i=1}^{M}O(l)\int \int l^d{(a_i)}^{-1}\frac{2f_{XY}(\mathbf{x},\mathbf{y})\left(f_X(\mathbf{x})f_Y(\mathbf{y})+{\delta }_f/(\beta n)\right)}{\alpha f_{XY}(\mathbf{x},\mathbf{y})+\beta f_X(\mathbf{x})f_Y(\mathbf{y})}\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}+O\left(l^{-d\eta }\right)}\\ {\displaystyle +{(n)}^{-3/2}\sum _{i=1}^{M}O(l)\int \int l^{-d/2}\sqrt{b_i}{\mathcal{A}}_{f,n}^{\beta ,\alpha }(\mathbf{x},\mathbf{y})\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}.}\end{array}$$

(A16)

Proof. Consider two Poisson variables $M_{n^{\prime }}$ and $N_{n^{″}}$ with mean $n^{\prime }$ and $n^{″}$ respectively and independent of one another and $\left\{{\mathbf{Z}}_i^{\prime }\right\}$ and $\left\{{\tilde{\mathbf{Z}}}_j\right\}$. Let ${\mathfrak{Z}}_{n^{\prime }}^{\prime }$ and ${\tilde{\mathfrak{Z}}}_{n^{″}}$ be the Poisson processes $\{{\mathbf{Z}}_1^{\prime },\cdots ,{\mathbf{Z}}_{M_{n^{\prime }}}^{\prime }\}$ and $\{{\tilde{\mathbf{Z}}}_1,\cdots ,{\tilde{\mathbf{Z}}}_{N_{n^{″}}}\}$. Likewise Appendix A.4, set ${\mathfrak{R}}_{n^{\prime },n^{″}}^{\prime }=\mathfrak{R}({\mathfrak{Z}}_{n^{\prime }}^{\prime },{\tilde{\mathfrak{Z}}}_{n^{″}})$. Applying Lemma 1, and (12) in [13], we can write

$$|{\mathfrak{R}}_{n^{\prime },n^{″}}^{\prime }-{\mathfrak{R}}_{n^{\prime },n^{″}}|\le c_d(|M_{n^{\prime }}-n^{\prime }|+|N_{n^{″}}-n^{″}|).$$

(A17)

Here $c_d$ denotes the largest possible degree of any vertex of the MST in ${\mathbb{R}}^d$. Following the arguments in [46], we have $\mathbb{E}\left[|M_{n^{\prime }}-n^{\prime }|\right]=O\left({n^{\prime }}^{1/2}\right)$ and $\mathbb{E}\left[|N_{n^{″}}-n^{″}|\right]=O\left({n^{″}}^{1/2}\right)$. Hence

$${\displaystyle \frac{\mathbb{E}\left[{\mathfrak{R}}_{n^{\prime },n^{″}}\right]}{n^{\prime }+n^{″}}=\frac{\mathbb{E}\left[{\mathfrak{R}}_{n^{\prime },n^{″}}^{\prime }\right]}{n^{\prime }+n^{″}}+O\left(c_d{(n^{\prime }+n^{″})}^{-1/2}\right).}$$

(A18)

Next let $n_i^{\prime }$ and $n_i^{″}$ be independent binomial random variables with marginal densities $B(n^{\prime },a_i{l}^{-d})$ and $B(n^{″},b_i{l}^{-d})$ such that $a_i,b_i$ are non-negative constants $a_i\le b_i$ and ${\displaystyle \sum _{i=1}^{l^d}{a}_i{l}^{-d}=\sum _{i=1}^{l^d}{b}_i{l}^{-d}=1}$. Therefore, using the subadditivity property in Lemma 2.2, [46], we can write

$$\begin{array}{c}{\displaystyle \mathbb{E}\left[{\mathfrak{R}}_{n^{\prime },n^{″}}^{\prime }\right]\le \sum _{i=1}^M\mathbb{E}\left[\mathbb{E}\left[{\mathfrak{R}}_{n_i^{\prime },n_i^{″}}^{\prime }|n_i^{\prime },n_i^{″}\right]\right]+2c_1{l}^{d-1}{(n^{\prime }+n^{″})}^{1/d},}\hfill \end{array}$$

(A19)

where $M=l^d$, and $\eta >0$ is the Hölder smoothness parameter. Furthermore, for given $n_i^{\prime }$, $n_i^{″}$, let ${\mathbf{W}}_1^{n_i^{\prime },n_i^{″}},{\mathbf{W}}_2^{n_i^{\prime },n_i^{″}},\cdots$ be independent variables with common densities for $(\mathbf{x},\mathbf{y})\in {\mathbb{R}}^d\times {\mathbb{R}}^d$:

$$g_{n_i^{\prime },n_i^{″}}(\mathbf{x},\mathbf{y})=\left(n_i^{\prime }{f}_{XY}(\mathbf{x},\mathbf{y})+n_i^{″}{\tilde{f}}_{\tilde{X}\tilde{Y}}(\mathbf{x},\mathbf{y})\right)/(n_i^{\prime }+n_i^{″}).$$

Denote $L_{n_i^{\prime },n_i^{″}}$ be an independent Poisson variable with mean $n_i^{\prime }+n_i^{″}$ and ${\mathfrak{F}}_{n_i^{\prime },n_i^{″}}^{\prime }=\left\{{\mathbf{W}}_1^{n_i^{\prime },n_i^{″}},\cdots ,{\mathbf{W}}_{L_{n_i^{\prime }.n_i^{″}}}^{n_i^{\prime },n_i^{″}}\right\}$ a non-homogeneous Poisson of rate $n_i^{\prime }{f}_{XY}+n_i^{″}{\tilde{f}}_{\tilde{X}\tilde{Y}}$. Let ${\mathfrak{F}}_{n_i^{\prime },n_i^{″}}$ be the non-Poisson point process $\left\{{\mathbf{W}}_1^{n_i^{\prime },n_i^{″}},\cdots {\mathbf{W}}_{n_i^{\prime }+n_i^{″}}^{n_i^{\prime },n_i^{″}}\right\}$. Assign a mark from the set $\{1,2\}$ to each point of ${\mathfrak{F}}_{n_i^{\prime },n_i^{″}}^{\prime }$. Let ${\tilde{\mathfrak{Z}}}_{n_i^{\prime }}^{\prime }$ be the sets of points marked 1 with each probability $n_i^{\prime }{f}_{XY}(\mathbf{x},\mathbf{y})/\left(n_i^{\prime }{f}_{XY}(\mathbf{x},\mathbf{y})+n_i^{″}{\tilde{f}}_{\tilde{X}\tilde{Y}}(\mathbf{x},\mathbf{y})\right)$ and let ${\tilde{\mathfrak{Z}}}_{n_i^{″}}^{″}$ be the set points with mark 2. Please note that owing to the marking theorem [47], ${\tilde{\mathfrak{Z}}}_{n_i^{\prime }}^{\prime }$ and ${\tilde{\mathfrak{Z}}}_{n_i^{″}}^{″}$ are independent Poisson processes with the same distribution as ${\mathfrak{Z}}_{n_i^{\prime }}^{\prime }$ and ${\tilde{\mathfrak{Z}}}_{n_i^{″}}$, respectively. Considering ${\tilde{\mathfrak{R}}}_{n_i^{\prime },n_i^{″}}^{\prime }$ as FR test statistic on nodes in ${\tilde{\mathfrak{Z}}}_{n_i^{\prime }}^{\prime }\cup {\tilde{\mathfrak{Z}}}_{n_i^{″}}^{″}$, we have

$$\mathbb{E}\left[{\mathfrak{R}}_{n_i^{\prime },n_i^{″}}^{\prime }|n_i^{\prime },n_i^{″}\right]=\mathbb{E}\left[{\tilde{\mathfrak{R}}}_{n_i^{\prime },n_i^{″}}^{\prime }|n_i^{\prime },n_i^{″}\right].$$

By the fact that $\mathbb{E}\left[|M_{n^{\prime }}+N_{n^{″}}-n^{\prime }-n^{″}|\right]=O({(n^{\prime }+n^{″})}^{1/2})$, we have

$$\begin{array}{c}{\displaystyle \mathbb{E}\left[{\tilde{\mathfrak{R}}}_{n_i^{\prime },n_i^{″}}^{\prime }|n_i^{\prime },n_i^{″}\right]=\mathbb{E}\left[\mathbb{E}\left[{\tilde{\mathfrak{R}}}_{n_i^{\prime },n_i^{″}}^{\prime }|{\mathfrak{F}}_{n_i^{\prime },n_i^{″}}^{\prime }\right]\right]}\hfill \\ {\displaystyle =\mathbb{E}\left[{\displaystyle \underset{s<j<n_i^{\prime }+n_i^{″}}{\sum \sum }{P}_{n_i^{\prime },n_i^{″}}({\mathbf{W}}_s^{n_i^{\prime },n_i^{″}},{\mathbf{W}}_j^{n_i^{\prime },n_i^{″}})\mathbf{1}\left\{({\mathbf{W}}_s^{n_i^{\prime },n_i^{″}},{\mathbf{W}}_j^{n_i^{\prime },n_i^{″}})\in {\mathfrak{F}}_{n_i^{\prime },n_i^{″}}\right\}}\right]+O\left({(n_i^{\prime }+n_i^{″})}^{1/2})\right).}\hfill \end{array}$$

Here ${\mathbf{z}}^{\prime }=({\mathbf{x}}^{\prime },{\mathbf{y}}^{\prime })$, ${\mathbf{z}}^{″}=({\mathbf{x}}^{″},{\mathbf{y}}^{″})$, and $P_{n_i^{\prime },n_i^{″}}({\mathbf{z}}^{\prime },{\mathbf{z}}^{″})$ is given in below:

$$\begin{array}{c}{\displaystyle P_{n_i^{\prime },n_i^{″}}({\mathbf{z}}^{\prime },{\mathbf{z}}^{″}):=P_r\left\{\mathrm{mark}{\mathbf{z}}^{\prime }\ne \mathrm{mark}{\mathbf{z}}^{″},({\mathbf{z}}^{\prime },{\mathbf{z}}^{″})\in {\mathfrak{F}}_{n_i^{\prime },n_i^{″}}\right\}}\hfill \\ {\displaystyle =\frac{n_i^{\prime }{f}_{XY}({\mathbf{x}}^{\prime },{\mathbf{y}}^{\prime })n_i^{″}{\tilde{f}}_{\tilde{X}\tilde{Y}}({\mathbf{x}}^{″},{\mathbf{y}}^{″})+n_i^{\prime }{\tilde{f}}_{\tilde{X}\tilde{Y}}({\mathbf{x}}^{\prime },{\mathbf{y}}^{\prime })n_i^{″}{f}_{X,Y}({\mathbf{x}}^{″},{\mathbf{y}}^{″})}{\left(n_i^{″}{f}_{XY}({\mathbf{x}}^{\prime },{\mathbf{y}}^{\prime })+n_i^{″}{\tilde{f}}_{\tilde{X}\tilde{Y}}({\mathbf{x}}^{\prime },{\mathbf{y}}^{\prime })\right)\left(n_1^{\prime }{f}_{XY}({\mathbf{x}}^{″},{\mathbf{y}}^{″})+n_i^{″}{\tilde{f}}_{\tilde{X}\tilde{Y}}({\mathbf{x}}^{″},{\mathbf{y}}^{″})\right)}.}\hfill \end{array}$$

Next set

$$\begin{array}{c}{\alpha }_i=n^{\prime }{a}_i{l}^d(1-a_i{l}^{-d})+{n^{\prime }}^2{a}_i^2,{\beta }_i=n^{″}{b}_i{l}^d(1-b_i{l}^{-d})+{n^{″}}^2{b}_i^2.\end{array}$$

By owing to the Lemma B.6 in [46] and applying analogous arguments, we can rewrite the expression in (A20):

$$\begin{array}{c}{\displaystyle \mathbb{E}\left[{{\mathfrak{R}}^{\prime }}_{n^{\prime },n^{″}}\right]\le \sum _{i=1}^M{a}_i{b}_i{l}^{-d}\int \int \frac{2n^{\prime }{n}^{″}{f}_{XY}(\mathbf{x},\mathbf{y})f_{\tilde{X}\tilde{Y}}(\mathbf{x},\mathbf{y})}{n^{\prime }{a}_i{f}_{X,Y}(\mathbf{x},\mathbf{y})+n^{″}{b}_i{f}_{\tilde{X}\tilde{Y}}(\mathbf{x},\mathbf{y})}\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}+2c_1{l}^{d-1}{(n^{\prime }+n^{″})}^{1/d}}\hfill \\ {\displaystyle +\sum _{i=1}^{M}2\int \int \frac{f_{XY}(\mathbf{x},\mathbf{y})f_{\tilde{X}\tilde{Y}}(\mathbf{x},\mathbf{y}){\left({\alpha }_i{\beta }_i\left(n^{\prime }{a}_i{l}^{-d}{f}_{XY}^2(\mathbf{x},\mathbf{y})+n^{″}{b}_i{l}^{-d}{f}_{\tilde{X}\tilde{Y}}^2(\mathbf{x},\mathbf{y})\right)\right)}^{1/2}}{{\left(n^{\prime }{a}_i{f}_{XY}(\mathbf{x},\mathbf{y})+n^{″}{b}_i{f}_{\tilde{X}\tilde{Y}}(\mathbf{x},\mathbf{y})\right)}^2}\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}}\hfill \\ {\displaystyle +\sum _{i=1}^M{\mathbb{E}}_{n_i^{\prime },n_i^{″}}\left[(n_i^{\prime }+n_i^{″}){\varsigma }_{\eta }(l,n_i^{\prime },n_i^{″})\right]+O\left(l^d{(n^{\prime }+n^{″})}^{1-\eta /d}\right)+O\left(l^d{(n^{\prime }+n^{″})}^{1/2}\right),}\hfill \end{array}$$

(A20)

where

$${\displaystyle {\varsigma }_{\eta }(l,n_i^{\prime },n_i^{″})=\left(O\left(\frac{l}{n_i^{\prime }+n_i^{″}}\right)-\frac{2l^d}{n_i^{\prime }+n_i^{″}}\right)\int g_{n_i^{\prime },n_i^{″}}({\mathbf{z}}^{\prime })P_{n_i^{\prime },n_i^{″}}({\mathbf{z}}^{\prime },{\mathbf{z}}^{\prime })\mathrm{d}{\mathbf{z}}^{\prime }+O(l^{-d\eta }).}$$

Going back to Lemma A3, we know that

$$f_{\tilde{X}\tilde{Y}}(\mathbf{x},\mathbf{y})=f_X(\mathbf{x})f_Y(\mathbf{y})+O\left({\displaystyle \frac{1}{\beta n}}\right).$$

Therefore, the first term on the RHS of (A20) is less and equal to

$$\begin{array}{c}{\displaystyle \sum _{i=1}^M{a}_i{b}_i{l}^{-d}\int \int \frac{2n^{\prime }{n}^{″}{f}_{XY}(\mathbf{x},\mathbf{y})f_X(\mathbf{x})f_Y(\mathbf{y})}{n^{\prime }{a}_i{f}_{X,Y}(\mathbf{x},\mathbf{y})+n^{″}{b}_i{f}_X(\mathbf{x})f_Y(\mathbf{y})}\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}}\hfill \\ {\displaystyle +\left({\displaystyle \frac{{\delta }_f}{\beta n}}\right)\sum _{i=1}^M{a}_i{b}_i{l}^{-d}\int \int \frac{2n^{\prime }{n}^{″}{f}_{XY}(\mathbf{x},\mathbf{y})}{n^{\prime }{a}_i{f}_{XY}(\mathbf{x},\mathbf{y})+n^{″}{b}_i{f}_X(\mathbf{x})f_Y(\mathbf{y})}\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y},}\hfill \end{array}$$

and the second term is less and equal to

$$\begin{array}{c}{\displaystyle \sum _{i=1}^{M}2\int \int f_{XY}(\mathbf{x},\mathbf{y})\left(f_X(\mathbf{x})f_Y(\mathbf{y})+{\delta }_f/(\beta n)\right)({\alpha }_i{\beta }_i(n^{\prime }{a}_i{l}^{-d}{f}_{XY}^2(\mathbf{x},\mathbf{y})+n^{″}{b}_i{l}^{-d}(f_X(\mathbf{x})f_Y(\mathbf{y})}\hfill \\ {\displaystyle +{\delta }_f/(\beta n){)}^2){)}^{1/2}/{\left(n^{\prime }{a}_i{f}_{XY}(\mathbf{x},\mathbf{y})+n^{″}{b}_i{f}_X(\mathbf{x})f_Y(\mathbf{y})\right)}^2\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y},}\hfill \end{array}$$

where

$${\delta }_f=\int \int |f_{XY}(\mathbf{x},\mathbf{y})-f_X(\mathbf{x})f_Y(\mathbf{y})|\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}.$$

Recall the definition of the dual MST and FR statistic denoted by ${\mathfrak{R}}_{n^{\prime },n^{″}}^{*}$ following [46]:

Definition A1.

(Dual MST, ${\mathrm{MST}}^{*}$ and dual FR statistic ${\mathfrak{R}}_{m,n}^{*}$) Let ${\mathbb{F}}_i$ be the set of corner points associated with a particular subsection $Q_i$, $1\le i\le l^d$ of ${[0,1]}^d$. Define the dual ${\mathrm{MST}}^{*}({\mathfrak{X}}_m\cup {\mathfrak{Y}}_n\cap Q_i)$ as the boundary MST graph in partition $Q_i$ [48], which contains ${\mathfrak{X}}_m$ and ${\mathfrak{Y}}_n$ points falling inside partition cell $Q_i$ and those corner points in ${\mathbb{F}}_i$ which minimize total MST length. Please note that it is allowed to connect the MSTs in $Q_i$ and $Q_j$ through points strictly contained in $Q_i$ and $Q_j$ and therefore corner points. Thus, the dual MST can connect the points in $Q_i\cup Q_j$ by direct edges to pair to another point in $Q_i\cup Q_j$ or by passing through the corner the corner points which are all connected in order to minimize the total weights. To clarify, assume that there are two points in $Q_i\cup Q_j$, then the dual MST consists of the two edges connecting these points to the corner if they are closer to a corner point otherwise the dual MST connects them to each other.

Furthermore, ${\mathfrak{R}}_{m,n}^{*}({\mathfrak{X}}_m,{\mathfrak{Y}}_n\cap Q_i)$ is defined as the number of edges in the ${\mathrm{MST}}^{*}$ graph connecting nodes from different samples and number of edges connecting to the corner points. Please note that the edges connected to the corner nodes (regardless of the type of points) are always counted in the dual FR test statistic ${\mathfrak{R}}_{m,n}^{*}$.

Similarly, consider the Poisson processes samples and the FR test statistic over these samples, denoted by ${{\mathfrak{R}}^{\prime }}_{n^{\prime },n^{″}}^{*}$. By superadditivity of the dual ${\mathfrak{R}}_{n^{\prime },n^{″}}^{*}$ [46], we have

$$\begin{array}{c}{\displaystyle \mathbb{E}\left[{\mathfrak{R}}_{n^{\prime },n^{″}}^{\prime *}\right]\ge \sum _{i=1}^M{a}_i{l}^{-d}\int \int \frac{2n^{\prime }{n}^{″}{f}_{XY}(\mathbf{x},\mathbf{y})\left(f_X(\mathbf{x})f_Y(\mathbf{y})-{\delta }_f/(\beta n)\right)}{n^{\prime }{f}_{XY}(\mathbf{x},\mathbf{y})+n^{″}\left(f_X(\mathbf{x})f_Y(\mathbf{y})-{\delta }_f/(\beta n)\right)}\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}}\hfill \\ {\displaystyle -\sum _{i=1}^M{\mathbb{E}}_{n_i^{\prime },n_i^{″}}\left[(n_i^{\prime }+n_i^{″}){\varsigma }_{\eta }(l,n_i^{\prime },n_i^{″})\right]-O\left(l^d{(n^{\prime }+n^{″})}^{1-\eta /d}\right)-O\left(l^d{(n^{\prime }+n^{″})}^{1/2}\right)-c_2{l}^d.}\hfill \end{array}$$

(A21)

The first term of RHS in (A21) is greater or equal to

$$\begin{array}{c}{\displaystyle \int \int \frac{2n^{\prime }{n}^{″}{f}_{XY}(\mathbf{x},\mathbf{y})f_X(\mathbf{x})f_Y(\mathbf{y})}{n^{\prime }{f}_{XY}(\mathbf{x},\mathbf{y})+n^{″}{f}_X(\mathbf{x})f_Y(\mathbf{y})}\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}-\frac{{\delta }_f}{\beta n}\int \int \frac{2n^{\prime }{n}^{″}{f}_{XY}(\mathbf{x},\mathbf{y})}{n^{\prime }{f}_{XY}(\mathbf{x},\mathbf{y})+n^{″}{f}_X(\mathbf{x})f_Y(\mathbf{y})}\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}.}\hfill \end{array}$$

Furthermore,

$${\displaystyle \frac{\mathbb{E}\left[{\mathfrak{R}}_{n^{\prime },n^{″}}^{\prime }\right]}{n}+\frac{c_d{2}^d}{n}\ge \frac{\mathbb{E}\left[{\mathfrak{R}}_{n^{\prime },n^{″}}^{\prime *}\right]}{n},}$$

where $c_d$ is the largest possible degree of any vertex of the MST in ${\mathbb{R}}^d$, as before. Consequently, we have

$$\begin{array}{c}{\displaystyle |\frac{\mathbb{E}\left[{\mathfrak{R}}_{n^{\prime },n^{″}}^{\prime }\right]}{n}-\int \int \frac{2\alpha \beta f_{XY}(\mathbf{x},\mathbf{y})f_X(\mathbf{x})f_Y(\mathbf{y})}{\alpha f_{XY}(\mathbf{x},\mathbf{y})+\beta f_X(\mathbf{x})f_Y(\mathbf{y})}\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}|\le \mathcal{B}(\alpha ,f_{XY},f_X{f}_Y),}\hfill \end{array}$$

(A22)

where $\mathcal{B}$ is defined in (A16) and ${\mathcal{A}}_{f,n}^{\beta ,\alpha }(\mathbf{x},\mathbf{y})$ has been introduced in (A14). The last line in (A22) follows from the fact that

$$\begin{array}{c}{\displaystyle \sum _{i=1}^M{\mathbb{E}}_{n_i^{\prime },n_i^{″}}\left[(n_i^{\prime }+n_i^{″}){\varsigma }_{\eta }(l,n_i^{\prime },n_i^{″})\right]\le \sum _{i=1}^{M}O(l)\int \int l^{-d/2}\sqrt{b_i}{\mathcal{A}}_{f,n}^{\beta ,n^{\prime }/n}(\mathbf{x},\mathbf{y})\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}}\hfill \\ {\displaystyle +\sum _{i=1}^{M}O(l)\int \int l^d{(a_i)}^{-1}\frac{2f_{XY}(\mathbf{x},\mathbf{y})\left(f_X(\mathbf{x})f_Y(\mathbf{y})+O({\delta }_f/(\beta n))\right)}{n^{\prime }{f}_{XY}(\mathbf{x},\mathbf{y})+n^{″}{f}_X(\mathbf{x})f_Y(\mathbf{y})}\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}.}\hfill \end{array}$$

Here ${\mathcal{A}}_{f,n}^{\beta ,n^{\prime }/n}(\mathbf{x},\mathbf{y})$ is given as (A14) by substituting $n^{\prime }/n$ in $\alpha$ such that $\beta =1-\alpha$. Hence, the proof of Theorem A1 is completed. □

Going back to the proof of (13), without loss of generality assume that $(n)l^{-d}>1$, for $d\ge 2$ and $0<\eta \le 1$. We select l as a function of n and $\beta$ to be the sequence increasing in n which minimizes the maximum of these rates:

$$l(n,\beta )=arg\underset{l}{min}max\left\{l^d{(n)}^{-\eta /d},l^{-\eta d},l^d{\beta }^{-1}{n}^{-1},c_d{2}^d{n}^{-1}\right\}.$$

(A23)

The solution $l=l(n,\beta )$ is obtained when $l^d{(n)}^{-\eta /d}=l^{-\eta d}$, or equivalently $l=\lfloor {(n)}^{\eta /(d^2(\eta +1))}\rfloor$ or when $l^d{\beta }^{-1}{n}^{-1}=l^{-\eta d}$, which implies $l=\lfloor {(\beta n)}^{1/(d(1+\eta ))}\rfloor$. Substitute this l in the bound (A13) to obtain the RHS expression in (13) for $d\ge 2$. □

Appendix A.6.

Our main goal in Section 2.4 was to find proportion $\alpha$ such that the parametric MSE rate depending on the joint density $f_{XY}$ and marginal densities $f_X,f_Y$ is minimized. Recalling the explicit bias bound in (A16), it can be seen that this function will be a complicated function of $f_{XY}$, $f_X{f}_Y$ and $\alpha$. By rearrangement of terms in (A13), we first find an upper bound for $\Delta$ in (A16), denoted by $\overline{\Delta }$, as follows:

$$\begin{array}{c}{\displaystyle \overline{\Delta }(\alpha ,f_{XY},f_X{f}_Y)=D(n,l_n,d,\eta )+\tilde{D}(n,l_n,d){\mathbb{E}}_{XY}\left[G_{f,n}^{\alpha ,\beta }(X,Y)\right],}\hfill \end{array}$$

(A24)

where $l_n:=⌊n^{\eta /(d^2(1+\eta ))}⌋$. From Appendix A.5 we know that optimal l is given by (A23). One can check that for $\alpha \le 1-n^{(\eta /d)-1}$, the optimal $l=⌊n^{\eta /(d^2(1+\eta ))}⌋$ provides a tighter bound. In (A24), the constants D and $\overline{D}$ are

$$\begin{array}{c}{\displaystyle D(n,l_n,d,\eta )=c_2{l}_n^d{n}^{-1}+c_d{2}^d{n}^{-1}+c^{\prime }{l}_n^d{n}^{-\eta /d}+c{l}_n^d{n}^{-1/d}+2c_1{l}_n^{d-1}{n}^{1/d-1}+c_3{l}_n^{-d\eta },}\hfill \end{array}$$

(A25)

$$\begin{array}{c}{\displaystyle \tilde{D}(n,l_n,d)=2+n^{-1}2c^{″}\sum _{i=1}^M{l}_n{l}_n^d{a}_i^{-1}+n^{-3/2}2c_1^{\prime }\sum _{i=1}^M{l}_n{l}_n^{d/2}\sqrt{b_i}{a}_i^2}\hfill \\ {\displaystyle +n^{-1}\sum _{i=1}^{M}2n^{-3/2}{l}_n^{-d/2}\frac{\sqrt{b_i}}{a_i^2}{\left(n{a}_i{l}_n^d+n^2{a}_i^2\right)}^{1/2}{\left(n{b}_i{l}_n^d+n^2{b}_i^2\right)}^{1/2}.}\hfill \end{array}$$

(A26)

And the function $G_{f,n}^{\alpha ,\beta }(\mathbf{x},\mathbf{y})$ is given as the following:

$$\begin{array}{c}{\displaystyle G_{f,n}^{\alpha ,\beta }(\mathbf{x},\mathbf{y})=\left(f_X(\mathbf{x})f_Y(\mathbf{y})+{\delta }_f/(n\beta )\right)(\sqrt{\alpha }{f}_{XY}(\mathbf{x},\mathbf{y})}\hfill \\ {\displaystyle +\sqrt{\beta }\left(f_X(\mathbf{x})f_Y(\mathbf{y})+{\delta }_f/(\beta n)\right)/{\left(\alpha f_{XY}(\mathbf{x},\mathbf{y})+\beta f_X(\mathbf{x})f_Y(\mathbf{y})\right)}^2,}\hfill \end{array}$$

(A27)

where ${\delta }_f$ is given in (A15). Next After all still the expression (A27) is complicated to optimize therefore we use the fact that $0\le \alpha ,\beta \le 1$ to bound the function ${\mathbf{G}}_{f,n}^{\alpha ,\beta }(\mathbf{x},\mathbf{y})$. Define the set $\mathsf{\Gamma }$

$$\mathsf{\Gamma }:=\left\{{ϵ}_{XY}:|{ϵ}_{XY}(\mathbf{t})-{ϵ}_{XY}({\mathbf{t}}^{\prime })|\le \overline{K}{\parallel \mathbf{t}-{\mathbf{t}}^{\prime }\parallel }_d^{\eta }\right\},$$

where

$$\overline{K}=C_{ϵ}^{U}K\left\{C_{XY}^L+C_X^L+C_Y^L{C}_X^L{C}_X^U\right\}.$$

Here K is the smoothness constant in the Hölder class. Notice that set $\mathsf{\Gamma }$ is a convex set. We bound $\overline{\Delta }$ by

$${\displaystyle \tilde{\Delta }(\alpha ,{ϵ}_{XY})=D(n,l_n,d,\eta )+\tilde{D}(n,l_n,d)C_{XY}^U\underset{{\mathbb{S}}_{XY}}{\int \int }{\tilde{G}}_{{ϵ}_{XY},n}^{\alpha ,\beta }(\mathbf{x},\mathbf{y})\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}.}$$

(A28)

Set $C_n=C_{XY}^{L}n/2$,

$${\displaystyle {\tilde{G}}_n^{\alpha ,\beta }({ϵ}_{XY})=\frac{\left({ϵ}_{XY}^{-1}(\mathbf{x},\mathbf{y})+{(\beta C_n)}^{-1})(1+{ϵ}_{XY}^{-1}(\mathbf{x}\mathbf{y})+{(\beta C_n)}^{-1}\right)}{{\left(\alpha +\beta {ϵ}_{XY}^{-1}(\mathbf{x},\mathbf{y})\right)}^2}.}$$

(A29)

This simplifies to

$${\displaystyle {\tilde{G}}_n^{\alpha ,\beta }({ϵ}_{XY})=\frac{\left(1+{(\beta C_n)}^{-1}{ϵ}_{XY}\right)\left(1+{ϵ}_{XY}+{(\beta C_n)}^{-1}{ϵ}_{XY}\right)}{{\left(\alpha {ϵ}_{XY}+\beta \right)}^2}.}$$

(A30)

Under the condition

$${\displaystyle \frac{2}{C_n}\le \alpha \le min\left\{{\displaystyle \frac{1}{2}+\frac{1}{2C_n},\frac{1}{3}+\frac{2}{3C_n}}\right\},}$$

(A31)

${\tilde{G}}_n^{\alpha ,\beta }({ϵ}_{XY})$ is an increasing function in $ϵ$. Furthermore, for ${\displaystyle \alpha \le \frac{1}{4}}$ and

$$C_{ϵ}^L\le {ϵ}_{XY}\le min\left\{C_{ϵ}^U,{\theta }^U(\alpha )\right\},\mathrm{where}{\theta }^U(\alpha )=\frac{1-4\alpha +1/C_n}{2\alpha },$$

(A32)

the function ${\tilde{G}}_n^{\alpha ,\beta }({ϵ}_{XY})$ is strictly concave. Next, to find an optimal $\alpha$ we consider the following optimization problem:

$$\begin{array}{cc}\underset{\alpha }{\min }\underset{{ϵ}_{XY}\in \mathsf{\Gamma }}{\max }\hfill & \tilde{\Delta }(\alpha ,{ϵ}_{XY})+c_d(1-\alpha )n^{-1}\hfill \\ \mathrm{subject}\mathrm{to}\hfill & C_{ϵ}^L\le {ϵ}_{XY}\le C_{ϵ}^U,\hfill \end{array}$$

(A33)

here ${ϵ}_{XY}=f_{XY}/f_X{f}_Y$, $C_{ϵ}^U=C_{XY}^U/C_X^L{C}_Y^L$ and $C_{ϵ}^L=C_{XY}^L/C_X^U{C}_Y^U$, such that $C_{ϵ}^L\le 1$. We know that under conditions (A31) and (A32), the function ${\tilde{G}}_n^{\alpha ,\beta }$ is strictly concave and increasing in ${ϵ}_{XY}$. We first solve the optimization problem:

$$\begin{array}{cc}\underset{{ϵ}_{XY}\in \mathsf{\Gamma }}{\max }\hfill & {\displaystyle \underset{{\mathbb{S}}_{XY}}{\int \int }{\tilde{G}}_n^{\alpha ,\beta }\left({ϵ}_{XY}(\mathbf{x},\mathbf{y})\right)\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}}\hfill \\ \mathrm{subject}\mathrm{to}\hfill & {\theta }_{ϵ}^L(\alpha )\mathbb{V}({\mathbb{S}}_{XY})\le \underset{{\mathbb{S}}_{XY}}{\int \int }{ϵ}_{XY}(\mathbf{x},\mathbf{y})\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}\hfill \\ & \le {\theta }_{ϵ}^U(\alpha )\mathbb{V}({\mathbb{S}}_{XY}),\hfill \end{array}$$

(A34)

where

$${\theta }_{ϵ}^L(\alpha ):=C_{ϵ}^L,{\theta }_{ϵ}^U(\alpha ):=min\{C_{ϵ}^U,{\theta }^U(\alpha )\}.$$

(A35)

The Lagrangian for this problem is

$$\begin{array}{c}{\displaystyle L({ϵ}_{XY},{\lambda }_1,{\lambda }_2)={\int \int }_{{\mathbb{S}}_{XY}}{\tilde{G}}_n^{\alpha ,\beta }\left({ϵ}_{XY}(\mathbf{x},\mathbf{y})\right)\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}-{\lambda }_1\left({\displaystyle {\int \int }_{{\mathbb{S}}_{XY}}{ϵ}_{XY}(\mathbf{x},\mathbf{y})\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}-{\theta }_{ϵ}^U(\alpha )\mathbb{V}({\mathbb{S}}_{XY})}\right)}\hfill \\ -{\lambda }_2\left({\displaystyle {\theta }_{ϵ}^L(\alpha )\mathbb{V}({\mathbb{S}}_{XY})-{\int \int }_{{\mathbb{S}}_{XY}}{ϵ}_{XY}(\mathbf{x},\mathbf{y})\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}}\right).\hfill \end{array}$$

In this case, the optimum ${ϵ}_{XY}^{*}$ is bounded, ${\theta }_{ϵ}^L(\alpha )\le {ϵ}_{XY}^{*}\le {\theta }_{ϵ}^U(\alpha ),$ and the Lagrangian multiplier ${\lambda }_1^{*},{\lambda }_2^{*}\ge 0$ is such that

$$\underset{{\lambda }_1,{\lambda }_2\ge 0}{\min }\underset{{ϵ}_{XY}\in \mathsf{\Gamma }}{\max }L({ϵ}_{XY},{\lambda }_1,{\lambda }_2)=L({ϵ}_{XY}^{*},{\lambda }_1^{*},{\lambda }_2^{*}).$$

Set ${\displaystyle G_n^{\prime }({ϵ}_{XY})=\frac{\mathrm{d}}{\mathrm{d}{ϵ}_{XY}}{\tilde{G}}_n^{\alpha ,\beta }({ϵ}_{XY}).}$ In view of the concavity of ${\tilde{G}}_{{ϵ}_{XY},n}^{\alpha ,\beta }$ and Lemma 1, page 227 in [49], maximizing $L({ϵ}_{XY},{\lambda }_1^{*},{\lambda }_2^{*})$ over ${ϵ}_{XY}$ is equivalent to

$${\displaystyle {\int \int }_{{\mathbb{S}}_{XY}}\left\{G_n^{\prime }\left({ϵ}_{XY}^{*}(\mathbf{x},\mathbf{y})\right)-({\lambda }_1^{*}-{\lambda }_2^{*})\right\}{ϵ}_{XY}(\mathbf{x},\mathbf{y})\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}\le 0,}$$

(A36)

for all ${\theta }_{ϵ}^L(\alpha )\le {ϵ}_{XY}^{*}\le {\theta }_{ϵ}^U(\alpha )$, and

$${\displaystyle {\int \int }_{{\mathbb{S}}_{XY}}\left\{G_n^{\prime }\left({ϵ}_{XY}^{*}(\mathbf{x},\mathbf{y})\right)-({\lambda }_1^{*}-{\lambda }_2^{*})\right\}{ϵ}_{XY}^{*}(\mathbf{x},\mathbf{y})\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}=0.}$$

(A37)

Denote ${G^{\prime }}_n^{-1}$ the inverse function of $G_n^{\prime }$. Since $G_n^{\prime }$ is strictly decreasing in ${ϵ}_{XY}^{*}$ (this is because ${\tilde{G}}_n^{\alpha ,\beta }({ϵ}_{XY})$ is strictly concave, so that ${G^{\prime }}_n^{-1}$ is continuous and strictly decreasing in ${ϵ}_{XY}^{*}$). From (A36) and (A37), we see immediately that on any interval ${\theta }_{ϵ}^L(\alpha )\le {ϵ}_{XY}^{*}\le {\theta }_{ϵ}^U(\alpha )$, we have ${ϵ}_{XY}^{*}={G^{\prime }}_n^{-1}({\lambda }_1^{*}-{\lambda }_2^{*})$. We can write then

$$G_n^{\prime }\left({\theta }_{ϵ}^U(\alpha )\right)\le {\lambda }_1^{*}-{\lambda }_2^{*}\le G_n^{\prime }\left({\theta }_{ϵ}^L(\alpha )\right),$$

and ${\lambda }_1^{*},{\lambda }_2^{*}\ge 0$. Next, we find the solution of

$$\underset{{\lambda }_1,{\lambda }_2\ge 0}{\min }{\overline{G}}_n^{\alpha ,\beta }({\lambda }_1,{\lambda }_2),\mathrm{where}$$

$$\begin{array}{c}{\overline{G}}_n^{\alpha ,\beta }({\lambda }_1,{\lambda }_2)=\mathbb{V}({\mathbb{S}}_{XY})\left\{{\tilde{G}}_n^{\alpha ,\beta }\left({G^{\prime }}_n^{-1}({\lambda }_1-{\lambda }_2)\right)-({\lambda }_1-{\lambda }_2){G^{\prime }}_n^{-1}({\lambda }_1-{\lambda }_2)+{\lambda }_1{\theta }_{ϵ}^U(\alpha )-{\lambda }_2{\theta }_{ϵ}^L(\alpha )\right\}.\hfill \end{array}$$

The function ${\overline{G}}_n^{\alpha ,\beta }({\lambda }_1,{\lambda }_2)$ is increasing in ${\lambda }_1$ and ${\lambda }_2$, and therefore it takes its minimum at $({\lambda }_1^{*},{\lambda }_2^{*})=(G_n^{\prime }\left({\theta }_{ϵ}^U(\alpha )\right),0)$. This implies that ${ϵ}_{XY}^{*}={\theta }_{ϵ}^U(\alpha )$. Returning to our primary minimization over $\alpha$:

$$\begin{array}{cc}\underset{\alpha }{\min }\hfill & \tilde{\Delta }(\alpha ,{ϵ}_{XY}^{*})+c_d(1-\alpha )n^{-1}\hfill \\ \mathrm{subject}\mathrm{to}\hfill & {\alpha }_0^L\le \alpha \le {\alpha }_0^U,\hfill \end{array}$$

(A38)

where ${\displaystyle {\alpha }_0^L=\frac{2}{C_n}}$ and ${\displaystyle {\alpha }_0^U=min\left\{\frac{1}{4},1-n^{\eta /d-1}\right\}}$. We know that ${\displaystyle \frac{1}{4}\le \frac{1}{3}+\frac{2}{3C_n}}$ and ${\displaystyle \frac{1}{4}\le \frac{1}{2}+\frac{1}{2C_n}}$, therefore the condition below

$${\displaystyle \frac{2}{C_n}\le \alpha \le min\left\{\frac{1}{4},1-n^{\eta /d-1}\right\},}$$

implies the constraint ${\alpha }_0^L\le \alpha \le {\alpha }_0^U$. Since the objective function (A38) is a complicated function in $\alpha$, it is not feasible to determine whether it is a convex function in $\alpha$. For this reason, let us solve the optimization problem in (A38) in a special case when $C_{ϵ}^U\le {\theta }^U(\alpha )$. This implies ${ϵ}_{XY}^{*}=C_{ϵ}^U$. Under assumption $C_{ϵ}^U$ the objective function in (A38) is convex in $\alpha$. Also, the case $C_{ϵ}^U\le {\theta }^U(\alpha )$ is equivalent to ${\displaystyle \alpha \le \frac{1+1/C_n}{4+2C_{ϵ}^U}}$. Therefore, in the optimization problem we have constraint

$${\displaystyle \frac{2}{C_n}\le \alpha \le min\left\{\frac{1}{4},\frac{1+1/C_n}{4+2C_{ϵ}^U},1-n^{\eta /d-1}\right\}.}$$

We know that $\tilde{\Delta }(\alpha ,{ϵ}_{XY}^{*})+c_d(1-\alpha )n^{-1}$ is convex over $\alpha \in [{\alpha }_0^L,{\alpha }_0^U]$. Therefore, the problem becomes ordinary convex optimization problem. Let $\tilde{\alpha }$, ${\tilde{\lambda }}_1$ and ${\tilde{\lambda }}_2$ be any points that satisfy the KKT conditions for this problem:

$$\begin{array}{c}{\alpha }_0^L-\tilde{\alpha }\le 0,\tilde{\alpha }-{\alpha }_0^U\le 0,{\tilde{\lambda }}_1,{\tilde{\lambda }}_2\ge 0,\hfill \\ {\tilde{\lambda }}_1({\alpha }_0^L-\tilde{\alpha })=0,{\tilde{\lambda }}_2(\tilde{\alpha }-{\alpha }_0^U)=0,\hfill \\ {\displaystyle \frac{\mathrm{d}}{\mathrm{d}\alpha }\left(\tilde{\Delta }(\tilde{\alpha },{ϵ}_{XY}^{*})+c_d(1-\tilde{\alpha })n^{-1}\right)-{\tilde{\lambda }}_1+{\tilde{\lambda }}_2=0.}\hfill \end{array}$$

(A39)

Recall $\mathsf{\Xi }(\alpha )$ from (20):

$$\begin{array}{c}{\displaystyle \mathsf{\Xi }(\alpha )=\frac{\mathrm{d}}{\mathrm{d}\alpha }\left(\tilde{\Delta }(\alpha ,{ϵ}_{XY}^{*})+c_d(1-\alpha )n^{-1}\right),}\hfill \end{array}$$

where $\tilde{\Delta }$ is given in (A28). So, the last condition in (A39) becomes $\mathsf{\Xi }(\tilde{\alpha })={\tilde{\lambda }}_1-{\tilde{\lambda }}_2$. We then have

$${\alpha }_0^L\le {\mathsf{\Xi }}^{-1}({\tilde{\lambda }}_1-{\tilde{\lambda }}_2)\le {\alpha }_0^U,$$

where ${\mathsf{\Xi }}^{-1}$ is inverse function of $\mathsf{\Xi }$. Since ${\alpha }_0^L\ne {\alpha }_0^U$, at least one of ${\tilde{\lambda }}_1$ or ${\tilde{\lambda }}_2$ should be zero:

• ${\tilde{\lambda }}_1=0$, ${\tilde{\lambda }}_2\ne 0$. Then $\tilde{\alpha }={\alpha }_0^U$ and implies ${\tilde{\lambda }}_2=-\mathsf{\Xi }({\alpha }_0^U)$. Since ${\tilde{\lambda }}_2>0$, so this leads to $\mathsf{\Xi }({\alpha }_0^U)<0$.
• ${\tilde{\lambda }}_2=0$, ${\tilde{\lambda }}_1\ne 0$. Then $\tilde{\alpha }={\alpha }_0^L$ and implies ${\tilde{\lambda }}_1=\mathsf{\Xi }({\alpha }_0^L)$. We know that ${\tilde{\lambda }}_1>0$, hence $\mathsf{\Xi }({\alpha }_0^L)>0$.
• ${\tilde{\lambda }}_1=0$, ${\tilde{\lambda }}_2=0$. Then $\tilde{\alpha }={\mathsf{\Xi }}^{-1}(0)$ and so ${\alpha }_0^L\le {\mathsf{\Xi }}^{-1}(0)\le {\alpha }_0^U$.

Consequently, by following the behavior of $\mathsf{\Xi }(\alpha )$ with respect to ${\alpha }_0^L$ and ${\alpha }_0^U$, we can often find the optimal $\tilde{\alpha }$, ${\tilde{\lambda }}_1$ and ${\tilde{\lambda }}_2$. For instance, if $\mathsf{\Xi }(\alpha )$ is positive for all $\alpha \in [{\alpha }_0^L,{\alpha }_0^U]$ then we conclude that $\tilde{\alpha }={\alpha }_0^L$.