Open Access
This article is
 freely available
 reusable
Entropy 2018, 20(8), 560; doi:10.3390/e20080560
Article
Ensemble Estimation of Information Divergence †
^{1}
Genetics Department and Applied Math Program, Yale University, New Haven, CT 06520, USA
^{2}
Intuit Inc., Mountain View, CA 94043, USA
^{3}
IBM Research, Cambridge, MA 02142, USA
^{4}
Electrical Engineering and Computer Science Department, University of Michigan, Ann Arbor, MI 48109, USA
*
Correspondence: [email protected]; Tel.: +17347640564
^{†}
This paper is an extended version of our paper published in the 2016 IEEE International Symposium on Information Theory (ISIT), Barcelona, Spain, 10–15 July 2016; pp. 1133–1137.
^{‡}
Current address: Department of Mathematics and Statistics, Utah State University, Logan, UT 84322, USA; [email protected].
Received: 29 June 2018 / Accepted: 26 July 2018 / Published: 27 July 2018
Abstract
:Recent work has focused on the problem of nonparametric estimation of information divergence functionals between two continuous random variables. Many existing approaches require either restrictive assumptions about the density support set or difficult calculations at the support set boundary which must be known a priori. The mean squared error (MSE) convergence rate of a leaveoneout kernel density plugin divergence functional estimator for general bounded density support sets is derived where knowledge of the support boundary, and therefore, the boundary correction is not required. The theory of optimally weighted ensemble estimation is generalized to derive a divergence estimator that achieves the parametric rate when the densities are sufficiently smooth. Guidelines for the tuning parameter selection and the asymptotic distribution of this estimator are provided. Based on the theory, an empirical estimator of Rényi$\alpha $ divergence is proposed that greatly outperforms the standard kernel density plugin estimator in terms of mean squared error, especially in high dimensions. The estimator is shown to be robust to the choice of tuning parameters. We show extensive simulation results that verify the theoretical results of our paper. Finally, we apply the proposed estimator to estimate the bounds on the Bayes error rate of a cell classification problem.
Keywords:
divergence; differential entropy; nonparametric estimation; central limit theorem; convergence rates; bayes error rate1. Introduction
Information divergences are integral functionals of two probability distributions and have many applications in the fields of information theory, statistics, signal processing, and machine learning. Some applications of divergences include estimating the decay rates of error probabilities [1], estimating bounds on the Bayes error [2,3,4,5,6,7,8] or the minimax error [9] for a classification problem, extending machine learning algorithms to distributional features [10,11,12,13], testing the hypothesis that two sets of samples come from the same probability distribution [14], clustering [15,16,17], feature selection and classification [18,19,20], blind source separation [21,22], image segmentation [23,24,25], and steganography [26]. For many more applications of divergence measures, see reference [27]. There are many information divergence families including Alpha and Betadivergences [28] as well as fdivergences [29,30]. In particular, the fdivergence family includes the wellknown Kullback–Leibler (KL) divergence [31], the Rényi$\alpha $ divergence integral [32], the Hellinger–Bhattacharyya distance [33,34], the Chernoff$\alpha $ divergence [5], the total variation distance, and the Henze–Penrose divergence [6].
Despite the many applications of divergences between continuous random variables, there are no nonparametric estimators of these functionals that achieve the parametric mean squared error (MSE) convergence rate, are simple to implement, do not require knowledge of the boundary of the density support set, and apply to a large set of divergence functionals. In this paper, we present the first information divergence estimator that achieves all of the above. Specifically, we address the problem of estimating divergence functionals when only a finite population of independent and identically distributed (i.i.d.) samples is available from the two ddimensional distributions that are unknown, nonparametric, and smooth. Our contributions are as follows:
 We propose the first information divergence estimator, referred to as EnDive, that is based on ensemble methods. The ensemble estimator takes a weighted average of an ensemble of weak kernel density plugin estimators of divergence where the weights are chosen to improve the MSE convergence rate. This ensemble construction makes it very easy to implement EnDive.
 We prove that the proposed ensemble divergence estimator achieves the optimal parametric MSE rate of $O\left(\frac{1}{N}\right)$, where N is the sample size when the densities are sufficiently smooth. In particular, EnDive achieves these rates without explicitly performing boundary correction which is required for most other estimators. Furthermore, we show that the convergence rates are uniform.
 We prove that EnDive obeys a central limit theorem and thus, can be used to perform inference tasks on the divergence such as testing that two populations have identical distributions or constructing confidence intervals.
1.1. Related Work
Much work has focused on the problem of estimating the entropy and the information divergence of discrete random variables [1,29,35,36,37,38,39,40,41,42,43]. However, the estimation problem for discrete random variables differs significantly from the continuous case and thus employs different tools for both estimation and analysis.
One approach to estimating the differential entropy and information divergence of continuous random variables is to assume a parametric model for the underlying probability distributions [44,45,46]. However, these methods perform poorly when the parametric model does not fit the data well. Unfortunately, the structure of the underlying data distribution is unknown for many applications, and thus the chance for model misspecification is high. Thus, in many of these applications, parametric methods are insufficient, and nonparametric estimators must be used.
While several nonparametric estimators of divergence functionals between continuous random variables have been previously defined, the convergence rates are known for only a few of them. Furthermore, the asymptotic distributions of these estimators are unknown for nearly all of them. For example, Póczos and Schneider [10] established a weak consistency for a biascorrected knearest neighbor (nn) estimator for Rényi$\alpha $ and other divergences of a similar form where k was fixed. Li et al. [47] examined knn estimators of entropy and the KL divergence using hyperspherical data. Wang et al. [48] provided a knn based estimator for KL divergence. Plugin histogram estimators of mutual information and divergence have been proven to be consistent [49,50,51,52]. Hero et al. [53] provided a consistent estimator for Rényi$\alpha $ divergence when one of the densities is known. However none of these works studied the convergence rates or the asymptotic distribution of their estimators.
There has been recent interest in deriving convergence rates for divergence estimators for continuous data [54,55,56,57,58,59,60]. The rates are typically derived in terms of a smoothness condition on the densities, such as the Hölder condition [61]:
Definition 1 (Hölder Class).
Let $\mathcal{X}\subset {\mathbb{R}}^{d}$ be a compact space. For $r=({r}_{1},\dots ,{r}_{d}),$ ${r}_{i}\in \mathbb{N},$ define $\leftr\right={\sum}_{i=1}^{d}{r}_{i}$ and ${D}^{r}=\frac{{\partial}^{\leftr\right}}{\partial {x}_{1}^{{r}_{1}}\dots \partial {x}_{d}^{{r}_{d}}}$. The Hölder class $\mathsf{\Sigma}(s,{K}_{H})$ of functions on ${L}_{2}(\mathcal{X})$ consists of the functions (f) that satisfy
for all $x,\phantom{\rule{0.166667em}{0ex}}y\in \mathcal{X}$ and for all r s.t. $\leftr\right\le \u230as\u230b$.
$$\left{D}^{r}f(x){D}^{r}f(y)\right\le {K}_{H}{\u2225xy\u2225}^{min(sr,1)},$$
From Definition 1, it is clear that if a function (f) belongs to $\mathsf{\Sigma}(s,{K}_{H})$, then f is continuously differentiable up to order $\u230as\u230b$. In this work, we show that EnDive achieves a parametric MSE convergence rate of $O(1/N)$ when $s\ge d$ and $s>\frac{d}{2}$, depending on the specific form of the divergence function.
Nguyen et al. [56] proposed an fdivergence estimator that estimates the likelihood ratio of the two densities by solving a convex optimization problem and then plugging it into the divergence formulas. The authors proved that the minimax MSE convergence rate is parametric when the likelihood ratio is a member of the bounded Hölder class $\mathsf{\Sigma}(s,{K}_{H})$ with $s\ge d/2$. However, this estimator is restricted to true fdivergences and may not apply to the broader class of divergence functionals that we consider here (as an example, the ${L}_{2}^{2}$ divergence is not an fdivergence). Additionally, solving the convex problem of [56] has similar computational complexity to that of training a support vector machine (SVM) (between $O({N}^{2})$ and $O({N}^{3})$), which can be demanding when N is large. In contrast, the EnDive estimator that we propose requires only the construction of simple density plugin estimates and the solution of an offline convex optimization problem. Therefore, the most computationally demanding step in the EnDive estimator is the calculation of the density estimates, which has a computational complexity no greater than $O({N}^{2})$.
Singh and Póczos [58,59] provided an estimator for Rényi$\alpha $ divergences as well as general density functionals that use a “mirror image” kernel density estimator. They proved that these estimators obtain an MSE convergence rate of $O\left(\frac{1}{N}\right)$ when $s\ge d$ for each of the densities. However their approach requires several computations at each boundary of the support of the densities which is difficult to implement as d gets large. Also, this computation requires knowledge of the support (specifically, the boundaries) of the densities which is unknown in most practical settings. In contrast, while our assumptions require the density support sets to be bounded and the boundaries to be smooth, knowledge of the support is not required to implement EnDive.
The “linear” and “quadratic” estimators presented by Krishnamurthy et al. [57] estimate divergence functionals that include the form $\int {f}_{1}^{\alpha}(x){f}_{2}^{\beta}(x)d\mu (x)$ for given $\alpha $ and $\beta $ where ${f}_{1}$ and ${f}_{2}$ are probability densities. These estimators achieve the parametric rate when $s\ge d/2$ and $s\ge d/4$ for the linear and quadratic estimators, respectively. However, the latter estimator is computationally infeasible for most functionals, and the former requires numerical integration for some divergence functionals which can be computationally difficult. Additionally, while a suitable $\alpha $$\beta $ indexed sequence of divergence functionals of this form can be constructed that converge to the KL divergence, this does not guarantee convergence of the corresponding sequence of divergence estimators, as shown in reference [57]. In contrast, EnDive can be used to estimate the KL divergence directly. Other important fdivergence functionals are also excluded from this form including some that bound the Bayes error [2,4,6]. In contrast, our method applies to a large class of divergence functionals and avoids numerical integration.
Finally, Kandasamy et al. [60] proposed influence functionbased estimators of distributional functionals including divergences that achieve the parametric rate when $s\ge d/2$. While this method can be applied to general functionals, the estimator requires numerical integration for some functionals. Additionally, the estimators in both Kandasamy et al. [60] and Krishnamurthy et al. [57] require an optimal kernel density estimator. This is difficult to construct when the density support is bounded as it requires difficult computations at the density support set boundary and therefore, knowledge of the density support set. In contrast, Endive does not require knowledge of the support boundary.
In addition to the MSE convergence rates, the asymptotic distribution of divergence estimators is of interest. Asymptotic normality has been established for certain divergences between a specific density estimator and the true density [62,63,64]. This differs from the problem we consider where we assume that both densities are unknown. The asymptotic distributions of the estimators in references [56,57,58,59] are currently unknown. Thus, it is difficult to use these estimators for hypothesis testing which is crucial in many scientific applications. Kandasamy et al. [60] derived the asymptotic distribution of their datasplitting estimator but did not prove similar results for their leaveoneout estimator. We establish a central limit theorem for EnDive which greatly enhances its applicability in scientific settings.
Our ensemble divergence estimator reduces to an ensemble entropy estimator as a special case when data from only one distribution is considered and the other density is set to a uniform measure (see reference [28] for more on the relationship between entropy and information divergence). The resultant entropy estimator differs from the ensemble entropy estimator proposed by Sricharan et al. [65] in several important ways. First, the density support set must be known for the estimator in reference [65] to perform the explicit boundary correction. In contrast, the EnDive estimator does not require any boundary correction. To show this requires a significantly different approach to prove the bias and variance rates of the EnDive estimator. Furthermore, the EnDive results apply under more general assumptions for the densities and the kernel used in the weak estimators. Finally, the central limit theorem applies to the EnDive estimator which is currently unknown for the estimator in reference [65].
We also note that Berrett et al. [66] proposed a modification of the Kozachenko and Leonenko estimator of entropy [67] that takes a weighted ensemble estimation approach. While their results require stronger assumptions for the smoothness of the densities than ours do, they did obtain the asymptotic distribution of their weighted estimator and they also showed that the asymptotic variance of the estimator is not increased by taking a weighted average. This latter point is an important selling point of the ensemble framework—we can improve the asymptotic bias of an estimator without increasing the asymptotic variance.
1.2. Organization and Notation
The paper is organized as follows. We first derive the MSE convergence rates in Section 2 for a weak divergence estimator, which is a kernel density plugin divergence estimator. We then generalize the theory of optimally weighted ensemble entropy estimation developed in reference [65] to obtain the ensemble divergence estimator EnDive from an ensemble of weak estimators in Section 3. A central limit theorem and uniform convergence rate for the ensemble estimator are also presented in Section 3. In Section 4, we provide guidelines for selecting the tuning parameters based on experiments and the theory derived in the previous sections. We then perform experiments in Section 4 that validate the theory and establish the robustness of the proposed estimators to the tuning parameters.
Bold face type is used for random variables and random vectors. The conditional expectation given a random variable $\mathbf{Z}$ is denoted as ${\mathbb{E}}_{\mathbf{Z}}$. The variance of a random variable is denoted as $\mathbb{V}$, and the bias of an estimator is denoted as $\mathbb{B}$.
2. The Divergence Functional Weak Estimator
This paper focuses on estimating functionals of the form
where $g(x,y)$ is a smooth functional, and ${f}_{1}$ and ${f}_{2}$ are smooth ddimensional probability densities. If $g\left({f}_{1}(x),{f}_{2}(x)\right)=g\left(\frac{{f}_{1}(x)}{{f}_{2}(x)}\right),$ g is convex, and $g(1)=0$, then $G\left({f}_{1},{f}_{2}\right)$ defines the family of fdivergences. Some common divergences that belong to this family include the KL divergence ($g(t)=lnt$) and the total variation distance ($g(t)=t1$). In this work, we consider a broader class of functionals than the fdivergences, since g is allowed to be very general.
$$G\left({f}_{1},{f}_{2}\right)=\int g\left({f}_{1}(x),{f}_{2}(x)\right){f}_{2}(x)dx,,$$
To estimate $G\left({f}_{1},{f}_{2}\right)$, we first define a weak plugin estimator based on kernel density estimators (KDEs), that is, a simple estimator that converges slowly to the true value $G\left({f}_{1},{f}_{2}\right)$ in terms of MSE. We then derive the bias and variance expressions for this weak estimator as a function of sample size and bandwidth. We then use the resulting bias and variance expressions to derive an ensemble estimator that takes a weighted average of weak estimators with different bandwidths and achieves superior MSE performance.
2.1. The Kernel Density Plugin Estimator
We use a kernel density plugin estimator of the divergence functional in (1) as the weak estimator. Assume that ${N}_{1}$ i.i.d. realizations $\left\{{\mathbf{Y}}_{1},\dots ,{\mathbf{Y}}_{{N}_{1}}\right\}$ are available from ${f}_{1}$ and ${N}_{2}$ i.i.d. realizations $\left\{{\mathbf{X}}_{1},\dots ,{\mathbf{X}}_{{N}_{2}}\right\}$ are available from ${f}_{2}$. Let ${h}_{i}>0$ be the kernel bandwidth for the density estimator of ${f}_{i}$. For simplicity of presentation, assume that ${N}_{1}={N}_{2}=N$ and ${h}_{1}={h}_{2}=h$. The results for the more general case of differing sample sizes and bandwidths are given in Appendix C. Let $K(\xb7)$ be a kernel function with $\int K(x)dx=1$ and ${\left\rightK\left\right}_{\infty}<\infty $ where ${\parallel K\parallel}_{\infty}$ is the ${\ell}_{\infty}$ norm of the kernel (K). The KDEs for ${f}_{1}$ and ${f}_{2}$ are, respectively,
where $M=N1$. $G\left({f}_{1},{f}_{2}\right)$ is then approximated as
$$\begin{array}{ccc}\hfill {\tilde{\mathbf{f}}}_{1,h}({\mathbf{X}}_{j})& =& \frac{1}{N{h}^{d}}{\displaystyle \sum _{i=1}^{N}}K\left(\frac{{\mathbf{X}}_{j}{\mathbf{Y}}_{i}}{h}\right),\hfill \\ \hfill {\tilde{\mathbf{f}}}_{2,h}({\mathbf{X}}_{j})& =& \frac{1}{M{h}^{d}}{\displaystyle \sum _{\begin{array}{c}i=1\\ i\ne j\end{array}}^{N}}K\left(\frac{{\mathbf{X}}_{j}{\mathbf{X}}_{i}}{h}\right),\hfill \end{array}$$
$${\tilde{\mathbf{G}}}_{h}=\frac{1}{N}{\displaystyle \sum _{i=1}^{N}}g\left({\tilde{\mathbf{f}}}_{1,h}\left({\mathbf{X}}_{i}\right),{\tilde{\mathbf{f}}}_{2,h}\left({\mathbf{X}}_{i}\right)\right).$$
2.2. Convergence Rates
For many estimators, MSE convergence rates are typically provided in the form of upper (or sometimes lower) bounds on the bias and the variance. Therefore, only the slowest converging terms (as a function of the sample size (N)) are presented in these cases. However, to apply our generalized ensemble theory to obtain estimators that guarantee the parametric MSE rate, we required explicit expressions for the bias of the weak estimators in terms of the sample size (N) and the kernel bandwidth (h). Thus, an upper bound was insufficient for our work. Furthermore, to guarantee the parametric rate, we required explicit expressions of all bias terms that converge to zero slower than $O\left(1/\sqrt{N}\right)$.
To obtain bias expressions, we required multiple assumptions on the densities ${f}_{1}$ and ${f}_{2}$, the functional g, and the kernel K. Similar to reference [7,54,65], the principal assumptions we make were that (1) ${f}_{1}$, ${f}_{2},$ and g are smooth; (2) ${f}_{1}$ and ${f}_{2}$ have common bounded support sets $\mathcal{S}$; and (3) ${f}_{1}$ and ${f}_{2}$ are strictly lower bounded on $\mathcal{S}$. We also assume (4) that the density support set is smooth with respect to the kernel ($K(u)$). The full technical assumptions and a discussion of them are contained in Appendix A. Given these assumptions, we have the following result on the bias of ${\tilde{\mathbf{G}}}_{h}$:
Theorem 1.
For a general g, the bias of the plugin estimator ${\tilde{\mathbf{G}}}_{h}$ is given by
$$\mathbb{B}\left[{\tilde{\mathbf{G}}}_{h}\right]={\displaystyle \sum _{j=1}^{\u230as\u230b}}{c}_{10,j}{h}^{j}+{c}_{11}\frac{1}{N{h}^{d}}+O\left({h}^{s}+\frac{1}{N{h}^{d}}\right).$$
To apply our generalized ensemble theory to the KDE plugin estimator (${\tilde{\mathbf{G}}}_{h}$), we required only an upper bound on its variance. The following variance result required much less strict assumptions than the bias results in Theorem 1:
Theorem 2.
Assume that the functional g in (1) is Lipschitz continuous in both of its arguments with the Lipschitz constant (${C}_{g}$). Then, the variance of the plugin estimator (${\tilde{\mathbf{G}}}_{h}$) is bounded by
$$\mathbb{V}\left[{\tilde{\mathbf{G}}}_{h}\right]\le {C}_{g}^{2}{\left\rightK\left\right}_{\infty}^{2}\frac{11}{N}.$$
From Theorems 1 and 2, we observe that $h\to 0$ and $N{h}^{d}\to \infty $ are required for ${\tilde{\mathbf{G}}}_{h}$ to be unbiased, while the variance of the plugin estimator depends primarily on the sample size (N). Note that the constants depend on the densities ${f}_{1}$ and ${f}_{2}$ and their derivatives which are often unknown.
2.3. Optimal MSE Rate
From Theorem 1, the dominating terms in the bias are observed to be $\mathsf{\Theta}\left(h\right)$ and $\mathsf{\Theta}\left(\frac{1}{N{h}^{d}}\right)$. If no bias correction is performed, the optimal choice of h that minimizes MSE is
$${h}^{\ast}=\mathsf{\Theta}\left({N}^{\frac{1}{d+1}}\right).$$
This results in a dominant bias term of order $\mathsf{\Theta}\left({N}^{\frac{1}{d+1}}\right)$. Note that this differs from the standard result for the optimal KDE bandwidth for minimum MSE density estimation which is $\mathsf{\Theta}\left({N}^{1/\left(d+4\right)}\right)$ for a symmetric uniform kernel when the boundary bias is ignored [68].
Figure 1 shows a heatmap showing the leading bias term $O\left(h\right)$ as a function of d and N when $h={N}^{\frac{1}{d+1}}$. The heatmap indicates that the bias of the plugin estimator in (2) is small only for relatively small values of d. This is consistent with the empirical results in reference [69] which examined the MSE of multiple plugin KDE and knn estimators. In the next section, we propose an ensemble estimator that achieves a superior convergence rate regardless of the dimensions (d) as long as the density is sufficiently smooth.
2.4. Proof Sketches of Theorems 1 and 2
To prove the bias expressions in Theorem 1, the bias is first decomposed into two parts by adding and subtracting $g\left({\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{1,h}(\mathbf{Z}),{\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{2,h}(\mathbf{Z})\right)$ within the expectation creating a “bias” term and a “variance” term. Applying a Taylor series expansion on the bias and variance terms results in expressions that depend on powers of ${\mathbb{B}}_{\mathbf{Z}}\left[{\tilde{\mathbf{f}}}_{i,h}(\mathbf{Z})\right]:={\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{i,h}(\mathbf{Z}){f}_{i}(\mathbf{Z})$ and ${\tilde{\mathbf{e}}}_{i,h}(\mathbf{Z}):={\tilde{\mathbf{f}}}_{i,h}(\mathbf{Z}){\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{i,h}(\mathbf{Z})$, respectively. Within the interior of the support, moment bounds can be derived from properties of the KDEs and a Taylor series expansion of the densities. Near the boundary of the support, the smoothness assumption on the boundary $\mathcal{A}.5$ is required to obtain an expression of the bias in terms of the KDE bandwidth (h) and the sample size (N). The full proof of Theorem 1 is given in Appendix E.
The proof of the variance result takes a different approach. The proof uses the Efron–Stein inequality [70] which bounds the variance by analyzing the expected squared difference between the plugin estimator when one sample is allowed to differ. This approach provides a bound on the variance under much less strict assumptions on the densities and the functional g than is required for Theorem 1. The full proof of Theorem 2 is given in Appendix F.
3. Weighted Ensemble Estimation
From Theorem 1 and Figure 1, we can observe that the bias of the MSEoptimal plugin estimator ${\tilde{\mathbf{G}}}_{h}$ decreases very slowly as a function of the sample size (N) when the data dimensions (d) are not small, resulting in a large MSE. However, by applying the theory of optimally weighted ensemble estimation, we can obtain an estimator with improved performance by taking a weighted sum of an ensemble of weak estimators where the weights are chosen to significantly reduce the bias.
The ensemble of weak estimators is formed by choosing different values of the bandwidth parameter h as follows. Set $\mathcal{L}=\left\{{l}_{1},\dots ,{l}_{L}\right\}$ to be real positive numbers that index $h({l}_{i})$. Thus, the parameter l indexes over different neighborhood sizes for the KDEs. Define the weight $w:=\left\{w\left({l}_{1}\right),\dots ,w\left({l}_{L}\right)\right\}$ and ${\tilde{\mathbf{G}}}_{w}:={\sum}_{l\in \mathcal{L}}w(l){\tilde{\mathbf{G}}}_{h(l)}.$ That is, for each estimator ${\tilde{\mathbf{G}}}_{h(l)}$ there is a corresponding weight value ($w(l)$). The key to reducing the MSE is to choose the weight vector (w) to reduce the lower order terms in the bias while minimizing the impact of the weighted average on the variance.
3.1. Finding the Optimal Weight
The theory of optimally weighted ensemble estimation is a general theory that is applicable to any estimation problem as long as the bias and variance of the estimator can be expressed in a specific way. An early version of this theory was presented in reference [65]. We now generalize this theory so that it can be applied to a wider variety of estimation problems. Let N be the number of available samples and let $\mathcal{L}=\left\{{l}_{1},\dots ,{l}_{L}\right\}$ be a set of index values. Given an indexed ensemble of estimators ${\left\{{\widehat{\mathbf{E}}}_{l}\right\}}_{l\in \mathcal{L}}$ of some parameter (E), the weighted ensemble estimator with weights $w=\left\{w\left({l}_{1}\right),\dots ,w\left({l}_{L}\right)\right\}$ satisfying ${\sum}_{l\in \mathcal{L}}w(l)=1$ is defined as
${\widehat{\mathbf{E}}}_{w}$ is asymptotically unbiased as long as the estimators ${\left\{{\widehat{\mathbf{E}}}_{l}\right\}}_{l\in \mathcal{L}}$ are asymptotically unbiased. Consider the following conditions on ${\left\{{\widehat{\mathbf{E}}}_{l}\right\}}_{l\in \mathcal{L}}$:
$${\widehat{\mathbf{E}}}_{w}={\displaystyle \sum _{l\in \mathcal{L}}}w\left(l\right){\widehat{\mathbf{E}}}_{l}.$$
 $\mathcal{C}.1$ The bias is expressible as$$\mathbb{B}\left[{\widehat{\mathbf{E}}}_{l}\right]={\displaystyle \sum _{i\in J}}{c}_{i}{\psi}_{i}(l){\varphi}_{i,d}(N)+O\left(\frac{1}{\sqrt{N}}\right),$$
 $\mathcal{C}.2$ The variance is expressible as$$\mathbb{V}\left[{\widehat{\mathbf{E}}}_{l}\right]={c}_{v}\left(\frac{1}{N}\right)+o\left(\frac{1}{N}\right).$$
Theorem 3.
Assume conditions $\mathcal{C}.1$ and $\mathcal{C}.2$ hold for an ensemble of estimators ${\left\{{\widehat{\mathbf{E}}}_{l}\right\}}_{l\in \mathcal{L}}$. Then, there exists a weight vector (${w}_{0}$) such that the MSE of the weighted ensemble estimator attains the parametric rate of convergence:
$$\mathbb{E}\left[{\left({\widehat{\mathbf{E}}}_{{w}_{0}}E\right)}^{2}\right]=O\left(\frac{1}{N}\right).$$
The weight vector (${w}_{0}$) is the solution to the following convex optimization problem:
$$\begin{array}{cc}\hfill {\displaystyle \underset{w}{min}}& {\left\rightw\left\right}_{2}\hfill \\ \hfill subject\phantom{\rule{0.166667em}{0ex}}to& {\displaystyle {\sum}_{l\in \mathcal{L}}}w(l)=1,\hfill \\ & {\gamma}_{w}(i)={\displaystyle {\sum}_{l\in \mathcal{L}}}w(l){\psi}_{i}(l)=0,\phantom{\rule{0.166667em}{0ex}}i\in J.\hfill \end{array}$$
Proof.
From condition $\mathcal{C}.1$, we can write the bias of the weighted estimator as
$$\mathbb{B}\left[{\widehat{\mathbf{E}}}_{w}\right]={\displaystyle \sum _{i\in J}}{c}_{i}{\gamma}_{w}(i){\varphi}_{i,d}(N)+O\left(\frac{\sqrt{L}{\left\rightw\left\right}_{2}}{\sqrt{N}}\right).$$
The variance of the weighted estimator is bounded as
$$\mathbb{V}\left[{\widehat{\mathbf{E}}}_{w}\right]\le \frac{{L\left\rightw\left\right}_{2}^{2}}{N}.$$
The optimization problem in (4) zeroes out the lowerorder bias terms and limits the ${\ell}_{2}$ norm of the weight vector (w) to prevent the variance from exploding. This results in an MSE rate of $O(1/N)$ when the dimensions (d) are fixed and when L is fixed independently of the sample size (N). Furthermore, a solution to (4) is guaranteed to exist if $L>I$ and the vectors ${a}_{i}=\left[{\psi}_{i}({l}_{1}),\dots ,{\psi}_{i}({l}_{L})\right]$ are linearly independent. This completes our sketch of the proof of Theorem 3. ☐
3.2. The EnDive Estimator
The parametric rate of $O\left(1/N\right)$ in MSE convergence can be achieved without requiring ${\gamma}_{w}(i)=0,\phantom{\rule{0.166667em}{0ex}}i\in J$. This can be accomplished by solving the following convex optimization problem in place of the optimization problem in Theorem 3:
where the parameter $\eta $ is chosen to achieve a tradeoff between bias and variance. Instead of forcing ${\gamma}_{w}(i)=0$, the relaxed optimization problem uses the weights to decrease the bias terms at a rate of $O\left(1/\sqrt{N}\right)$, yielding an MSE convergence rate of $O(1/N).$ In fact, it was shown in reference [71] that the optimization problem in (6) guarantees the parametric MSE rate as long as the conditions of Theorem 3 are satisfied and a solution to the optimization problem in (4) exists (the conditions for this existence are given in the proof of Theorem 3).
$$\begin{array}{cc}\hfill {\displaystyle \underset{w}{min}}& \u03f5\hfill \\ \hfill subject\phantom{\rule{0.166667em}{0ex}}to& {\sum}_{l\in \mathcal{L}}w(l)=1,\hfill \\ & \left{\gamma}_{w}(i){N}^{\frac{1}{2}}{\varphi}_{i,d}(N)\right\le \u03f5,\phantom{\rule{0.166667em}{0ex}}\phantom{\rule{0.166667em}{0ex}}i\in J,\hfill \\ & {\u2225w\u2225}_{2}^{2}\le \eta \u03f5,\hfill \end{array}$$
We now construct a divergence ensemble estimator from an ensemble of plugin KDE divergence estimators. Consider first the bias result in (3) where g is general, and assume that $s\ge d$. In this case, the bias contains a $O\left(\frac{1}{{h}^{d}N}\right)$ term. To guarantee the parametric MSE rate, any remaining lowerorder bias terms in the ensemble estimator must be no slower than $O\left(1/\sqrt{N}\right)$. Let $h(l)=l{N}^{1/(2d)}$ where $l\in \mathcal{L}$. Then $O\left(\frac{1}{h{(l)}^{d}N}\right)=O\left(\frac{1}{{l}^{d}\sqrt{N}}\right)$. We therefore obtain an ensemble of plugin estimators ${\left\{{\tilde{\mathbf{G}}}_{h(l)}\right\}}_{l\in \mathcal{L}}$ and a weighted ensemble estimator ${\tilde{\mathbf{G}}}_{w}={\sum}_{l\in \mathcal{L}}w(l){\tilde{\mathbf{G}}}_{h(l)}$. The bias of each estimator in the ensemble satisfies the condition $\mathcal{C}.1$ with ${\psi}_{i}(l)={l}^{i}$ and ${\varphi}_{i,d}(N)={N}^{i/(2d)}$ for $i=1,\dots ,d$. To obtain a uniform bound on the bias with respect to w and $\mathcal{L}$, we also include the function ${\psi}_{d+1}(l)={l}^{d}$ with corresponding ${\varphi}_{d+1,d}(N)={N}^{1/2}$. The variance also satisfies the condition $\mathcal{C}.2$. The optimal weight (${w}_{0}$) is found by using (6) to obtain an optimally weighted plugin divergence functional estimator ${\tilde{\mathbf{G}}}_{{w}_{0}}$ with an MSE convergence rate of $O\left(\frac{1}{N}\right)$ as long as $s\ge d$ and $L\ge d$. Otherwise, if $s<d$, we can only guarantee the MSE rate up to $O\left(\frac{1}{{N}^{s/d}}\right)$. We refer to this estimator as the Ensemble Divergence (EnDive) estimator and denote it as ${\tilde{\mathbf{G}}}_{\mathrm{EnDive}}$.
We note that for some functionals (g) (including the KL divergence and the Renyi$\alpha $ divergence integral), we can modify the EnDive estimator to obtain the parametric rate under the less strict assumption that $s>d/2$. For details on this approach, see Appendix B.
3.3. Central Limit Theorem
The following theorem shows that an appropriately normalized ensemble estimator ${\tilde{\mathbf{G}}}_{w}$ converges in distribution to a normal random variable under rather general conditions. Thus, the same result applies to the EnDive estimator ${\tilde{\mathbf{G}}}_{\mathrm{EnDive}}$. This enables us to perform hypothesis testing on the divergence functional which is very useful in many scientific applications. The proof is based on the Efron–Stein inequality and an application of Slutsky’s Theorem (Appendix G).
Theorem 4.
Assume that the functional g is Lipschitz in both arguments with the Lipschitz constant ${C}_{g}$. Further assume that $h(l)=o(1)$, $N\to \infty $, and $Nh{(l)}^{d}\to \infty $ for each $l\in \mathcal{L}$. Then, for a fixed $\mathcal{L}$, the asymptotic distribution of the weighted ensemble estimator ${\tilde{\mathbf{G}}}_{w}$ is
where $\mathbf{S}$ is a standard normal random variable.
$$Pr\left(\left({\tilde{\mathbf{G}}}_{w}\mathbb{E}\left[{\tilde{\mathbf{G}}}_{w}\right]\right)/\sqrt{\mathbb{V}\left[{\tilde{\mathbf{G}}}_{w}\right]}\le t\right)\to Pr(\mathbf{S}\le t),$$
3.4. Uniform Convergence Rates
Here, we show that the optimally weighted ensemble estimators achieve the parametric MSE convergence rate uniformly. Denote the subset of $\mathsf{\Sigma}\left(s,{K}_{H}\right)$ with densities bounded between ${\u03f5}_{0}$ and ${\u03f5}_{\infty}$ as $\mathsf{\Sigma}\left(s,{K}_{H},{\u03f5}_{0},{\u03f5}_{\infty}\right)$.
Theorem 5.
Let ${\tilde{\mathbf{G}}}_{\mathrm{EnDive}}$ be the EnDive estimator of the functional
where p and q are ddimensional probability densities. Additionally, let $r=d$ and assume that $s>r$. Then,
where C is a constant.
$$G(p,q)=\int g\left(p(x),q(x)\right)q(x)dx,$$
$$\underset{p,q\in \mathsf{\Sigma}\left(s,{K}_{H},{\u03f5}_{0},{\u03f5}_{\infty}\right)}{sup}}\mathbb{E}\left[{\left({\tilde{\mathbf{G}}}_{{w}_{0}}G(p,q)\right)}^{2}\right]\le \frac{C}{N},$$
The proof decomposes the MSE into the variance plus the square of the bias. The variance is bounded easily by using Theorem 2. To bound the bias, we show that the constants in the bias terms are continuous with respect to the densities p and q under an appropriate norm. We then show that $\mathsf{\Sigma}\left(s,{K}_{H},{\u03f5}_{0},{\u03f5}_{\infty}\right)$ is compact with respect to this norm and then apply an extreme value theorem. Details are given in Appendix H.
4. Experimental Results
In this section, we discuss the choice of tuning parameters and validate the EnDive estimator’s convergence rates and the central limit theorem. We then use the EnDive estimator to estimate bounds on the Bayes error for a singlecell bone marrow data classification problem.
4.1. Tuning Parameter Selection
The optimization problem in (6) has parameters $\eta $, L, and $\mathcal{L}$. By applying (6), and the resulting MSE of the ensemble estimator is
where each term in the sum comes from the bias and variance, respectively. From this expression and (6), we see that the parameter $\eta $ provides a tradeoff between bias and variance. Increasing $\eta $ enables the norm of the weight vector to be larger. This means the feasible region for the variable w increases in size as $\eta $ increases which can result in decreased bias. However, as $\eta $ contributes to the variance term, increasing $\eta $ may result in increased variance.
$$O\left({\u03f5}^{2}/N\right)+O\left(L{\eta}^{2}{\u03f5}^{2}/N\right),$$
If all of the constants in (3) and an exact expression for the variance of the ensemble estimator were known, then $\eta $ could be chosen to optimize this tradeoff in bias and variance and thus minimize the MSE. Since these constants are unknown, we can only choose $\eta $ based on the asymptotic results. From (8), this would suggest setting $\eta =1/\sqrt{L}$. In practice, we find that for finite sample sizes, the variance in the ensemble estimator is less than the upper bound of $L{\eta}^{2}{\u03f5}^{2}/N$. Thus, setting $\eta =1/\sqrt{L}$ is unnecessarily restrictive. We find that, in practice, setting $\eta =1$ works well.
Upon first glance, it appears that for fixed L, the set $\mathcal{L}$ that parameterizes the kernel widths can, in theory, be chosen by minimizing $\u03f5$ in (6) over $\mathcal{L}$ in addition to w. However, adding this constraint results in a nonconvex optimization problem since w does not lie in the nonnegative orthant. A parameter search over possible values for $\mathcal{L}$ is another possibility. However, this may not be practical as $\u03f5$ generally decreases as the size and spread of $\mathcal{L}$ increases. In addition, for finite sample sizes, decreasing $\u03f5$ does not always directly correspond to a decrease in MSE, as very high or very low values of $h(l)$ can lead to inaccurate density estimates, resulting in a larger MSE.
Given these limitations, we provide the following recommendations for $\mathcal{L}$. Denote the value of the minimum value of l such that ${\tilde{\mathbf{f}}}_{i,h({l}_{min})}({\mathbf{X}}_{j})>0$$\forall i=1,2$ as ${l}_{min}$ and the diameter of the support $\mathcal{S}$ as D. To ensure the KDEs are bounded away from zero, we require that $min(\mathcal{L})\ge {l}_{min}$. As demonstrated in Figure 2, the weights in ${w}_{0}$ are generally largest for the smallest values of $\mathcal{L}$. This indicates that $min(\mathcal{L})$ should also be sufficiently larger than ${l}_{min}$ to render an adequate density estimate. Similarly, $max(\mathcal{L})$ should be sufficiently smaller than the diamter (D) as high bandwidth values can lead to high bias in the KDEs. Once these values are chosen, all other $\mathcal{L}$ values can then be chosen to be equally spaced between $min(\mathcal{L})$ and $max(\mathcal{L})$.
An efficient way to choose ${l}_{min}$ and ${l}_{max}$ is to select the integers ${k}_{min}$ and ${k}_{max}$ and compute the ${k}_{min}$ and ${k}_{max}$ nearest neighbor distances of all the data points. The bandwidths $h({l}_{min})$ and $h({l}_{max})$ can then be chosen to be the maximums of these corresponding distances. The parameters ${l}_{min}$ and ${l}_{max}$ can then be computed from the expression $h(l)=l{N}^{\frac{1}{2d}}$. This choice ensures that a minimum of ${k}_{min}$ points are within the kernel bandwidth for the density estimates at all points and that a maximum of ${k}_{max}$ points are within the kernel bandwidth for the density estimates at one of the points.
Once $min(\mathcal{L})$ and $max(\mathcal{L})$ have been chosen, the similarity of bandwidth values $h(l)$ and basis functions ${\psi}_{i,d}(l)$ increases as L increases, resulting in a negligible decrease in the bias. Hence, L should be chosen to be large enough for sufficient bias but small enough so that the bandwidth values $h(l)$ are sufficiently distinct. In our experiments, we found $30\le L\le 60$ to be sufficient.
4.2. Convergence Rates Validation: Rényi$\alpha $ Divergence
To validate our theoretical convergence rate results, we estimated the Rényi$\alpha $ divergence integral between two truncated multivariate Gaussian distributions with varying dimension and sample sizes. The densities had means of ${\overline{\mu}}_{1}=0.7\times {\overline{1}}_{d}$, ${\overline{\mu}}_{2}=0.3\times {\overline{1}}_{d}$ and covariance matrices of $0.4\times {I}_{d}$, where ${\overline{1}}_{d}$ is a ddimensional vector of ones, and ${I}_{d}$ is a $d\times d$ identity matrix. We restricted the Gaussians to the unit cube and used $\alpha =0.5$.
The left plots in Figure 3 show the MSE (200 trials) of the standard plugin estimator implemented with a uniform kernel and the proposed optimally weighted estimator EnDive for various dimensions and sample sizes. The parameter set $\mathcal{L}$ was selected based on a range of knearest neighbor distances. The bandwidth used for the standard plugin estimator was selected by setting ${h}_{fixed}({l}^{\ast})={l}^{\ast}{N}^{\frac{1}{d+1}}$, where ${l}^{\ast}$ was chosen from $\mathcal{L}$ to minimize the MSE of the plugin estimator. For all dimensions and sample sizes, EnDive outperformed the plugin estimator in terms of MSE. EnDive was also less biased than the plugin estimator and even had lower variance at smaller sample sizes (e.g., $N=100$). This reflects the strength of ensemble estimators—the weighted sum of a set of relatively poor estimators can result in a very good estimator. Note also that for the larger values of N, the ensemble estimator MSE rates approached the theoretical rate based on the estimated log–log slope given in Table 1.
To illustrate the difference between the problems of density estimation and divergence functional estimation, we estimated the average pointwise squared error between the KDE ${\tilde{\mathbf{f}}}_{1,h}$ and ${f}_{1}$ in the previous experiment. We used exactly the same bandwidth and kernel as the standard plugin estimators in Figure 3 and calculated the pointwise error at 10,000 points sampled from ${f}_{1}$. The results are shown in Figure 4. From these results, we see that the KDEs performed worse as the dimension of the densities increased. Additionally, we observe by comparing Figure 3 and Figure 4, the average pointwise squared error decreased at a much slower rate as a function of the sample size (N) than the MSE of the plugin divergence estimators, especially for larger dimensions (d).
Our experiments indicated that the proposed ensemble estimator is not sensitive to the tuning parameters. See reference [72] for more details.
4.3. Central Limit Theorem Validation: KL Divergence
To verify the central limit theorem of the EnDive estimator, we estimated the KL divergence between two truncated Gaussian densities, again restricted to the unit cube. We conducted two experiments where (1) the densities were different with means of ${\overline{\mu}}_{1}=0.7\times {\overline{1}}_{d}$, ${\overline{\mu}}_{2}=0.3\times {\overline{1}}_{d}$ and covariances of matrices ${\sigma}_{i}\times {I}_{d}$, ${\sigma}_{1}=0.1$, ${\sigma}_{2}=0.3$; and where (2) the densities were the same with means of $0.3\times {\overline{1}}_{d}$ and covariance matrices of $0.3\times {I}_{d}$. For both experiments, we chose $d=6$ and four different sample sizes (N). We found that the correspondence between the quantiles of the standard normal distribution and the quantiles of the centered and scaled EnDive estimator was very high under all settings (see Table 2 and Figure 5) which validates Theorem 4.
4.4. Bayes Error Rate Estimation on SingleCell Data
Using the EnDive estimator, we estimated bounds on the Bayes error rate (BER) of a classification problem involving MARSseq singlecell RNAsequencing (scRNAseq) data measured from developing mouse bone marrow cells enriched for the myeloid and erythroid lineages [73]. However, we first demonstrated the ability of EnDive to estimate the bounds on the BER of a simulated problem. In this simulation, the data were drawn from two classes where each class distribution was a $d=10$ dimensional Gaussian distribution with different means and the identity covariance matrix. We considered two cases, namely, the distance between the means was 1 or 3. The BER was calculated in both cases. We then estimated upper and lower bounds on the BER by estimating the Henze–Penrose (HP) divergence [4,6]. Figure 6 shows the average estimated upper and lower bounds on the BER with standard error bars for both cases. For all tested sample sizes, the BER was within one standard deviation of the estimated lower bound. The lower bound was also closer, on average, to the BER for most of the tested sample sizes (lower sample sizes with smaller distances between means were the exceptions). Generally, these resuls indicate that the true BER is relatively close to the estimated lower bound, on average.
We then estimated similar bounds on the scRNAseq classification problem using EnDive. We considered the three most common cell types within the data: erythrocytes (eryth.), monocytes (mono.), and basophils (baso.) ($N=1095,\phantom{\rule{0.166667em}{0ex}}559,\phantom{\rule{0.166667em}{0ex}}300,$ respectively). We estimated the upper and lower bounds on the pairwise BER between these classes using different combinations of genes selected from the Kyoto Encyclopedia of Genes and Genomes (KEGG) pathways associated with the hematopoietic cell lineage [74,75,76]. Each collection of genes contained 11–14 genes. The upper and lower bounds on the BER were estimated using the Henze–Penrose divergence [4,6]. The standard deviations of the bounds for the KEGGbased genes were estimated via 1000 bootstrap iterations. The KEGGbased bounds were compared to BER bounds obtained from 1000 random selections of 12 genes. In all cases, we compared the bounds to the performance of a quadratic discriminant analysis classifier (QDA) with 10fold cross validation. Note that to correct for undersampling in scRNAseq data, we first imputed the undersampled data using MAGIC [77].
All results are given in Table 3. From these results, we note that erythrocytes are relatively easy to distinguish from the other two cell types as the BER lower bounds were within nearly two standard deviations of zero when using genes associated with platelet, erythrocyte, and neutrophil development as well as a random selection of 12 genes. This is corroborated by the QDA crossvalidated results which were all within two standard deviations of either the upper or lower bound for these gene sets. In contrast, the macrophageassociated genes seem to be less useful for distinguishing erythrocytes than the other gene sets.
We also found that basophils are difficult to distinguish from monocytes using these gene sets. Assuming the relative abundance of each cell type is representative of the population, a trivial upper bound on the BER is $300/(300+559)\approx 0.35$ which is between all of the estimated lower and upper bounds. The QDA results were also relatively high (and may be overfitting the data in some cases based on the estimated BER bounds), suggesting that different genes should be explored for this classification problem.
5. Conclusions
We derived the MSE convergence rates for a kernel density plugin estimator for a large class of divergence functionals. We generalized the theory of optimally weighted ensemble estimation and derived an ensemble divergence estimator EnDive that achieves the parametric rate when the densities are more than d times differentiable. The estimator we derived can be applied to general bounded density support sets and can be implemented without knowledge of the support, which is a distinct advantage over other competing estimators. We also derived the asymptotic distribution of the estimator, provided some guidelines for tuning parameter selection, and experimentally validated the theoretical convergence rates for the case of empirical estimation of the Rényi$\alpha $ divergence integral. We then performed experiments to examine the estimator’s robustness to the choice of tuning parameters, validated the central limit theorem for KL divergence estimation, and estimated bounds on the Bayes error rate for a single cell classification problem.
We note that based on the proof techniques employed in our work, our weighted ensemble estimators are easily extended beyond divergence estimation to more general distributional functionals which may be integral functionals of any number of probability distributions. We also show in Appendix B that EnDive can be easily modified to obtain an estimator that achieves the parametric rate when the densities are more than $d/2$ times differentiable and the functional g has a specific form that includes the Rényi and KL divergences. Future work includes extending this modification to functionals with more general forms. An important divergence of interest in this context is the Henze–Penrose divergence that we used to bound the Bayes error. Further future work will focus on extending this work on divergence estimation to knn based estimators where knowledge of the support is, again, not required. This will improve the computational burden, as knn estimators require fewer computations than standard KDEs.
Author Contributions
K.M. wrote this article primarily as part of his PhD dissertation under the supervision of A.H. and in collaboration with K.S. A.H, K.M., and K.S. edited the paper. K.S. provided the primary contribution to the proof of Theorem A1 and assisted with all other proofs. K.M. provided the primary contributions for the proofs of all other theorems and performed all other experiments. K.G. contributed to the bias proof.
Funding
This research was funded by Army Research Office (ARO) Multidisciplinary University Research Initiative (MURI) grant number W911NF1510479, National Science Foundation (NSF) grant number CCF1217880, and a National Science Foundation (NSF) Graduate Research Fellowship to the first author under grant number F031543.
Conflicts of Interest
The authors declare no conflict of interest. The founding sponsors had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, and in the decision to publish the results.
Abbreviations
The following abbreviations are used in this manuscript:
KL  Kullback–Leibler 
MSE  Mean squared error 
SVM  Support vector machine 
KDE  kernel density estimator 
EnDive  Ensemble Divergence 
BER  Bayes error rate 
scRNAseq  Singlecell RNAsequencing 
HP  Henze–Penrose 
Appendix A. Bias Assumptions
Our full assumptions to prove the bias expressions for the estimator ${\tilde{\mathbf{G}}}_{h}$ were as follows:
 $(\mathcal{A}.0)$: Assume that the kernel K is symmetric, is a product kernel, and has bounded support in each dimension.
 $(\mathcal{A}.1)$: Assume there exist constants ${\u03f5}_{0},{\u03f5}_{\infty}$, such that $0<{\u03f5}_{0}\le {f}_{i}(x)\le {\u03f5}_{\infty}<\infty ,\phantom{\rule{0.166667em}{0ex}}\forall x\in S.$
 $(\mathcal{A}.2)$: Assume that the densities are ${f}_{i}\in \mathsf{\Sigma}(s,{K}_{H})$ in the interior of $\mathcal{S}$ with $s\ge 2$.
 $(\mathcal{A}.3)$: Assume that g has an infinite number of mixed derivatives.
 $(\mathcal{A}.4$): Assume that $\left\frac{{\partial}^{k+l}g(x,y)}{\partial {x}^{k}\partial {y}^{l}}\right$, $k,l=0,1,\dots $ are strictly upper bounded for ${\u03f5}_{0}\le x,y\le {\u03f5}_{\infty}$.
 $(\mathcal{A}.5)$: Assume the following boundary smoothness condition: Let ${p}_{x}(u):{\mathbb{R}}^{d}\to \mathbb{R}$ be a polynomial in u of order $q\le r=\u230as\u230b$ whose coefficients are a function of x and are $rq$ times differentiable. Then, assume that$${\int}_{x\in \mathcal{S}}{\left({\int}_{u:K(u)>0,\phantom{\rule{0.166667em}{0ex}}x+uh\notin \mathcal{S}}K(u){p}_{x}(u)du\right)}^{t}dx={v}_{t}(h),$$$${v}_{t}(h)={\displaystyle \sum _{i=1}^{rq}}{e}_{i,q,t}{h}^{i}+o\left({h}^{rq}\right),$$
We focused on finite support kernels for simplicity in the proofs, although it is likely that our results extend to some infinitely supported kernels as well. We assumed relatively strong conditions on the smoothness of g in $\mathcal{A}.3$ to enable us to obtain an estimator that achieves good convergence rates without knowledge of the boundary of the support set. While this smoothness condition may seem restrictive, in practice, nearly all divergence and entropy functionals of interest satisfy this condition. Functionals of interest that do not satisfy this assumption (e.g., the total variation distance) typically have at least one point that is not differentiable which violates the assumptions of all competing estimators [54,57,58,59,60,65]. We also note that to obtain simply an upper bound on the bias for the plugin estimator, much less restrictive assumptions on the functional g are sufficient.
Assumption $\mathcal{A}.5$ requires the boundary of the density support set to be smooth with respect to the kernel ($K(u)$) in the sense that the expectation of the area outside of $\mathcal{S}$ with respect to any random variable u with smooth distribution is a smooth function of the bandwidth (h). Note that we do not require knowledge of the support of the unknown densities to actually implement the estimator (${\tilde{\mathbf{G}}}_{h}$). As long as assumptions $\mathcal{A}.0$–$\mathcal{A}.5$ are satisfied, then the bias results we obtain are valid, and therefore, we can obtain the parametric rate with the EnDive estimator. This is in contrast to many other estimators of information theoretic measures such as those presented in references [59,60,65]. In these cases, the boundary of the support set must be known precisely to perform boundary correction to obtain the parametric rate, since the boundary correction is an explicit step in these estimators. In contrast, we do not need to explicitly perform a boundary correction.
It is not necessary for the boundary of $\mathcal{S}$ to have smooth contours with no edges or corners as assumption $\mathcal{A}.5$ is satisfied by the following case:
Theorem A1.
Assumption $\mathcal{A}.5$ is satisfied when $\mathcal{S}={[1,1]}^{d}$ and when K is the uniform rectangular kernel; that is, $K(x)=1$ for all ${x:\phantom{\rule{0.166667em}{0ex}}\left\rightx\left\right}_{1}\le 1/2$.
The proof is given in Appendix D. The methods used to prove this can be easily extended to show that $\mathcal{A}.5$ is satisfied with the uniform rectangular kernel and other similar supports with flat surfaces and corners. Furthermore, we showed in reference [78] that $\mathcal{A}.5$ is satisfied using the uniform spherical kernel with a density support set equal to the unit cube. Note that assumption $\mathcal{A}.0$ is trivially satisfied by the uniform rectangular kernel as well. Again, this is easily extended to more complicated density support sets that have boundaries that contain flat surfaces and corners. Determining other combinations of kernels and density support sets that satisfy $\mathcal{A}.5$ is left for future work.
Densities for which assumptions $\mathcal{A}$.1–$\mathcal{A}.2$ hold include the truncated Gaussian distribution and the beta distribution on the unit cube. Functions for which assumptions $\mathcal{A}$.3–$\mathcal{A}.4$ hold include $g(x,y)=ln\left(\frac{x}{y}\right)$ and $g(x,y)={\left(\frac{x}{y}\right)}^{\alpha}.$
Appendix B. Modified EnDive
If the functional g has a specific form, we can modify the EnDive estimator to obtain an estimator that achieves the parametric rate when $s>d/2$. Specifically, we have the following theorem:
Theorem A2.
Assume that assumptions $\mathcal{A}$.0–$\mathcal{A}.5$ hold. Furthermore, if $g(x,y)$has $k,l$th order mixed derivatives $\frac{{\partial}^{k+l}g(x,y)}{\partial {x}^{k}\partial {y}^{l}}$ that depend on $x,y$ only through ${x}^{\alpha}{y}^{\beta}$ for some $\alpha ,\beta \in \mathbb{R}$, then for any positive integer $\lambda \ge 2$, the bias of ${\tilde{\mathbf{G}}}_{h}$ is
$$\begin{array}{ccc}\hfill \mathbb{B}\left[{\tilde{\mathbf{G}}}_{h}\right]& =& {\displaystyle \sum _{j=1}^{\u230as\u230b}}{c}_{10,j}{h}^{j}+{\displaystyle \sum _{q=1}^{\lambda /2}}{\displaystyle \sum _{j=0}^{\u230as\u230b}}{c}_{11,q,j}\frac{{h}^{j}}{{\left(N{h}^{d}\right)}^{q}}\hfill \\ & & +O\left({h}^{s}+\frac{1}{{\left(N{h}^{d}\right)}^{\frac{\lambda}{2}}}\right).\hfill \end{array}$$
Divergence functionals that satisfy the mixed derivatives condition required for (A1) include the KL divergence and the Rényi$\alpha $ divergence. Obtaining similar terms for other divergence functionals requires us to separate the dependence on h of the derivatives of g evaluated at ${\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{i,h}(\mathbf{Z})$. This is left for future work. See Appendix E for details.
As compared to (3), there are many more terms in (A1). These terms enable us to modify the EnDive estimator to achieve the parametric MSE convergence rate when $s>d/2$ for an appropriate choice of bandwidths, whereas the terms in (3) requires $s\ge d$ to achieve the same rate. This is accomplished by letting $h(l)$ decrease at a faster rate, as follows.
Let $\delta >0$ and $h(l)=l{N}^{\frac{1}{d+\delta}}$ where $l\in \mathcal{L}$. The bias of each estimator in the resulting ensemble has terms proportional to ${l}^{jdq}{N}^{\frac{j+q}{d+\delta}}$, where $j,q\ge 0$ and $j+q>0$. Then, the bias of ${\tilde{\mathbf{G}}}_{h(l)}$ satisfies condition $\mathcal{C}.1$ if ${\varphi}_{j,q,d}(N)={N}^{\frac{j+q}{d+\delta}}$, ${\psi}_{j,q}(l)={l}^{jdq}$, and
as long as $L>\leftJ\right=I$. The variance also satisfies condition $\mathcal{C}.2$. The optimal weight (${w}_{0}$) is found by using (6) to obtain an optimally weighted plugin divergence functional estimator ${\tilde{\mathbf{G}}}_{{w}_{0}}$ that achieves the parametric convergence rate if $\lambda \ge d/\delta +1$ and if $s\ge (d+\delta )/2$. Otherwise, if $s<(d+\delta )/2$, we can only guarantee the MSE rate up to $O\left(\frac{1}{{N}^{2s/(d+\delta )}}\right)$. We refer to this estimator as the modified EnDive estimator and denote it as ${\tilde{\mathbf{G}}}_{\mathrm{Mod}}$. The ensemble estimator ${\tilde{\mathbf{G}}}_{\mathrm{Mod}}$ is summarized in Algorithm 1 when $\delta =1$.
$$J=\left\{\{j,q\}:0<j+q<(d+1)/2,\phantom{\rule{0.166667em}{0ex}}q\in \{0,1,2,\dots ,\lambda /2\},\phantom{\rule{0.166667em}{0ex}}j\in \{0,1,2,\dots ,\u230as\u230b\}\right\},$$
Algorithm A1: The Modified EnDive Estimator 

The parametric rate can be achieved with ${\tilde{\mathbf{G}}}_{\mathrm{Mod}}$ under less strict assumptions on the smoothness of the densities than those required for ${\tilde{\mathbf{G}}}_{\mathrm{EnDive}}$. Since $\delta >0$ can be arbitrary, it is theoretically possible to achieve the parametric rate with the modified estimator as long as $s>d/2$. This is consistent with the rate achieved by the more complex estimators proposed in reference [57]. We also note that the central limit theorem applies and that the convergence is uniform as Theorem 5 applies for $s>\u230a(d+\delta )/2\u230b$ and $s\ge (d+\delta )/2$.
These rate improvements come at a cost for the number of parameters (L) required to implement the weighted ensemble estimator. If $s\ge \frac{d+\delta}{2}$, then the size of J for ${\tilde{\mathbf{G}}}_{\mathrm{Mod}}$ is in the order of $\frac{{d}^{2}}{8\delta}$. This may lead to increased variance in the ensemble estimator as indicated by (5).
So far, ${\tilde{\mathbf{G}}}_{\mathrm{Mod}}$ can only be applied to functionals ($g(x,y)$) with mixed derivatives of the form of ${x}^{\alpha}{y}^{\beta}$. Future work is required to extend this estimator to other functionals of interest.
Appendix C. General Results
Here we present the generalized forms of Theorems 1 and 2 where the sample sizes and bandwidths of the two datasets are allowed to differ. In this case, the KDEs are
where ${M}_{2}={N}_{2}1$. $G\left({f}_{1},{f}_{2}\right)$ is then approximated as
$$\begin{array}{ccc}\hfill {\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j})& =& \frac{1}{{N}_{1}{h}_{1}^{d}}{\displaystyle \sum _{i=1}^{{N}_{1}}}K\left(\frac{{\mathbf{X}}_{j}{\mathbf{Y}}_{i}}{{h}_{1}}\right),\hfill \\ \hfill {\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})& =& \frac{1}{{M}_{2}{h}_{2}^{d}}{\displaystyle \sum _{\begin{array}{c}i=1\\ i\ne j\end{array}}^{{N}_{2}}}K\left(\frac{{\mathbf{X}}_{j}{\mathbf{X}}_{i}}{{h}_{2}}\right),\hfill \end{array}$$
$${\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}=\frac{1}{{N}_{2}}{\displaystyle \sum _{i=1}^{{N}_{2}}}g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}\left({\mathbf{X}}_{i}\right),{\tilde{\mathbf{f}}}_{2,{h}_{2}}\left({\mathbf{X}}_{i}\right)\right).$$
We also generalize the bias result to the case where the kernel (K) has the order $\nu $ which means that the jth moment of the kernel ${K}_{i}$ defined as $\int {t}^{j}{K}_{i}(t)dt$ is zero for all $j=1,\dots ,\nu 1$ and $i=1,\dots ,d$ where ${K}_{i}$ is the kernel in the ith coordinate. Note that symmetric product kernels have the order $\nu \ge 2$. The following theorem on the bias follows under assumptions $\mathcal{A}$.0–$\mathcal{A}.5$:
Theorem A3.
For general g, the bias of the plugin estimator (${\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}$) is of the form
$$\begin{array}{ccc}\hfill \mathbb{B}\left[{\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}\right]& =& {\displaystyle \sum _{j=1}^{r}}\left({c}_{4,1,j}{h}_{1}^{j}+{c}_{4,2,j}{h}_{2}^{j}\right)+{\displaystyle \sum _{j=1}^{r}}{\displaystyle \sum _{i=1}^{r}}{c}_{5,i,j}{h}_{1}^{j}{h}_{2}^{i}+O\left({h}_{1}^{s}+{h}_{2}^{s}\right)\hfill \\ & & +{c}_{9,1}\frac{1}{{N}_{1}{h}_{1}^{d}}+{c}_{9,2}\frac{1}{{N}_{2}{h}_{2}^{d}}+o\left(\frac{1}{{N}_{1}{h}_{1}^{d}}+\frac{1}{{N}_{2}{h}_{2}^{d}}\right).\hfill \end{array}$$
Furthermore, if $g(x,y)$ has $k,l$th order mixed derivatives $\frac{{\partial}^{k+l}g(x,y)}{\partial {x}^{k}\partial {y}^{l}}$ that depend on $x,y$ only through ${x}^{\alpha}{y}^{\beta}$ for some $\alpha ,\beta \in \mathbb{R}$, then for any positive integer $\lambda \ge 2$, the bias is of the form
$$\begin{array}{ccc}\hfill \mathbb{B}\left[{\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}\right]& =& {\displaystyle \sum _{j=1}^{r}}\left({c}_{4,1,j}{h}_{1}^{j}+{c}_{4,2,j}{h}_{2}^{j}\right)+{\displaystyle \sum _{j=1}^{r}}{\displaystyle \sum _{i=1}^{r}}{c}_{5,i,j}{h}_{1}^{j}{h}_{2}^{i}+O\left({h}_{1}^{s}+{h}_{2}^{s}\right)\hfill \\ & & {\displaystyle \sum _{j=1}^{\lambda /2}}{\displaystyle \sum _{m=0}^{r}}\left({c}_{9,1,j,m}\frac{{h}_{1}^{m}}{{\left({N}_{1}{h}_{1}^{d}\right)}^{j}}+{c}_{9,2,j,m}\frac{{h}_{2}^{m}}{{\left({N}_{2}{h}_{2}^{d}\right)}^{j}}\right)\hfill \\ & & +{\displaystyle \sum _{j=1}^{\lambda /2}}{\displaystyle \sum _{m=0}^{r}}{\displaystyle \sum _{i=1}^{\lambda /2}}{\displaystyle \sum _{n=0}^{r}}{c}_{9,j,i,m,n}\frac{{h}_{1}^{m}{h}_{2}^{n}}{{\left({N}_{1}{h}_{1}^{d}\right)}^{j}{\left({N}_{2}{h}_{2}^{d}\right)}^{i}}\hfill \\ & & +O\left(\frac{1}{{\left({N}_{1}{h}_{1}^{d}\right)}^{\frac{\lambda}{2}}}+\frac{1}{{\left({N}_{2}{h}_{2}^{d}\right)}^{\frac{\lambda}{2}}}\right).\hfill \end{array}$$
Note that the bandwidth and sample size terms do not depend on the order of the kernel ($\nu $). Thus, using a higherorder kernel does not provide any benefit to the convergence rates. This lack of improvement is due to the bias of the density estimators at the boundary of the density support sets. To obtain better convergence rates using higherorder kernels, boundary correction would be necessary [57,60]. In contrast, we improve the convergence rates by using a weighted ensemble that does not require boundary correction.
The variance result requires much less strict assumptions than the bias results:
Theorem A4.
Assume that the functionalg in (1) is Lipschitz continuous in both of its arguments with the Lipschitz constant ${C}_{g}$. Then, the variance of the plugin estimator (${\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}$) is bounded by
$$\mathbb{V}\left[{\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}\right]\le {C}_{g}^{2}{\left\rightK\left\right}_{\infty}^{2}\left(\frac{10}{{N}_{2}}+\frac{{N}_{1}}{{N}_{2}^{2}}\right).$$
The proofs of these theorems are in Appendix E and Appendix F. Theorems 1 and 2 then follow.
Appendix D. Proof of Theorem A1 (Boundary Conditions)
Consider a uniform rectangular kernel $K(x)$ that satisfies $K(x)=1$ for all x, such that ${\left\rightx\left\right}_{1}\le 1/2$. Also, consider the family of probability densities (f) with rectangular support $\mathcal{S}={[1,1]}^{d}$. We prove Theorem A1 which is that that $\mathcal{S}$ satisfies the following smoothness condition $(\mathcal{A}.5$): for any polynomial ${p}_{x}(u):{\mathbb{R}}^{d}\to \mathbb{R}$ of order $q\le r=\u230as\u230b$ with coefficients that are $rq$ times differentiable wrt x,
where ${v}_{t}(h)$ has the expansion
$${\int}_{x\in \mathcal{S}}{\left({\int}_{{u:\left\rightu\left\right}_{1}\le \frac{1}{2},\phantom{\rule{0.166667em}{0ex}}x+uh\notin \mathcal{S}}{p}_{x}(u)du\right)}^{t}dx={v}_{t}(h),$$
$${v}_{t}(h)={\displaystyle \sum _{i=1}^{rq}}{e}_{i,q,t}{h}^{i}+o\left({h}^{rq}\right).$$
Note that the inner integral forces the xs under consideration to be boundary points via the constraint $x+uh\notin \mathcal{S}$.
Appendix D.1. Single Coordinate Boundary Point
We begin by focusing on points x that are boundary points by virtue of a single coordinate ${x}_{i}$, such that ${x}_{i}+{u}_{i}h\notin \mathcal{S}$. Without loss of generality, assume that ${x}_{i}+{u}_{i}h>1$. The inner integral in (A6) can then be evaluated first with respect to (wrt) all coordinates other than i. Since all of these coordinates lie within the support, the inner integral over these coordinates will amount to integration of the polynomial ${p}_{x}(u)$ over a symmetric $d1$ dimensional rectangular region $\left{u}_{j}\right\le \frac{1}{2}$ for all $j\ne i$. This yields a function ${\sum}_{m=1}^{q}{\tilde{p}}_{m}(x){u}_{i}^{m}$ where the coefficients ${\tilde{p}}_{m}(x)$ are each $rq$ times differentiable wrt x.
With respect to the ${u}_{i}$ coordinate, the inner integral will have limits from $\frac{1{x}_{i}}{h}$ to $\frac{1}{2}$ for some $1>{x}_{i}>1\frac{h}{2}$. Consider the ${\tilde{p}}_{q}(x){u}_{i}^{q}$ monomial term. The inner integral wrt this term yields
$$\sum _{m=1}^{q}}{\tilde{p}}_{m}(x){\int}_{\frac{1{x}_{i}}{h}}^{\frac{1}{2}}{u}_{i}^{m}d{u}_{i}={\displaystyle \sum _{m=1}^{q}}{\tilde{p}}_{m}(x)\frac{1}{m+1}\left(\frac{1}{{2}^{m+1}}{\left(\frac{1{x}_{i}}{h}\right)}^{m+1}\right).$$
Raising the righthandside of (A7) to the power of t results in an expression of the form
where the coefficients ${\stackrel{\u02c7}{p}}_{j}(x)$ are $rq$ times differentiable wrt x. Integrating (A8) over all the coordinates in x other than ${x}_{i}$ results in an expression of the form
where, again, the coefficients ${\overline{p}}_{j}({x}_{i})$ are $rq$ times differentiable wrt ${x}_{i}$. Note that since the other coordinates of x other than ${x}_{i}$ are far away from the boundary, the coefficients ${\overline{p}}_{j}({x}_{i})$ are independent of h. To evaluate the integral of (A9), consider the $rq$ term Taylor series expansion of ${\overline{p}}_{j}({x}_{i})$ around ${x}_{i}=1$. This will yield terms of the form
for $0\le j\le rq$, and $0\le k\le qt$. Combining terms results in the expansion ${v}_{t}(h)={\sum}_{i=1}^{rq}{e}_{i,q,t}{h}^{i}+o\left({h}^{rq}\right)$.
$$\sum _{j=0}^{qt}}{\stackrel{\u02c7}{p}}_{j}(x){\left(\frac{1{x}_{i}}{h}\right)}^{j},$$
$$\sum _{j=0}^{qt}}{\overline{p}}_{j}({x}_{i}){\left(\frac{1{x}_{i}}{h}\right)}^{j},$$
$$\begin{array}{ccc}\hfill {\int}_{1h/2}^{1}\frac{{\left(1{x}_{i}\right)}^{j+k}}{{h}^{k}}d{x}_{i}& =& {\left.\frac{{\left(1{x}_{i}\right)}^{j+k+1}}{{h}^{k}(j+k+1)}\right}_{{x}_{i}=1h/2}^{{x}_{i}=1}\hfill \\ & =& \frac{{h}^{j+1}}{(j+k+1){2}^{j+k+1}},\hfill \end{array}$$
Appendix D.2. Multiple Coordinate Boundary Point
The case where multiple coordinates of point x are near the boundary is a straightforward extension of the single boundary point case, so we only sketch the main ideas here. As an example, consider the case where two of the coordinates are near the boundary. Assume for notational ease that they are ${x}_{1}$ and ${x}_{2}$ and that ${x}_{1}+{u}_{1}h>1$ and ${x}_{2}+{u}_{2}h>1$. The inner integral in (A6) can again be evaluated first wrt all coordinates other than 1 and 2. This yields a function ${\sum}_{m,j=1}^{q}{\tilde{p}}_{m,j}(x){u}_{1}^{m}{u}_{2}^{j}$ where the coefficients ${\tilde{p}}_{m,j}(x)$ are each $rq$ times differentiable wrt x. Integrating this wrt ${x}_{1}$ and ${x}_{2}$ and then raising the result to the power of t yields a double sum similar to (A8). Integrating this over all the coordinates in x other than ${x}_{1}$ and ${x}_{2}$ gives a double sum similar to (A9). Then, a Taylor series expansion of the coefficients and integration over ${x}_{1}$ and ${x}_{2}$ yields the result.
Appendix E. Proof of Theorem A3 (Bias)
In this appendix, we prove the bias results in Theorem A3. The bias of the base kernel density plugin estimator ${\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}$ can be expressed as
where $\mathbf{Z}$ is drawn from ${f}_{2}$. The first term is the “variance” term, while the second is the “bias” term. We bound these terms using Taylor series expansions under the assumption that g is infinitely differentiable. The Taylor series expansion of the variance term in (A11) will depend on variancelike terms of the KDEs, while the Taylor series expansion of the bias term in (A11) will depend on the bias of the KDEs.
$$\begin{array}{ccc}\hfill \mathbb{B}\left[{\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}\right]& =& \mathbb{E}\left[g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\right)g\left({f}_{1}(\mathbf{Z}),{f}_{2}(\mathbf{Z})\right)\right]\hfill \\ & =& \mathbb{E}\left[g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\right)g\left({\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z}),{\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\right)\right]\hfill \\ & & +\mathbb{E}\left[g\left({\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z}),{\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\right)g\left({f}_{1}(\mathbf{Z}),{f}_{2}(\mathbf{Z})\right)\right],\hfill \end{array}$$
The Taylor series expansion of $g\left({\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z}),{\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\right)$ around ${f}_{1}(\mathbf{Z})$ and ${f}_{2}(\mathbf{Z})$ is
where ${\mathbb{B}}_{\mathbf{Z}}^{j}\left[{\tilde{\mathbf{f}}}_{i,{h}_{i}}(\mathbf{Z})\right]={\left({\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{i,{h}_{i}}(\mathbf{Z}){f}_{i}(\mathbf{Z})\right)}^{j}$ is the bias of ${\tilde{\mathbf{f}}}_{i,{h}_{i}}$ at the point $\mathbf{Z}$ raised to the power of j. This expansion can be used to control the second term (the bias term) in (A11). To accomplish this, we require an expression for ${\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{i,{h}_{i}}(\mathbf{Z}){f}_{i}(\mathbf{Z})={\mathbb{B}}_{\mathbf{Z}}\left[{\tilde{\mathbf{f}}}_{i,{h}_{i}}(\mathbf{Z})\right]$.
$$\begin{array}{c}\hfill g\left({\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z}),{\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\right)={\displaystyle \sum _{i=0}^{\infty}}{\displaystyle \sum _{j=0}^{\infty}}\left({\left.\frac{{\partial}^{i+j}g(x,y)}{\partial {x}^{i}\partial {y}^{j}}\right}_{\begin{array}{c}x={f}_{1}(\mathbf{Z})\\ y={f}_{2}(\mathbf{Z})\end{array}}\right)\frac{{\mathbb{B}}_{\mathbf{Z}}^{i}\left[{\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z})\right]{\mathbb{B}}_{\mathbf{Z}}^{j}\left[{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\right]}{i!j!},\end{array}$$
To obtain an expression for ${\mathbb{B}}_{\mathbf{Z}}\left[{\tilde{\mathbf{f}}}_{i,{h}_{i}}(\mathbf{Z})\right]$, we separately consider the cases when $\mathbf{Z}$ is in the interior of the support $\mathcal{S}$ or when $\mathbf{Z}$ is near the boundary of the support. A point $X\in \mathcal{S}$ is defined to be in the interior of $\mathcal{S}$ if for all $Y\notin \mathcal{S}$, $K\left(\frac{XY}{{h}_{i}}\right)=0$. A point $X\in \mathcal{S}$ is near the boundary of the support if it is not in the interior. Denote the region in the interior and near the boundary wrt ${h}_{i}$ as ${\mathcal{S}}_{{I}_{i}}$ and ${\mathcal{S}}_{{B}_{i}}$, respectively. We will need the following:
Lemma A1.
Let $\mathbf{Z}$ be a realization of the density ${f}_{2}$ independent of ${\tilde{\mathbf{f}}}_{i,{h}_{i}}$ for $i=1,2$. Assume that the densities ${f}_{1}$ and ${f}_{2}$ belong to $\mathsf{\Sigma}(s,L)$. Then, for $\mathbf{Z}\in {\mathcal{S}}_{{I}_{i}}$,
$${\mathbb{E}}_{\mathbf{Z}}\left[{\tilde{\mathbf{f}}}_{i,{h}_{i}}(\mathbf{Z})\right]={f}_{i}(\mathbf{Z})+{\displaystyle \sum _{j=\nu /2}^{\u230as/2\u230b}}{c}_{i,j}(\mathbf{Z}){h}_{i}^{2j}+O\left({h}_{i}^{s}\right).$$
Proof.
Obtaining the lower order terms in (A12) is a common result in kernel density estimation. However, since we also require the higher order terms, we present the proof here. Additionally, some of the results in this proof will be useful later. From the linearity of the KDE, we have that if $\mathbf{X}$ is drawn from ${f}_{i}$ and is independent of $\mathbf{Z}$, then
where the last step follows on from the substitution $t=\frac{x\mathbf{Z}}{{h}_{i}}$. Since the density (${f}_{i}$) belongs to $\mathsf{\Sigma}(s,K)$, by using multiindex notation we can expand it to
where $\alpha !={\alpha}_{1}!{\alpha}_{2}!\dots {\alpha}_{d}!$ and ${Y}^{\alpha}={Y}_{1}^{{\alpha}_{1}}{Y}_{2}^{{\alpha}_{2}}\dots {Y}_{d}^{{\alpha}_{d}}$. Combining (A13) and (A14) gives
where the last step follows from the fact that K is symmetric and of order $\nu $. ☐
$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{i,{h}_{i}}(\mathbf{Z})& =& {\mathbb{E}}_{\mathbf{Z}}\left[\frac{1}{{h}_{i}^{d}}K\left(\frac{\mathbf{X}\mathbf{Z}}{{h}_{i}}\right)\right]\hfill \\ & =& \int \frac{1}{{h}_{i}^{d}}K\left(\frac{x\mathbf{Z}}{{h}_{i}}\right){f}_{i}(x)dx\hfill \\ & =& \int K\left(t\right){f}_{i}(t{h}_{i}+\mathbf{Z})dt,\hfill \end{array}$$
$${f}_{i}(t{h}_{i}+\mathbf{Z})={f}_{i}(\mathbf{Z})+{\displaystyle \sum _{0<\left\alpha \right\le \u230as\u230b}}\frac{{D}^{\alpha}{f}_{i}(\mathbf{Z})}{\alpha !}{\left(t{h}_{i}\right)}^{\alpha}+O\left({\u2225t{h}_{i}\u2225}^{s}\right),$$
$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{i,{h}_{i}}(\mathbf{Z})& =& {f}_{i}(\mathbf{Z})+{\displaystyle \sum _{0<\left\alpha \right\le \u230as\u230b}}\frac{{D}^{\alpha}{f}_{i}(\mathbf{Z})}{\alpha !}{h}_{i}^{\left\alpha \right}\int {t}^{\alpha}K(t)dt+O({h}_{i}^{s})\hfill \\ & =& {f}_{i}(\mathbf{Z})+{\displaystyle \sum _{j=\nu /2}^{\u230as/2\u230b}}{c}_{i,j}(\mathbf{Z}){h}_{i}^{2j}+O({h}_{i}^{s}),\hfill \end{array}$$
To obtain a similar result for the case when $\mathbf{Z}$ is near the boundary of $\mathcal{S}$, we use the assumption $\mathcal{A}.5$.
Lemma A2.
Let $\gamma (x,y)$ be an arbitrary function satisfying ${sup}_{x,y}\left\gamma (x,y)\right<\infty $. Let $\mathcal{S}$ satisfy the boundary smoothness conditions of Assumption $\mathcal{A}.5$. Assume that the densities ${f}_{1}$ and ${f}_{2}$ belong to $\mathsf{\Sigma}(s,L)$, and let $\mathbf{Z}$ be a realization of the ${f}_{2}$ density independently of ${\tilde{\mathbf{f}}}_{i,{h}_{i}}$ for $i=1,2$. Let ${h}^{{}^{\prime}}=min\left({h}_{1},{h}_{2}\right)$. Then,
$$\begin{array}{ccc}\hfill \mathbb{E}\left[{1}_{\left\{\mathbf{Z}\in {\mathcal{S}}_{{B}_{i}}\right\}}\gamma \left({f}_{1}(\mathbf{Z}),{f}_{2}(\mathbf{Z})\right){\mathbb{B}}_{\mathbf{Z}}^{t}\left[{\tilde{\mathbf{f}}}_{i,{h}_{i}}(\mathbf{Z})\right]\right]& =& {\displaystyle \sum _{j=1}^{r}}{c}_{4,i,j,t}{h}_{i}^{j}+o\left({h}_{i}^{r}\right)\hfill \end{array}$$
$$\begin{array}{ccc}\hfill \mathbb{E}\left[{1}_{\left\{\mathbf{Z}\in {\mathcal{S}}_{{B}_{1}}\cap {\mathcal{S}}_{{B}_{2}}\right\}}\gamma \left({f}_{1}(\mathbf{Z}),{f}_{2}(\mathbf{Z})\right){\mathbb{B}}_{\mathbf{Z}}^{t}\left[{\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z})\right]{\mathbb{B}}_{\mathbf{Z}}^{q}\left[{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\right]\right]& =& {\displaystyle \sum _{j=0}^{r1}}{\displaystyle \sum _{i=0}^{r1}}{c}_{4,j,i,q,t}{h}_{1}^{j}{h}_{2}^{i}{h}^{{}^{\prime}}+o\left({\left({h}^{{}^{\prime}}\right)}^{r}\right)\hfill \end{array}$$
Proof.
For a fixed X near the boundary of $\mathcal{S}$, we have
$$\begin{array}{ccc}\hfill \mathbb{E}\left[{\tilde{\mathbf{f}}}_{i,{h}_{i}}(X)\right]{f}_{i}(X)& =& \frac{1}{{h}_{i}^{d}}{\int}_{Y:Y\in \mathcal{S}}K\left(\frac{XY}{{h}_{i}}\right){f}_{i}(Y)dY{f}_{i}(X)\hfill \\ & =& \left[\frac{1}{{h}_{i}^{d}}{\int}_{Y:K\left(\frac{XY}{{h}_{i}}\right)>0}K\left(\frac{XY}{{h}_{i}}\right){f}_{i}(Y)dY{f}_{i}(X)\right]\hfill \\ & & \left[\frac{1}{{h}_{i}^{d}}{\int}_{Y:Y\notin \mathcal{S}}K\left(\frac{XY}{{h}_{i}}\right){f}_{i}(Y)dY\right]\hfill \\ & =& {T}_{1,i}(X){T}_{2,i}(X).\hfill \end{array}$$
Note that, in ${T}_{1,i}(X)$, we are extending the integral beyond the support of the ${f}_{i}$ density. However, by using the same Taylor series expansion method as in the proof of Lemma A1, we always evaluate ${f}_{i}$ and its derivatives at point X which is within the support of ${f}_{i}$. Thus, it does not matter how we define an extension of ${f}_{i}$ since the Taylor series will remain the same. Thus, ${T}_{1,i}(X)$ results in an identical expression to that obtained from (A12).
For the ${T}_{2,i}(X)$ term, we expand it using multiindex notation as
$$\begin{array}{ccc}\hfill {T}_{2,i}(X)& =& \frac{1}{{h}_{i}^{d}}{\int}_{Y:Y\notin \mathcal{S}}K\left(\frac{XY}{{h}_{i}}\right){f}_{i}(Y)dY\hfill \\ & =& {\int}_{u:{h}_{i}u+X\notin \mathcal{S},K(u)>0}K\left(u\right){f}_{i}(X+{h}_{i}u)du\hfill \\ & =& {\displaystyle \sum _{\left\alpha \right\le r}}\frac{{h}_{i}^{\left\alpha \right}}{\alpha !}{\int}_{u:{h}_{i}u+X\notin \mathcal{S},K(u)>0}K\left(u\right){D}^{\alpha}{f}_{i}(X){u}^{\alpha}du+o\left({h}_{i}^{r}\right).\hfill \end{array}$$
Recognizing that the $\left\alpha \right$th derivative of ${f}_{i}$ is $r\left\alpha \right$ times differentiable, we can apply assumption $\mathcal{A}.5$ to obtain the expectation of ${T}_{2,i}(X)$ wrt X:
$$\begin{array}{ccc}\hfill \mathbb{E}\left[{T}_{2,i}(\mathbf{X})\right]& =& \frac{1}{{h}_{i}^{d}}{\int}_{X}{\int}_{Y:Y\notin \mathcal{S}}K\left(\frac{XY}{{h}_{i}}\right){f}_{i}(Y)dY{f}_{2}(X)dx\hfill \\ & =& {\displaystyle \sum _{\left\alpha \right\le r}}\frac{{h}_{i}^{\left\alpha \right}}{\alpha !}{\int}_{X}{\int}_{u:{h}_{i}u+X\notin \mathcal{S},K(u)>0}K\left(u\right){D}^{\alpha}{f}_{i}(X){u}^{\alpha}du{f}_{2}(X)dX+o\left({h}_{i}^{r}\right)\hfill \\ & =& {\displaystyle \sum _{\left\alpha \right\le r}}\frac{{h}_{i}^{\left\alpha \right}}{\alpha !}\left[{\displaystyle \sum _{1\le \left\beta \right\le r\left\alpha \right}}{e}_{\beta ,r\left\alpha \right}{h}_{i}^{\left\beta \right}+o\left({h}_{i}^{r\left\alpha \right}\right)\right]+o\left({h}_{i}^{r}\right)\hfill \\ & =& {\displaystyle \sum _{j=1}^{r}}{e}_{j}{h}_{i}^{j}+o\left({h}_{i}^{r}\right).\hfill \end{array}$$
Similarly, we find that
$$\begin{array}{ccc}\hfill \mathbb{E}\left[{\left({T}_{2,i}(\mathbf{X})\right)}^{t}\right]& =& \frac{1}{{h}_{i}^{dt}}{\int}_{X}{\left({\int}_{Y:Y\notin \mathcal{S}}K\left(\frac{XY}{{h}_{i}}\right){f}_{i}(Y)dY\right)}^{t}{f}_{2}(X)dx\hfill \\ & =& {\int}_{X}{\left({\displaystyle \sum _{\left\alpha \right\le r}}\frac{{h}_{i}^{\left\alpha \right}}{\alpha !}{\int}_{u:{h}_{i}u+X\notin \mathcal{S},K(u)>0}K\left(u\right){D}^{\alpha}{f}_{i}(X){u}^{\alpha}du\right)}^{t}{f}_{2}(X)dX\hfill \\ & =& {\displaystyle \sum _{j=1}^{r}}{e}_{j,t}{h}_{i}^{j}+o\left({h}_{i}^{r}\right).\hfill \end{array}$$
Combining these results gives
where the constants are functionals of the kernel $\gamma $ and the densities.
$$\begin{array}{ccc}\hfill \mathbb{E}\left[{1}_{\left\{\mathbf{Z}\in {\mathcal{S}}_{B}\right\}}\gamma \left({f}_{1}(\mathbf{Z}),{f}_{2}(\mathbf{Z})\right){\left({\mathbb{E}}_{\mathbf{Z}}\left[{\tilde{\mathbf{f}}}_{i,{h}_{i}}(\mathbf{Z})\right]{f}_{i}(\mathbf{Z})\right)}^{t}\right]& =& \mathbb{E}\left[\gamma \left({f}_{1}(\mathbf{Z}),{f}_{2}(\mathbf{Z})\right){\left({T}_{1,i}(\mathbf{Z}){T}_{2,i}(\mathbf{Z})\right)}^{t}\right]\hfill \\ & =& \mathbb{E}\left[\gamma \left({f}_{1}(\mathbf{Z}),{f}_{2}(\mathbf{Z})\right){\displaystyle \sum _{j=0}^{t}}\left(\genfrac{}{}{0pt}{}{t}{j}\right){\left({T}_{1,i}(\mathbf{Z})\right)}^{j}{\left({T}_{2,i}(\mathbf{Z})\right)}^{tj}\right]\hfill \\ & =& {\displaystyle \sum _{j=1}^{r}}{c}_{4,i,j,t}{h}_{i}^{j}+o\left({h}_{i}^{r}\right),\hfill \end{array}$$
The expression in (A16) can be proved in a similar manner. ☐
Applying Lemmas A1 and A2 to (A11) gives
$$\mathbb{E}\left[g\left({\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z}),{\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\right)g\left({f}_{1}(\mathbf{Z}),{f}_{2}(\mathbf{Z})\right)\right]={\displaystyle \sum _{j=1}^{r}}\left({c}_{4,1,j}{h}_{1}^{j}+{c}_{4,2,j}{h}_{2}^{j}\right)+{\displaystyle \sum _{j=0}^{r1}}{\displaystyle \sum _{i=0}^{r1}}{c}_{5,i,j}{h}_{1}^{j}{h}_{2}^{i}{h}^{{}^{\prime}}+o\left({h}_{1}^{r}+{h}_{2}^{r}\right).$$
For the variance term (the first term) in (A10), the truncated Taylor series expansion of $g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\right)$ around ${\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z})$ and ${\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})$ gives
where ${\tilde{\mathbf{e}}}_{i,{h}_{i}}(\mathbf{Z}):={\tilde{\mathbf{f}}}_{i,{h}_{i}}(\mathbf{Z}){\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{i,{h}_{i}}(\mathbf{Z})$. To control the variance term in (A11), we thus require expressions for ${\mathbb{E}}_{\mathbf{Z}}\left[{\tilde{\mathbf{e}}}_{i,{h}_{i}}^{j}(\mathbf{Z})\right]$.
$$\begin{array}{ccc}\hfill g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\right)& =& {\displaystyle \sum _{i=0}^{\lambda}}{\displaystyle \sum _{j=0}^{\lambda}}\left({\left.\frac{{\partial}^{i+j}g(x,y)}{\partial {x}^{i}\partial {y}^{j}}\right}_{\begin{array}{c}x={\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z})\\ y={\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\end{array}}\right)\frac{{\tilde{\mathbf{e}}}_{1,{h}_{1}}^{i}(\mathbf{Z}){\tilde{\mathbf{e}}}_{2,{h}_{2}}^{j}(\mathbf{Z})}{i!j!}+o\left({\tilde{\mathbf{e}}}_{1,{h}_{1}}^{\lambda}(\mathbf{Z})+{\tilde{\mathbf{e}}}_{2,{h}_{2}}^{\lambda}(\mathbf{Z})\right)\hfill \end{array}$$
Lemma A3.
Let $\mathbf{Z}$ be a realization of the ${f}_{2}$ density that is in the interior of the support and is independent of ${\tilde{\mathbf{f}}}_{i,{h}_{i}}$ for $i=1,2$. Let $n(q)$ be the set of integer divisors of q including 1 but excluding q. Then,
where ${c}_{6,i,q,j,m}$ is a functional of ${f}_{1}$ and ${f}_{2}.$
$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\mathbf{Z}}\left[{\tilde{\mathbf{e}}}_{i,{h}_{i}}^{q}(\mathbf{Z})\right]& =& \left\{\begin{array}{cc}{\displaystyle {\sum}_{j\in n(q)}}\frac{1}{{\left({N}_{2}{h}_{2}^{d}\right)}^{qj}}{\displaystyle {\sum}_{m=0}^{\u230as/2\u230b}}{c}_{6,i,q,j,m}(\mathbf{Z}){h}_{i}^{2m}+O\left(\frac{1}{{N}_{i}}\right),\hfill & q\ge 2\hfill \\ 0,\hfill & q=1,\hfill \end{array}\right.\hfill \end{array}$$
$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\mathbf{Z}}\left[{\tilde{\mathbf{e}}}_{1,{h}_{1}}^{q}(\mathbf{Z}){\tilde{\mathbf{e}}}_{2,{h}_{2}}^{l}(\mathbf{Z})\right]& =& \left\{\begin{array}{cc}\left({\displaystyle {\sum}_{i\in n(q)}}\frac{1}{{\left({N}_{1}{h}_{1}^{d}\right)}^{qi}}{\displaystyle {\sum}_{m=0}^{\u230as/2\u230b}}{c}_{6,1,q,i,m}(\mathbf{Z}){h}_{1}^{2m}\right)\times q,\hfill & \phantom{\rule{0.166667em}{0ex}}l\ge 2\hfill \\ \left({\displaystyle {\sum}_{j\in n(l)}}\frac{1}{{\left({N}_{2}{h}_{2}^{d}\right)}^{lj}}{\displaystyle {\sum}_{t=0}^{\u230as/2\u230b}}{c}_{6,2,l,j,t}(\mathbf{Z}){h}_{2}^{2t}\right)+O\left(\frac{1}{{N}_{1}}+\frac{1}{{N}_{2}}\right),\hfill \\ 0,\hfill & q=1\phantom{\rule{0.166667em}{0ex}}\mathrm{or}\phantom{\rule{4.pt}{0ex}}l=1\hfill \end{array}\right.\hfill \end{array}$$
Proof.
Define the random variable ${\mathbf{V}}_{i}(\mathbf{Z})=K\left(\frac{{\mathbf{X}}_{i}\mathbf{Z}}{{h}_{2}}\right){\mathbb{E}}_{\mathbf{Z}}K\left(\frac{{\mathbf{X}}_{i}\mathbf{Z}}{{h}_{2}}\right)$. This gives
$$\begin{array}{ccc}\hfill {\tilde{\mathbf{e}}}_{2,{h}_{2}}(\mathbf{Z})& =& {\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z}){\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\hfill \\ & =& \frac{1}{{N}_{2}{h}_{2}^{d}}{\displaystyle \sum _{i=1}^{{N}_{2}}}{\mathbf{V}}_{i}(\mathbf{Z}).\hfill \end{array}$$
Clearly, ${\mathbb{E}}_{\mathbf{Z}}{\mathbf{V}}_{i}(\mathbf{Z})=0$. From (A13), we have for integer $j\ge 1$
where the constants ${c}_{3,2,j,m}$ depend on density ${f}_{2}$, its derivatives, and the moments of kernel ${K}^{j}$. Note that since K is symmetric, the odd moments of ${K}^{j}$ are zero for $\mathbf{Z}$ in the interior of the support. However, all even moments may now be nonzero since ${K}^{j}$ may now be nonnegative. In accordance with the binomial theorem,
$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\mathbf{Z}}\left[{K}^{j}\left(\frac{{\mathbf{X}}_{i}\mathbf{Z}}{{h}_{2}}\right)\right]& =& \int {K}^{j}\left(t\right){f}_{2}(t{h}_{2}+\mathbf{Z})dt\hfill \\ & =& {h}_{2}^{d}{\displaystyle \sum _{m=0}^{\u230as/2\u230b}}{c}_{3,2,j,m}(\mathbf{Z}){h}_{2}^{2m},\hfill \end{array}$$
$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\mathbf{Z}}\left[{\mathbf{V}}_{i}^{j}(\mathbf{Z})\right]& =& {\displaystyle \sum _{k=0}^{j}}\left(\genfrac{}{}{0pt}{}{j}{k}\right){\mathbb{E}}_{\mathbf{Z}}\left[{K}^{k}\left(\frac{{\mathbf{X}}_{i}\mathbf{Z}}{{h}_{2}}\right)\right]{\mathbb{E}}_{\mathbf{Z}}{\left[K\left(\frac{{\mathbf{X}}_{i}\mathbf{Z}}{{h}_{2}}\right)\right]}^{jk}\hfill \\ & =& {\displaystyle \sum _{k=0}^{j}}\left(\genfrac{}{}{0pt}{}{j}{k}\right){h}_{2}^{d}O\left({h}_{2}^{d(jk)}\right){\displaystyle \sum _{m=0}^{\u230as/2\u230b}}{c}_{3,2,k,m}(\mathbf{Z}){h}_{2}^{2m}\hfill \\ & =& {h}_{2}^{d}{\displaystyle \sum _{m=0}^{\u230as/2\u230b}}{c}_{3,2,j,m}(\mathbf{Z}){h}_{2}^{2m}+O\left({h}^{2d}\right).\hfill \end{array}$$
We can use these expressions to simplify ${\mathbb{E}}_{\mathbf{Z}}\left[{\tilde{\mathbf{e}}}_{2,{h}_{2}}^{q}(\mathbf{Z})\right]$. As an example, let $q=2$. Then, since the ${\mathbf{X}}_{i}s$ are independent,
$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\mathbf{Z}}\left[{\tilde{\mathbf{e}}}_{2,{h}_{2}}^{2}(\mathbf{Z})\right]& =& \frac{1}{{N}_{2}{h}_{2}^{2d}}{\mathbb{E}}_{\mathbf{Z}}{\mathbf{V}}_{i}^{2}(\mathbf{Z})\hfill \\ & =& \frac{1}{{N}_{2}{h}_{2}^{d}}{\displaystyle \sum _{m=0}^{\u230as/2\u230b}}{c}_{3,2,2,m}(\mathbf{Z}){h}_{2}^{2m}+O\left(\frac{1}{{N}_{2}}\right).\hfill \end{array}$$
Similarly, we find that
$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\mathbf{Z}}\left[{\tilde{\mathbf{e}}}_{2,{h}_{2}}^{3}(\mathbf{Z})\right]& =& \frac{1}{{N}_{2}^{2}{h}_{2}^{3d}}{\mathbb{E}}_{\mathbf{Z}}{\mathbf{V}}_{i}^{3}(\mathbf{Z})\hfill \\ & =& \frac{1}{{\left({N}_{2}{h}_{2}^{d}\right)}^{2}}{\displaystyle \sum _{m=0}^{\u230as/2\u230b}}{c}_{3,2,3,m}(\mathbf{Z}){h}_{2}^{2m}+o\left(\frac{1}{{N}_{2}}\right).\hfill \end{array}$$
For $q=4$, we have
$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\mathbf{Z}}\left[{\tilde{\mathbf{e}}}_{2,{h}_{2}}^{4}(\mathbf{Z})\right]& =& \frac{1}{{N}_{2}^{3}{h}_{2}^{4d}}{\mathbb{E}}_{\mathbf{Z}}{\mathbf{V}}_{i}^{4}(\mathbf{Z})+\frac{{N}_{2}1}{{N}_{2}^{3}{h}_{2}^{4d}}{\left({\mathbb{E}}_{\mathbf{Z}}{\mathbf{V}}_{i}^{2}(\mathbf{Z})\right)}^{2}\hfill \\ & =& \frac{1}{{\left({N}_{2}{h}_{2}^{d}\right)}^{3}}{\displaystyle \sum _{m=0}^{\u230as/2\u230b}}{c}_{3,2,4,m}(\mathbf{Z}){h}_{2}^{2m}+\frac{1}{{\left({N}_{2}{h}_{2}^{d}\right)}^{2}}{\displaystyle \sum _{m=0}^{\u230as/2\u230b}}{c}_{6,2,2,m}(\mathbf{Z}){h}_{2}^{2m}+o\left(\frac{1}{{N}_{2}}\right).\hfill \end{array}$$
The pattern for $q\ge 2$ is then,
$${\mathbb{E}}_{\mathbf{Z}}\left[{\tilde{\mathbf{e}}}_{2,{h}_{2}}^{q}(\mathbf{Z})\right]={\displaystyle \sum _{i\in n(q)}}\frac{1}{{\left({N}_{2}{h}_{2}^{d}\right)}^{qi}}{\displaystyle \sum _{m=0}^{\u230as/2\u230b}}{c}_{6,2,q,i,m}(\mathbf{Z}){h}_{2}^{2m}+O\left(\frac{1}{{N}_{2}}\right).$$
For any integer (q), the largest possible factor is $q/2$. Thus, for a given q, the smallest possible exponent on the ${N}_{2}{h}_{2}^{d}$ term is $q/2$. This increases as q increases. A similar expression holds for ${\mathbb{E}}_{\mathbf{Z}}\left[{\tilde{\mathbf{e}}}_{1,{h}_{1}}^{q}(\mathbf{Z})\right]$, except the ${\mathbf{X}}_{i}$s are replaced with ${\mathbf{Y}}_{i}$, ${f}_{2}$ is replaced with ${f}_{1}$, and ${N}_{2}$ and ${h}_{2}$ are replaced with ${N}_{1}$ and ${h}_{1}$, respectively, all resulting in different constants. Then, since ${\tilde{\mathbf{e}}}_{1,{h}_{1}}(\mathbf{Z})$ and ${\tilde{\mathbf{e}}}_{2,{h}_{2}}(\mathbf{Z})$ are conditionally independent given $\mathbf{Z}$,
☐
$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\mathbf{Z}}\left[{\tilde{\mathbf{e}}}_{1,{h}_{1}}^{q}(\mathbf{Z}){\tilde{\mathbf{e}}}_{2,{h}_{2}}^{l}(\mathbf{Z})\right]& =& \left({\displaystyle \sum _{i\in n(q)}}\frac{1}{{\left({N}_{1}{h}_{1}^{d}\right)}^{qi}}{\displaystyle \sum _{m=0}^{\u230as/2\u230b}}{c}_{6,1,q,i,m}(\mathbf{Z}){h}_{1}^{2m}\right)\left({\displaystyle \sum _{j\in n(l)}}\frac{1}{{\left({N}_{2}{h}_{2}^{d}\right)}^{lj}}{\displaystyle \sum _{t=0}^{\u230as/2\u230b}}{c}_{6,2,l,j,t}(\mathbf{Z}){h}_{2}^{2t}\right)\hfill \\ & & +O\left(\frac{1}{{N}_{1}}+\frac{1}{{N}_{2}}\right).\hfill \end{array}$$
Applying Lemma A3 to (A18) when taking the conditional expectation given $\mathbf{Z}$ in the interior gives an expression of the form
$$\begin{array}{c}\hfill {\displaystyle \sum _{j=1}^{\lambda /2}}{\displaystyle \sum _{m=0}^{\u230as/2\u230b}}\left({c}_{7,1,j,m}\left({\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z}),{\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\right)\frac{{h}_{1}^{2m}}{{\left({N}_{1}{h}_{1}^{d}\right)}^{j}}+{c}_{7,2,j,m}\left({\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z}),{\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\right)\frac{{h}_{2}^{2m}}{{\left({N}_{2}{h}_{2}^{d}\right)}^{j}}\right)\\ \hfill +{\displaystyle \sum _{j=1}^{\lambda /2}}{\displaystyle \sum _{m=0}^{\u230as/2\u230b}}{\displaystyle \sum _{i=1}^{\lambda /2}}{\displaystyle \sum _{n=0}^{\u230as/2\u230b}}{c}_{7,j,i,m,n}\left({\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z}),{\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\right)\frac{{h}_{1}^{2m}{h}_{2}^{2n}}{{\left({N}_{1}{h}_{1}^{d}\right)}^{j}{\left({N}_{2}{h}_{2}^{d}\right)}^{i}}\\ \hfill +O\left(\frac{1}{{\left({N}_{1}{h}_{1}^{d}\right)}^{\frac{\lambda}{2}}}+\frac{1}{{\left({N}_{2}{h}_{2}^{d}\right)}^{\frac{\lambda}{2}}}\right).\end{array}$$
Note that the functionals ${c}_{7,i,j,m}$ and ${c}_{7,j,i,m,n}$ depend on the derivatives of g and ${\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{i,{h}_{i}}(\mathbf{Z})$ which depend on ${h}_{i}$. To apply an ensemble estimation, we need to separate the dependence on ${h}_{i}$ from the constants. If we use ODin1, then it is sufficient to note that in the interior of the support, ${\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{i,{h}_{i}}(\mathbf{Z})={f}_{i}(\mathbf{Z})+o(1)$ and therefore, $c\left({\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z}),{\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\right)=c\left({f}_{1}(\mathbf{Z}),{f}_{2}(\mathbf{Z})\right)+o(1)$ for some functional c. The terms in (A22) reduce to
$${c}_{7,1,1,0}\left({f}_{1}(\mathbf{Z}),{f}_{2}(\mathbf{Z})\right)\frac{1}{{N}_{1}{h}_{1}^{d}}+{c}_{7,2,1,0}\left({f}_{1}(\mathbf{Z}),{f}_{2}(\mathbf{Z})\right)\frac{1}{{N}_{2}{h}_{2}^{d}}+o\left(\frac{1}{{N}_{1}{h}_{1}^{d}}+\frac{1}{{N}_{2}{h}_{2}^{d}}\right).$$
For ODin2, we need the higher order terms. To separate the dependence on ${h}_{i}$ from the constants, we need more information about the functional g and its derivatives. Consider a special case where the functional $g(x,y)$ has derivatives of the form of ${x}^{\alpha}{y}^{\beta}$ with $\alpha ,\beta <0.$ This includes the important cases of the KL divergence and the Renyi divergence. The generalized binomial theorem states that if $\left(\genfrac{}{}{0pt}{}{\alpha}{m}\right):=\frac{\alpha (\alpha 1)\dots (\alpha m+1)}{m!}$ and if q and t are real numbers with $\leftq\right>\leftt\right$, then for any complex number ($\alpha $),
$${(q+t)}^{\alpha}={\displaystyle \sum _{m=0}^{\infty}}\left(\genfrac{}{}{0pt}{}{\alpha}{m}\right){q}^{\alpha m}{t}^{m}.$$
Since the densities are bounded away from zero, for sufficiently small ${h}_{i}$, we have ${f}_{i}(\mathbf{Z})>\left{\sum}_{j=\nu /2}^{\u230as/2\u230b}{c}_{i,j}(\mathbf{Z}){h}_{i}^{2j}+O\left({h}_{i}^{s}\right)\right.$ Applying the generalized binomial theorem and Lemma A1 gives
$${\left({\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z})\right)}^{\alpha}={\displaystyle \sum _{m=0}^{\infty}}\left(\genfrac{}{}{0pt}{}{\alpha}{m}\right){f}_{i}^{\alpha m}(\mathbf{Z}){\left({\displaystyle \sum _{j=\nu /2}^{\u230as/2\u230b}}{c}_{i,j}(\mathbf{Z}){h}_{i}^{2j}+O\left({h}_{i}^{s}\right)\right)}^{m}.$$
Since m is an integer, the exponents of the ${h}_{i}$ terms are also integers. Thus, (A22) gives, in this case,
$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\mathbf{Z}}\left[g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\right)g\left({\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z}),{\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\right)\right]& =& {\displaystyle \sum _{j=1}^{\lambda /2}}{\displaystyle \sum _{m=0}^{\u230as/2\u230b}}\left({c}_{8,1,j,m}\left(\mathbf{Z}\right)\frac{{h}_{1}^{2m}}{{\left({N}_{1}{h}_{1}^{d}\right)}^{j}}+{c}_{8,2,j,m}\left(\mathbf{Z}\right)\frac{{h}_{2}^{2m}}{{\left({N}_{2}{h}_{2}^{d}\right)}^{j}}\right)\hfill \\ & & +{\displaystyle \sum _{j=1}^{\lambda /2}}{\displaystyle \sum _{m=0}^{\u230as/2\u230b}}{\displaystyle \sum _{i=1}^{\lambda /2}}{\displaystyle \sum _{n=0}^{\u230as/2\u230b}}{c}_{8,j,i,m,n}\left(\mathbf{Z}\right)\frac{{h}_{1}^{2m}{h}_{2}^{2n}}{{\left({N}_{1}{h}_{1}^{d}\right)}^{j}{\left({N}_{2}{h}_{2}^{d}\right)}^{i}}\hfill \\ & & +O\left(\frac{1}{{\left({N}_{1}{h}_{1}^{d}\right)}^{\frac{\lambda}{2}}}+\frac{1}{{\left({N}_{2}{h}_{2}^{d}\right)}^{\frac{\lambda}{2}}}+{h}_{1}^{s}+{h}_{2}^{s}\right).\hfill \end{array}$$
As before, the case for $\mathbf{Z}$ close to the boundary of the support is more complicated. However, by using a similar technique to the proof of Lemma A2 for $\mathbf{Z}$ at the boundary and combining with previous results, we find that for general g,
$$\mathbb{E}\left[g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\right)g\left({\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z}),{\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\right)\right]={c}_{9,1}\frac{1}{{N}_{1}{h}_{1}^{d}}+{c}_{9,2}\frac{1}{{N}_{2}{h}_{2}^{d}}+o\left(\frac{1}{{N}_{1}{h}_{1}^{d}}+\frac{1}{{N}_{2}{h}_{2}^{d}}\right).$$
If $g(x,y)$ has derivatives of the form of ${x}^{\alpha}{y}^{\beta}$ with $\alpha ,\beta <0$, then we can similarly obtain
$$\begin{array}{ccc}\hfill \mathbb{E}\left[g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\right)g\left({\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{Z}),{\mathbb{E}}_{\mathbf{Z}}{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{Z})\right)\right]& =& {\displaystyle \sum _{j=1}^{\lambda /2}}{\displaystyle \sum _{m=0}^{r}}\left({c}_{9,1,j,m}\frac{{h}_{1}^{m}}{{\left({N}_{1}{h}_{1}^{d}\right)}^{j}}+{c}_{9,2,j,m}\frac{{h}_{2}^{m}}{{\left({N}_{2}{h}_{2}^{d}\right)}^{j}}\right)\hfill \\ & & +{\displaystyle \sum _{j=1}^{\lambda /2}}{\displaystyle \sum _{m=0}^{r}}{\displaystyle \sum _{i=1}^{\lambda /2}}{\displaystyle \sum _{n=0}^{r}}{c}_{9,j,i,m,n}\frac{{h}_{1}^{m}{h}_{2}^{n}}{{\left({N}_{1}{h}_{1}^{d}\right)}^{j}{\left({N}_{2}{h}_{2}^{d}\right)}^{i}}\hfill \\ & & +O\left(\frac{1}{{\left({N}_{1}{h}_{1}^{d}\right)}^{\frac{\lambda}{2}}}+\frac{1}{{\left({N}_{2}{h}_{2}^{d}\right)}^{\frac{\lambda}{2}}}+{h}_{1}^{s}+{h}_{2}^{s}\right).\hfill \end{array}$$
Appendix F. Proof of Theorem A4 (Variance)
To bound the variance of the plugin estimator ${\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}$, we use the Efron–Stein inequality [70]:
Lemma A4 (Efron–Stein Inequality).
Let ${\mathbf{X}}_{1},\dots ,{\mathbf{X}}_{n},{\mathbf{X}}_{1}^{{}^{\prime}},\dots ,{\mathbf{X}}_{n}^{{}^{\prime}}$ be independent random variables on the space $\mathcal{S}$. Then, if $f:\mathcal{S}\times \dots \times \mathcal{S}\to \mathbb{R}$, we have
$$\mathbb{V}\left[f({\mathbf{X}}_{1},\dots ,{\mathbf{X}}_{n})\right]\le \frac{1}{2}{\displaystyle \sum _{i=1}^{n}}\mathbb{E}\left[{\left(f({\mathbf{X}}_{1},\dots ,{\mathbf{X}}_{n})f({\mathbf{X}}_{1},\dots ,{\mathbf{X}}_{i}^{{}^{\prime}},\dots ,{\mathbf{X}}_{n})\right)}^{2}\right].$$
Suppose we have samples $\left\{{\mathbf{X}}_{1},\dots ,{\mathbf{X}}_{{N}_{2}},{\mathbf{Y}}_{1},\dots ,{\mathbf{Y}}_{{N}_{1}}\right\}$ and $\left\{{\mathbf{X}}_{1}^{{}^{\prime}},\dots ,{\mathbf{X}}_{{N}_{2}},{\mathbf{Y}}_{1},\dots ,{\mathbf{Y}}_{{N}_{1}}\right\}$ and denote the respective estimators as ${\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}$ and ${\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}^{{}^{\prime}}$. We have
$$\begin{array}{ccc}\hfill \left{\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}{\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}^{{}^{\prime}}\right& \le & \frac{1}{{N}_{2}}\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}^{{}^{\prime}}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1}^{{}^{\prime}})\right)\right\hfill \\ & & +\frac{1}{{N}_{2}}{\displaystyle \sum _{j=2}^{{N}_{2}}}\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}^{{}^{\prime}}({\mathbf{X}}_{j})\right)\right.\hfill \end{array}$$
Since g is Lipschitz continuous with the constant ${C}_{g}$, we have
where the last step follows from Jensen’s inequality. By making the substitutions ${\mathbf{u}}_{i}=\frac{{\mathbf{X}}_{1}{\mathbf{Y}}_{i}}{{h}_{1}}$ and ${\mathbf{u}}_{i}^{{}^{\prime}}=\frac{{\mathbf{X}}_{1}^{{}^{\prime}}{\mathbf{Y}}_{i}}{{h}_{1}}$, this gives
$$\begin{array}{ccc}\hfill \leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}^{{}^{\prime}}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1}^{{}^{\prime}})\right)\right& \le & {C}_{g}\left(\left{\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}){\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}^{{}^{\prime}})\right+\left{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1}){\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1}^{{}^{\prime}})\right\right),\hfill \end{array}$$
$$\begin{array}{ccc}\hfill \left{\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}){\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}^{{}^{\prime}})\right& =& \frac{1}{{N}_{1}{h}_{1}^{d}}\left{\displaystyle \sum _{i=1}^{{N}_{1}}}\left(K\left(\frac{{\mathbf{X}}_{1}{\mathbf{Y}}_{i}}{{h}_{1}}\right)K\left(\frac{{\mathbf{X}}_{1}^{{}^{\prime}}{\mathbf{Y}}_{i}}{{h}_{1}}\right)\right)\right\hfill \\ & \le & \frac{1}{{N}_{1}{h}_{1}^{d}}{\displaystyle \sum _{i=1}^{{N}_{1}}}\leftK\left(\frac{{\mathbf{X}}_{1}{\mathbf{Y}}_{i}}{{h}_{1}}\right)K\left(\frac{{\mathbf{X}}_{1}^{{}^{\prime}}{\mathbf{Y}}_{i}}{{h}_{1}}\right)\right\hfill \\ \hfill \Rightarrow \mathbb{E}\left[{\left{\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}){\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}^{{}^{\prime}})\right}^{2}\right]& \le & \frac{1}{{N}_{1}{h}_{1}^{2d}}{\displaystyle \sum _{i=1}^{{N}_{1}}}\mathbb{E}\left[{\left(K\left(\frac{{\mathbf{X}}_{1}{\mathbf{Y}}_{i}}{{h}_{1}}\right)K\left(\frac{{\mathbf{X}}_{1}^{{}^{\prime}}{\mathbf{Y}}_{i}}{{h}_{1}}\right)\right)}^{2}\right],\hfill \end{array}$$
$$\begin{array}{ccc}\hfill \frac{1}{{h}_{1}^{2d}}\mathbb{E}\left[{\left(K\left(\frac{{\mathbf{X}}_{1}{\mathbf{Y}}_{i}}{{h}_{1}}\right)K\left(\frac{{\mathbf{X}}_{1}^{{}^{\prime}}{\mathbf{Y}}_{i}}{{h}_{1}}\right)\right)}^{2}\right]& =& \frac{1}{{h}^{2d}}\int {\left(K\left(\frac{{\mathbf{X}}_{1}{\mathbf{Y}}_{i}}{{h}_{1}}\right)K\left(\frac{{\mathbf{X}}_{1}^{{}^{\prime}}{\mathbf{Y}}_{i}}{{h}_{1}}\right)\right)}^{2}\times \hfill \\ & & {f}_{2}({\mathbf{X}}_{1}){f}_{2}({\mathbf{X}}_{1}^{{}^{\prime}}){f}_{1}({\mathbf{Y}}_{i})d{\mathbf{X}}_{1}d{\mathbf{X}}_{1}^{{}^{\prime}}d{\mathbf{Y}}_{i}\hfill \\ & \le & {2\left\rightK\left\right}_{\infty}^{2}.\hfill \end{array}$$
Combining this with (A29) gives
$$\mathbb{E}\left[{\left{\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}){\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}^{{}^{\prime}})\right}^{2}\right]\le {2\left\rightK\left\right}_{\infty}^{2}.$$
Similarly,
$$\mathbb{E}\left[{\left{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1}){\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1}^{{}^{\prime}})\right}^{2}\right]\le {2\left\rightK\left\right}_{\infty}^{2}.$$
Combining these results with (A27) gives
$$\mathbb{E}\left[{\left(g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}^{{}^{\prime}}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1}^{{}^{\prime}})\right)\right)}^{2}\right]\le 8{C}_{g}^{2}{\left\rightK\left\right}_{\infty}^{2}.$$
The second term in (A26) is controlled in a similar way. From the Lipschitz condition,
$$\begin{array}{ccc}\hfill {\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}^{{}^{\prime}}({\mathbf{X}}_{j})\right)\right}^{2}& \le & {C}_{g}^{2}{\left{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j}){\tilde{\mathbf{f}}}_{2,{h}_{2}}^{{}^{\prime}}({\mathbf{X}}_{j})\right}^{2}\hfill \\ & =& \frac{{C}_{g}^{2}}{{M}_{2}^{2}{h}_{2}^{2d}}{\left(K\left(\frac{{\mathbf{X}}_{j}{\mathbf{X}}_{1}}{h}\right)K\left(\frac{{\mathbf{X}}_{j}{\mathbf{X}}_{1}^{{}^{\prime}}}{h}\right)\right)}^{2}.\hfill \end{array}$$
The ${h}_{2}^{2d}$ terms are eliminated by making the substitutions of ${\mathbf{u}}_{j}=\frac{{\mathbf{X}}_{j}{\mathbf{X}}_{1}}{{h}_{2}}$ and ${\mathbf{u}}_{j}^{{}^{\prime}}=\frac{{\mathbf{X}}_{j}{\mathbf{X}}_{1}^{{}^{\prime}}}{{h}_{2}}$ within the expectation to obtain
where we use the Cauchy Schwarz inequality to bound the expectation within each summand. Finally, applying Jensen’s inequality and (A29) and (A32) gives
$$\begin{array}{ccc}\hfill \mathbb{E}\left[{\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}^{{}^{\prime}}({\mathbf{X}}_{j})\right)\right}^{2}\right]& \le & \frac{2{C}_{g}^{2}{\left\rightK\left\right}_{\infty}^{2}}{{M}_{2}^{2}}\hfill \end{array}$$
$$\Rightarrow \mathbb{E}\left[{\left({\displaystyle \sum _{j=2}^{{N}_{2}}}\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}^{{}^{\prime}}({\mathbf{X}}_{j})\right)\right\right)}^{2}\right]$$
$$\begin{array}{cc}=& {\displaystyle \sum _{j=2}^{{N}_{2}}}{\displaystyle \sum _{i=2}^{{N}_{2}}}\mathbb{E}\left[\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}^{{}^{\prime}}({\mathbf{X}}_{j})\right)\right\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{i}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{i})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{i}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}^{{}^{\prime}}({\mathbf{X}}_{i})\right)\right\right]\hfill \\ \le & {M}_{2}^{2}\mathbb{E}\left[{\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}^{{}^{\prime}}({\mathbf{X}}_{j})\right)\right}^{2}\right]\hfill \\ \le & 2{C}_{g}^{2}{\left\rightK\left\right}_{\infty}^{2},\hfill \end{array}$$
$$\begin{array}{ccc}\hfill \mathbb{E}\left[{\left{\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}{\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}^{{}^{\prime}}\right}^{2}\right]& \le & \frac{2}{{N}_{2}^{2}}\mathbb{E}\left[{\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}^{{}^{\prime}}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1}^{{}^{\prime}})\right)\right}^{2}\right]\hfill \\ & & +\frac{2}{{N}_{2}^{2}}\mathbb{E}\left[{\left({\displaystyle \sum _{j=2}^{{N}_{2}}}\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}^{{}^{\prime}}({\mathbf{X}}_{j})\right)\right\right)}^{2}\right]\hfill \\ & \le & \frac{20{C}_{g}^{2}{\left\rightK\left\right}_{\infty}^{2}}{{N}_{2}^{2}}.\hfill \end{array}$$
Now, suppose we have samples $\left\{{\mathbf{X}}_{1},\dots ,{\mathbf{X}}_{{N}_{2}},{\mathbf{Y}}_{1},\dots ,{\mathbf{Y}}_{{N}_{1}}\right\}$ and $\left\{{\mathbf{X}}_{1},\dots ,{\mathbf{X}}_{{N}_{2}},{\mathbf{Y}}_{1}^{{}^{\prime}},\dots ,{\mathbf{Y}}_{{N}_{1}}\right\}$ and denote the respective estimators as ${\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}$ and ${\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}^{{}^{\prime}}$. Then,
$$\begin{array}{ccc}\hfill \leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}^{{}^{\prime}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)\right& \le & {C}_{g}\left{\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}){\tilde{\mathbf{f}}}_{1,{h}_{1}}^{{}^{\prime}}({\mathbf{X}}_{j})\right\hfill \\ & =& \frac{{C}_{g}}{{N}_{1}{h}_{1}^{d}}\leftK\left(\frac{{\mathbf{X}}_{j}{\mathbf{Y}}_{1}}{{h}_{1}}\right)K\left(\frac{{\mathbf{X}}_{j}{\mathbf{Y}}_{1}^{{}^{\prime}}}{{h}_{1}}\right)\right\hfill \\ \hfill \Rightarrow \mathbb{E}\left[{\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}^{{}^{\prime}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)\right}^{2}\right]& \le & \frac{2{C}_{g}^{2}{\left\rightK\left\right}_{\infty}^{2}}{{N}_{1}^{2}}.\hfill \end{array}$$
Thus, using a similar argument as was used to obtain (A32),
$$\begin{array}{ccc}\hfill \mathbb{E}\left[{\left{\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}{\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}^{{}^{\prime}}\right}^{2}\right]& \le & \frac{1}{{N}_{2}^{2}}\mathbb{E}\left[{\left({\displaystyle \sum _{j=1}^{{N}_{2}}}\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}^{{}^{\prime}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)\right\right)}^{2}\right]\hfill \\ & \le & \frac{2{C}_{g}^{2}{\left\rightK\left\right}_{\infty}^{2}}{{N}_{2}^{2}}.\hfill \end{array}$$
Applying the Efron–Stein inequality gives
$$\mathbb{V}\left[{\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}\right]\le \frac{10{C}_{g}^{2}{\left\rightK\left\right}_{\infty}^{2}}{{N}_{2}}+\frac{{C}_{g}^{2}{\left\rightK\left\right}_{\infty}^{2}{N}_{1}}{{N}_{2}^{2}}.$$
Appendix G. Proof of Theorem 4 (CLT)
We are interested in the asymptotic distribution of
$$\begin{array}{ccc}\hfill \sqrt{{N}_{2}}\left({\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}\mathbb{E}\left[{\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}\right]\right)& =& \frac{1}{\sqrt{{N}_{2}}}{\displaystyle \sum _{j=1}^{{N}_{2}}}\left(g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right){\mathbb{E}}_{{\mathbf{X}}_{j}}\left[g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)\right]\right)\hfill \\ & & +\frac{1}{\sqrt{{N}_{2}}}{\displaystyle \sum _{j=1}^{{N}_{2}}}\left({\mathbb{E}}_{{\mathbf{X}}_{j}}\left[g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)\right]\mathbb{E}\left[g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)\right]\right).\hfill \end{array}$$
Note that in the standard central limit theorem [79], the second term converges in distribution to a Gaussian random variable. If the first term converges in probability to a constant (specifically, 0), then we can use Slutsky’s theorem [80] to find the asymptotic distribution. So, now, we focus on the first term which we denote as ${\mathbf{V}}_{{N}_{2}}$.
To prove convergence in probability, we use Chebyshev’s inequality. Note that $\mathbb{E}\left[{\mathbf{V}}_{{N}_{2}}\right]=0$. To bound the variance of ${\mathbf{V}}_{{N}_{2}}$, we again use the Efron–Stein inequality. Let ${\mathbf{X}}_{1}^{{}^{\prime}}$ be drawn from ${f}_{2}$ and denote ${\mathbf{V}}_{{N}_{2}}$ and ${\mathbf{V}}_{{N}_{2}}^{{}^{\prime}}$ as the sequences using ${\mathbf{X}}_{1}$ and ${\mathbf{X}}_{1}^{{}^{\prime}}$, respectively. Then,
$$\begin{array}{ccc}\hfill {\mathbf{V}}_{{N}_{2}}{\mathbf{V}}_{{N}_{2}}^{{}^{\prime}}& =& \frac{1}{\sqrt{{N}_{2}}}\left(g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1})\right){\mathbb{E}}_{{\mathbf{X}}_{1}}\left[g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1})\right)\right]\right)\hfill \\ & & \frac{1}{\sqrt{{N}_{2}}}\left(g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}^{{}^{\prime}}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1}^{{}^{\prime}})\right){\mathbb{E}}_{{\mathbf{X}}_{1}^{{}^{\prime}}}\left[g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}^{{}^{\prime}}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1}^{{}^{\prime}})\right)\right]\right)\hfill \\ & & +\frac{1}{\sqrt{{N}_{2}}}{\displaystyle \sum _{j=2}^{{N}_{2}}}\left(g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}^{{}^{\prime}}({\mathbf{X}}_{j})\right)\right).\hfill \end{array}$$
Note that
$$\mathbb{E}\left[{\left(g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1})\right){\mathbb{E}}_{{\mathbf{X}}_{1}}\left[g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1})\right)\right]\right)}^{2}\right]=\mathbb{E}\left[{\mathbb{V}}_{{\mathbf{X}}_{1}}\left[g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1})\right)\right]\right].$$
We use the Efron–Stein inequality to bound ${\mathbb{V}}_{{\mathbf{X}}_{1}}\left[g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1})\right)\right]$.
We do this by bounding the conditional expectation of the term
where ${\mathbf{X}}_{i}$ is replaced with ${\mathbf{X}}_{i}^{{}^{\prime}}$ in the KDEs for some $i\ne 1$. Using similar steps as in Appendix F, we have
$$\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}^{{}^{\prime}}({\mathbf{X}}_{1})\right)\right,$$
$${\mathbb{E}}_{{\mathbf{X}}_{1}}\left[{\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}^{{}^{\prime}}({\mathbf{X}}_{1})\right)\right}^{2}\right]=O\left(\frac{1}{{N}_{2}^{2}}\right).$$
A similar result is obtained when ${\mathbf{Y}}_{i}$ is replaced with ${\mathbf{Y}}_{\mathbf{i}}^{{}^{\prime}}$. Then, based on the Efron–Stein inequality, ${\mathbb{V}}_{{\mathbf{X}}_{1}}\left[g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1})\right)\right]=O\left(\frac{1}{{N}_{2}}+\frac{1}{{N}_{1}}\right)$.
Therefore,
$$\mathbb{E}\left[\frac{1}{{N}_{2}}{\left(g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1})\right){\mathbb{E}}_{{\mathbf{X}}_{1}}\left[g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1})\right)\right]\right)}^{2}\right]=O\left(\frac{1}{{N}_{2}^{2}}+\frac{1}{{N}_{1}{N}_{2}}\right).$$
A similar result holds for the $g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}^{{}^{\prime}}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1}^{{}^{\prime}})\right)$ terms in (A33).
For the third term in (A33),
$$\mathbb{E}\left[{\left({\displaystyle \sum _{j=2}^{{N}_{2}}}\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}^{{}^{\prime}}({\mathbf{X}}_{j})\right)\right\right)}^{2}\right]$$
$$={\displaystyle \sum _{j=2}^{{N}_{2}}}{\displaystyle \sum _{i=2}^{{N}_{2}}}\mathbb{E}\left[\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}^{{}^{\prime}}({\mathbf{X}}_{j})\right)\right\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{i}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{i})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{i}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}^{{}^{\prime}}({\mathbf{X}}_{i})\right)\right\right].$$
There are ${M}_{2}$ terms where $i=j$, and we have from Appendix F (see (A30)) that
$$\mathbb{E}\left[{\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}^{{}^{\prime}}({\mathbf{X}}_{j})\right)\right}^{2}\right]\le \frac{2{C}_{g}^{2}{\left\rightK\left\right}_{\infty}^{2}}{{M}_{2}^{2}}.$$
Thus, these terms are $O\left(\frac{1}{{M}_{2}}\right)$. There are ${M}_{2}^{2}{M}_{2}$ terms when $i\ne j$. In this case, we can do four substitutions of the form ${\mathbf{u}}_{j}=\frac{{\mathbf{X}}_{j}{\mathbf{X}}_{1}}{{h}_{2}}$ to obtain
$$\mathbb{E}\left[\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}^{{}^{\prime}}({\mathbf{X}}_{j})\right)\right\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{i}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{i})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{i}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}^{{}^{\prime}}({\mathbf{X}}_{i})\right)\right\right]\le \frac{4{C}_{g}^{2}{\left\rightK\left\right}_{\infty}^{2}{h}_{2}^{2d}}{{M}_{2}^{2}}.$$
Then, since ${h}_{2}^{d}=o(1)$, we get
$$\mathbb{E}\left[{\left({\displaystyle \sum _{j=2}^{{N}_{2}}}\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}^{{}^{\prime}}({\mathbf{X}}_{j})\right)\right\right)}^{2}\right]=o(1),$$
$$\begin{array}{ccc}\hfill \Rightarrow \mathbb{E}\left[{\left({\mathbf{V}}_{{N}_{2}}{\mathbf{V}}_{{N}_{2}}^{{}^{\prime}}\right)}^{2}\right]& \le & \frac{3}{{N}_{2}}\mathbb{E}\left[{\left(g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1})\right){\mathbb{E}}_{{\mathbf{X}}_{1}}\left[g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1})\right)\right]\right)}^{2}\right]\hfill \\ & & +\frac{3}{{N}_{2}}\mathbb{E}\left[{\left(g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}^{{}^{\prime}}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1}^{{}^{\prime}})\right){\mathbb{E}}_{{\mathbf{X}}_{1}^{{}^{\prime}}}\left[g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{1}^{{}^{\prime}}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{1}^{{}^{\prime}})\right)\right]\right)}^{2}\right]\hfill \\ & & +\frac{3}{{N}_{2}}\mathbb{E}\left[{\left({\displaystyle \sum _{j=2}^{{N}_{2}}}\left(g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}^{{}^{\prime}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}^{{}^{\prime}}({\mathbf{X}}_{j})\right)\right)\right)}^{2}\right]\hfill \\ & =& o\left(\frac{1}{{N}_{2}}\right).\hfill \end{array}$$
Now, consider samples $\left\{{\mathbf{X}}_{1},\dots ,{\mathbf{X}}_{{N}_{2}},{\mathbf{Y}}_{1},\dots ,{\mathbf{Y}}_{{N}_{1}}\right\}$ and $\left\{{\mathbf{X}}_{1},\dots ,{\mathbf{X}}_{{N}_{2}},{\mathbf{Y}}_{1}^{{}^{\prime}},\dots ,{\mathbf{Y}}_{{N}_{1}}\right\}$ and the respective sequences ${\mathbf{V}}_{{N}_{2}}$ and ${\mathbf{V}}_{{N}_{2}}^{{}^{\prime}}$. Then,
$$\begin{array}{ccc}\hfill {\mathbf{V}}_{{N}_{2}}{\mathbf{V}}_{{N}_{2}}^{{}^{\prime}}& =& \frac{1}{\sqrt{{N}_{2}}}{\displaystyle \sum _{j=1}^{{N}_{2}}}\left(g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}^{{}^{\prime}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)\right).\hfill \end{array}$$
Using a similar argument as that used to obtain (A33), we have that if ${h}_{1}^{d}=o(1)$ and ${N}_{1}\to \infty $, then
$$\mathbb{E}\left[{\left({\displaystyle \sum _{j=2}^{{N}_{2}}}\leftg\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}^{{}^{\prime}}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}({\mathbf{X}}_{j})\right)\right\right)}^{2}\right]=o(1)$$
$$\Rightarrow \mathbb{E}\left[{\left({\mathbf{V}}_{{N}_{2}}{\mathbf{V}}_{{N}_{2}}^{{}^{\prime}}\right)}^{2}\right]=o\left(\frac{1}{{N}_{2}}\right).$$
Applying the Efron–Stein inequality gives
$$\mathbb{V}\left[{\mathbf{V}}_{{N}_{2}}\right]=o\left(\frac{{N}_{2}+{N}_{1}}{{N}_{2}}\right)=o(1).$$
Thus, based on Chebyshev’s inequality,
and therefore, ${\mathbf{V}}_{{N}_{2}}$ converges to zero in probability. Based on Slutsky’s theorem, $\sqrt{{N}_{2}}\left({\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}\mathbb{E}\left[{\tilde{\mathbf{G}}}_{{h}_{1},{h}_{2}}\right]\right)$ converges in distribution to a zero mean Gaussian random variable with variance
where $\mathbf{X}$ is drawn from ${f}_{2}$.
$$Pr\left(\left{\mathbf{V}}_{{N}_{2}}\right>\u03f5\right)\le \frac{\mathbb{V}\left[{\mathbf{V}}_{{N}_{2}}\right]}{{\u03f5}^{2}}=o(1),$$
$$\mathbb{V}\left[{\mathbb{E}}_{\mathbf{X}}\left[g\left({\tilde{\mathbf{f}}}_{1,{h}_{1}}(\mathbf{X}),{\tilde{\mathbf{f}}}_{2,{h}_{2}}(\mathbf{X})\right)\right]\right],$$
For the weighted ensemble estimator, we wish to know the asymptotic distribution of $\sqrt{{N}_{2}}\left({\tilde{\mathbf{G}}}_{w}\mathbb{E}\left[{\tilde{\mathbf{G}}}_{w}\right]\right)$ where ${\tilde{\mathbf{G}}}_{w}={\sum}_{l\in \overline{l}}w(l){\tilde{\mathbf{G}}}_{h(l)}$. We have
$$\begin{array}{ccc}\hfill \sqrt{{N}_{2}}\left({\tilde{\mathbf{G}}}_{w}\mathbb{E}\left[{\tilde{\mathbf{G}}}_{w}\right]\right)& =& \frac{1}{\sqrt{{N}_{2}}}{\displaystyle \sum _{j=1}^{{N}_{2}}}{\displaystyle \sum _{l\in \overline{l}}}w(l)\left(g\left({\tilde{\mathbf{f}}}_{1,h(l)}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,h(l)}({\mathbf{X}}_{j})\right){\mathbb{E}}_{{\mathbf{X}}_{j}}\left[g\left({\tilde{\mathbf{f}}}_{1,h(l)}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,h(l)}({\mathbf{X}}_{j})\right)\right]\right)\hfill \\ & & +\frac{1}{\sqrt{{N}_{2}}}{\displaystyle \sum _{j=1}^{{N}_{2}}}\left({\mathbb{E}}_{{\mathbf{X}}_{j}}\left[{\displaystyle \sum _{l\in \overline{l}}}w(l)g\left({\tilde{\mathbf{f}}}_{1,h(l)}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,h(l)}({\mathbf{X}}_{j})\right)\right]\mathbb{E}\left[{\displaystyle \sum _{l\in \overline{l}}}w(l)g\left({\tilde{\mathbf{f}}}_{1,h(l)}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,h(l)}({\mathbf{X}}_{j})\right)\right]\right).\hfill \end{array}$$
The second term again converges in distribution to a Gaussian random variable by the central limit theorem. The mean and variance are, respectively, zero and
$$\mathbb{V}\left[{\displaystyle \sum _{l\in \overline{l}}}w(l){\mathbb{E}}_{\mathbf{X}}\left[g\left({\tilde{\mathbf{f}}}_{1,h(l)}(\mathbf{X}),{\tilde{\mathbf{f}}}_{2,h(l)}(\mathbf{X})\right)\right]\right].$$
The first term is equal to
where ${o}_{P}(1)$ denotes the convergence to zero as a probability. In the last step, we use the fact that if two random variables converge in probability to constants, then their linear combination converges in probability to the linear combination of the constants. Combining this result with Slutsky’s theorem completes the proof.
$$\begin{array}{ccc}\hfill {\displaystyle \sum _{l\in \overline{l}}}w(l)\left(\frac{1}{\sqrt{{N}_{2}}}{\displaystyle \sum _{j=1}^{{N}_{2}}}\left(g\left({\tilde{\mathbf{f}}}_{1,h(l)}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,h(l)}({\mathbf{X}}_{j})\right){\mathbb{E}}_{{\mathbf{X}}_{j}}\left[g\left({\tilde{\mathbf{f}}}_{1,h(l)}({\mathbf{X}}_{j}),{\tilde{\mathbf{f}}}_{2,h(l)}({\mathbf{X}}_{j})\right)\right]\right)\right)& =& {\displaystyle \sum _{l\in \overline{l}}}w(l){o}_{P}(1)\hfill \\ & =& {o}_{P}(1),\hfill \end{array}$$
Appendix H. Proof of Theorem 5 (Uniform MSE)
Since the MSE is equal to the square of the bias plus the variance, we can upper bound the left hand side of (7) with
$$\begin{array}{cc}\hfill {\displaystyle \underset{p,q\in \mathsf{\Sigma}(s,{K}_{H},{\u03f5}_{0},{\u03f5}_{\infty})}{sup}}\mathbb{E}\left[{\left({\tilde{\mathbf{G}}}_{{w}_{0}}G(p,q)\right)}^{2}\right]& ={\displaystyle \underset{p,q\in \mathsf{\Sigma}(s,{K}_{H},{\u03f5}_{0},{\u03f5}_{\infty})}{sup}}\left(\mathrm{Bias}{\left({\tilde{\mathbf{G}}}_{{w}_{0}}\right)}^{2}+\mathrm{Var}\left({\tilde{\mathbf{G}}}_{{w}_{0}}\right)\right)\hfill \\ & \le {\displaystyle \underset{p,q\in \mathsf{\Sigma}(s,{K}_{H},{\u03f5}_{0},{\u03f5}_{\infty})}{sup}}\mathrm{Bias}{\left({\tilde{\mathbf{G}}}_{{w}_{0}}\right)}^{2}+{\displaystyle \underset{p,q\in \mathsf{\Sigma}(s,{K}_{H},{\u03f5}_{0},{\u03f5}_{\infty})}{sup}}\mathrm{Var}\left({\tilde{\mathbf{G}}}_{{w}_{0}}\right).\hfill \end{array}$$
From the assumptions (Lipschitz, kernel bounded, weight calculated from the relaxed optimization problem), we have
where the last step follows on from the fact that all of the terms are independent of p and q.
$$\begin{array}{ccc}\hfill {\displaystyle \underset{p,q\in \mathsf{\Sigma}(s,{K}_{H},{\u03f5}_{0},{\u03f5}_{\infty})}{sup}}\mathrm{Var}\left({\tilde{\mathbf{G}}}_{{w}_{0}}\right)& \le & {\displaystyle \underset{p,q\in \mathsf{\Sigma}(s,K,{\u03f5}_{0},{\u03f5}_{\infty})}{sup}}\frac{11{C}_{g}^{2}\left\right{w}_{0}{\left\right}_{2}^{2}{\left\rightK\left\right}_{\infty}}{N}\hfill \\ & =& \frac{11{C}_{g}^{2}\left\right{w}_{0}{\left\right}_{2}^{2}{\left\rightK\left\right}_{\infty}}{N},\hfill \end{array}$$
For the bias, recall that if g is infinitely differentiable and if the optimal weight (${w}_{0}$) is calculated using the relaxed convex optimization problem, then
We use a topology argument to bound the supremum of this term. We use the Extreme Value Theorem [81]:
$$\begin{array}{ccc}\hfill \mathrm{Bias}\left({\tilde{\mathbf{G}}}_{{w}_{0}}\right)& =& {\displaystyle \sum _{i\in J}}{c}_{i}(p,q)\u03f5{N}^{1/2},\hfill \\ \hfill \Rightarrow \mathrm{Bias}{\left({\tilde{\mathbf{G}}}_{{w}_{0}}\right)}^{2}& =& \frac{{\u03f5}^{2}}{N}{\left({\displaystyle \sum _{i\in J}}{c}_{i}(p,q)\right)}^{2}.\hfill \end{array}$$
Theorem A5 (Extreme Value Theorem)
Let $f:X\to \mathbb{R}$ be continuous. If X is compact, then points $c,d\in X$ s.t. $f(c)\le f(x)\le f(d)$ exist for every $x\in X$.
Based on this theorem, f achieves its minimum and maximum on X. Our approach is to first show that the functionals ${c}_{i}(p,q)$ are continuous wrt p and q in some appropriate norm. We then show that the space $\mathsf{\Sigma}(s,{K}_{H},{\u03f5}_{0},{\u03f5}_{\infty})$ is compact wrt this norm. The Extreme Value Theorem can then be applied to bound the supremum of (A34).
We first define the norm. Let $\alpha =sr>0$. We use the standard norm on the space $\mathsf{\Sigma}(s,{K}_{H})$ [82]:
where
$$\begin{array}{ccc}\hfill \left\rightf\left\right& =& {\left\rightf\left\right}_{\mathsf{\Sigma}(s,{K}_{H})}\hfill \\ & =& {\left\rightf\left\right}_{{C}^{r}}+{\displaystyle \underset{\left\beta \right=r}{max}}{\left{D}^{\beta}f\right}_{{C}^{0,\alpha}}\hfill \end{array}$$
$$\begin{array}{ccc}\hfill {\left\rightf\left\right}_{{C}^{r}}& =& {\displaystyle \underset{\left\beta \right\le r}{max}}{\displaystyle \underset{x\in \mathcal{S}}{sup}}\left{D}^{\beta}f(x)\right,\hfill \\ \hfill {\leftf\right}_{{C}^{0,\alpha}}& =& {\displaystyle \underset{x\ne y\in \mathcal{S}}{sup}}\frac{f(x)f(y)}{{xy}^{\alpha}}.\hfill \end{array}$$
Lemma A5.
The functionals ${c}_{i}(p,q)$ are continuous wrt the norm $max(\leftp\right{}_{{C}^{r}},\left\rightq{}_{{C}^{r}})$.
Proof.
The functionals ${c}_{i}(p,q)$ depend on terms of the form
$$c(p,q)=\int \left({\left.\frac{{\partial}^{i+j}g({t}_{1},{t}_{2})}{\partial {t}_{1}^{i}\partial {t}_{2}^{j}}\right}_{\begin{array}{c}{t}_{1}=p(x)\\ {t}_{2}=q(x)\end{array}}\right){D}^{\beta}p(x){D}^{\gamma}q(x)q(x)dx.$$
It is sufficient to show that this is continuous. Let $\u03f5>0$ and $max\left(\left\rightp{p}_{0}{\left\right}_{{C}^{r}},\left\rightq{q}_{0}{\left\right}_{{C}^{r}}\right)<\delta $ where $\delta >0$ will be chosen later. Then, by applying the triangle inequality for integration and adding and subtracting terms, we have
$$c(p,q)c({p}_{0},{q}_{0})$$
$$\begin{array}{cc}\le \hfill & \hfill \int \left\left({\left.\frac{{\partial}^{i+j}g({t}_{1},{t}_{2})}{\partial {t}_{1}^{i}\partial {t}_{2}^{j}}\right}_{\begin{array}{c}{t}_{1}=p(x)\\ {t}_{2}=q(x)\end{array}}\right){D}^{\beta}p(x){D}^{\gamma}q(x)\left(q(x){q}_{0}(x)\right)\rightdx\\ & \hfill +\int \left\left({\left.\frac{{\partial}^{i+j}g({t}_{1},{t}_{2})}{\partial {t}_{1}^{i}\partial {t}_{2}^{j}}\right}_{\begin{array}{c}{t}_{1}=p(x)\\ {t}_{2}=q(x)\end{array}}\right){D}^{\beta}p(x){q}_{0}(x)\left({D}^{\gamma}q(x){D}^{\gamma}{q}_{0}(x)\right)\rightdx\\ & \hfill +\int \left{D}^{\beta}{p}_{0}(x){D}^{\gamma}{q}_{0}(x){q}_{0}(x)\left(\left({\left.\frac{{\partial}^{i+j}g({t}_{1},{t}_{2})}{\partial {t}_{1}^{i}\partial {t}_{2}^{j}}\right}_{\begin{array}{c}{t}_{1}=p(x)\\ {t}_{2}=q(x)\end{array}}\right)\right.\right.\\ & \hfill \left.\left.\left({\left.\frac{{\partial}^{i+j}g({t}_{1},{t}_{2})}{\partial {t}_{1}^{i}\partial {t}_{2}^{j}}\right}_{\begin{array}{c}{t}_{1}={p}_{0}(x)\\ {t}_{2}={q}_{0}(x)\end{array}}\right)\right)\rightdx\\ & \hfill +\int \left\left({\left.\frac{{\partial}^{i+j}g({t}_{1},{t}_{2})}{\partial {t}_{1}^{i}\partial {t}_{2}^{j}}\right}_{\begin{array}{c}{t}_{1}=p(x)\\ {t}_{2}=q(x)\end{array}}\right){D}^{\gamma}{q}_{0}(x){q}_{0}(x)\left({D}^{\beta}p(x){D}^{\beta}{p}_{0}(x)\right)\rightdx.\end{array}$$
Based on Assumption $\mathcal{A}.4$, the absolute value of the mixed derivatives of g is bounded on the range defined for p and q by some constant (${C}_{i,j}$). Also, ${q}_{0}(x)\le {\u03f5}_{\infty}$. Furthermore, since ${D}^{\gamma}{q}_{0}$ and ${D}^{\beta}p$ are continuous, and since $\mathcal{S}\subset {\mathbb{R}}^{d}$ is compact, then the absolute value of derivatives ${D}^{\gamma}{q}_{0}$ and ${D}^{\beta}p$ is also bounded by a constant (${\u03f5}_{\infty}^{{}^{\prime}}$). Let ${\delta}_{0}>0$. Then, since the mixed derivatives of g are continuous on the interval $[{\u03f5}_{0},{\u03f5}_{\infty}]$, they are uniformly continuous. Therefore, we can choose a small enough $\delta $ such that (s.t.)
$$\left\left({\left.\frac{{\partial}^{i+j}g({t}_{1},{t}_{2})}{\partial {t}_{1}^{i}\partial {t}_{2}^{j}}\right}_{\begin{array}{c}{t}_{1}=p(x)\\ {t}_{2}=q(x)\end{array}}\right)\left({\left.\frac{{\partial}^{i+j}g({t}_{1},{t}_{2})}{\partial {t}_{1}^{i}\partial {t}_{2}^{j}}\right}_{\begin{array}{c}{t}_{1}={p}_{0}(x)\\ {t}_{2}={q}_{0}(x)\end{array}}\right)\right<{\delta}_{0}.$$
Combining all of these results with (A36) gives
where $\lambda (\mathcal{S})$ is the Lebesgue measure of $\mathcal{S}$. This is bounded since $\mathcal{S}$ is compact. Let ${\delta}_{0}^{{}^{\prime}}>0$ be s.t. if $max\left(\left\rightp{p}_{0}{\left\right}_{{C}^{r}},\left\rightq{q}_{0}{\left\right}_{{C}^{r}}\right)<{\delta}_{0}^{{}^{\prime}}$, then (A37) is less than $\frac{\u03f5}{4\lambda (\mathcal{S}){\u03f5}_{\infty}^{{}^{\prime}}{\u03f5}_{\infty}}$. Let ${\delta}_{1}=\frac{\u03f5}{4\lambda (\mathcal{S}){C}_{ij}{\u03f5}_{\infty}^{{}^{\prime}}(1+{\u03f5}_{\infty})}$. Then, if $\delta <min({\delta}_{0}^{{}^{\prime}},{\delta}_{1})$,
☐
$$\begin{array}{ccc}\hfill c(p,q)c({p}_{0},{q}_{0})& \le & \lambda (\mathcal{S})\delta {C}_{ij}{\u03f5}_{\infty}^{{}^{\prime}}\left(2+{\u03f5}_{\infty}\right)\hfill \\ & & +\lambda (\mathcal{S}){\u03f5}_{\infty}^{{}^{\prime}}{\u03f5}_{\infty}\left(2{\delta}_{0}+{C}_{ij}\delta \right),\hfill \end{array}$$
$$c(p,q)c({p}_{0},{q}_{0})<\u03f5.$$
Given that each ${c}_{i}(p,q)$ is continuous, then ${\left({\sum}_{i\in J}{c}_{i}(p,q)\right)}^{2}$ is also continuous wrt p and q.
We now argue that $\mathsf{\Sigma}(s,{K}_{H})$ is compact. First, a set is relatively compact if its closure is compact. Based on the Arzela–Ascoli theorem [83], the space $\mathsf{\Sigma}(s,{K}_{H})$ is relatively compact in the topology induced by the ${\u2225\xb7\u2225}_{\mathsf{\Sigma}(t,{K}_{H})}$ norm for any $t<s$. We choose $t=r$. It can then be shown that under the ${\u2225\xb7\u2225}_{\mathsf{\Sigma}(r,{K}_{H})}$ norm, $\mathsf{\Sigma}(s,{K}_{H})$ is complete [82]. Since $\mathsf{\Sigma}(s,{K}_{H})$ is contained in a metric space, it is also closed and therefore, equal to its closure. Thus, $\mathsf{\Sigma}(s,{K}_{H})$ is compact. Then, since $\mathsf{\Sigma}(s,{K}_{H},{\u03f5}_{0},{\u03f5}_{\infty})$ is closed in $\mathsf{\Sigma}(s,{K}_{H})$, it is also compact. Therefore, since for each $p,q\in \mathsf{\Sigma}(s,{K}_{H},{\u03f5}_{0},{\u03f5}_{\infty})$, ${\left({\sum}_{i\in J}{c}_{i}(p,q)\right)}^{2}<\infty $, based on the Extreme Value Theorem, we have
where we use the fact that J is finite (see Section 3.2 or Appendix B for the set J).
$$\begin{array}{ccc}\hfill {\displaystyle \underset{p,q\in \mathsf{\Sigma}(s,{K}_{H},{\u03f5}_{0},{\u03f5}_{\infty})}{sup}}\mathrm{Bias}{\left({\tilde{\mathbf{G}}}_{{w}_{0}}\right)}^{2}& =& {\displaystyle \underset{p,q\in \mathsf{\Sigma}(s,{K}_{H},{\u03f5}_{0},{\u03f5}_{\infty})}{sup}}\frac{{\u03f5}^{2}}{N}{\left({\displaystyle \sum _{i\in J}}{c}_{i}(p,q)\right)}^{2}\hfill \\ & =& \frac{{\u03f5}^{2}}{N}C,\hfill \end{array}$$
References
 Cover, T.M.; Thomas, J.A. Elements of Information Theory; John Wiley & Sons: New York, NY, USA, 2012. [Google Scholar]
 AviItzhak, H.; Diep, T. Arbitrarily tight upper and lower bounds on the Bayesian probability of error. IEEE Trans. Pattern Anal. Mach. Intell. 1996, 18, 89–91. [Google Scholar] [CrossRef]
 Hashlamoun, W.A.; Varshney, P.K.; Samarasooriya, V. A tight upper bound on the Bayesian probability of error. IEEE Trans. Pattern Anal. Mach. Intell. 1994, 16, 220–224. [Google Scholar] [CrossRef]
 Moon, K.; Delouille, V.; Hero, A.O., III. Meta learning of bounds on the Bayes classifier error. In Proceedings of the 2015 IEEE Signal Processing and Signal Processing Education Workshop (SP/SPE), Salt Lake City, UT, USA, 9–12 August 2015; pp. 13–18. [Google Scholar]
 Chernoff, H. A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. Ann. Math. Stat. 1952, 23, 493–507. [Google Scholar] [CrossRef]
 Berisha, V.; Wisler, A.; Hero, A.O., III; Spanias, A. Empirically Estimable Classification Bounds Based on a New Divergence Measure. IEEE Trans. Signal Process. 2016, 64, 580–591. [Google Scholar] [CrossRef] [PubMed]
 Moon, K.R.; Hero, A.O., III. Multivariate fDivergence Estimation With Confidence. In Proceedings of the 27th International Conference on Neural Information Processing Systems (NIPS 2014), Montreal, QC, Canada, 8–13 December 2014; pp. 2420–2428. [Google Scholar]
 Gliske, S.V.; Moon, K.R.; Stacey, W.C.; Hero, A.O., III. The intrinsic value of HFO features as a biomarker of epileptic activity. In Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai, China, 20–25 March 2016. [Google Scholar]
 Loh, P.L. On Lower Bounds for Statistical Learning Theory. Entropy 2017, 19, 617. [Google Scholar] [CrossRef]
 Póczos, B.; Schneider, J.G. On the estimation of alphadivergences. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL, USA, 11–13 April 2011; pp. 609–617. [Google Scholar]
 Oliva, J.; Póczos, B.; Schneider, J. Distribution to distribution regression. In Proceedings of the 30th International Conference on Machine Learning, Atlanta, GA, USA, 16–21 June 2013; pp. 1049–1057. [Google Scholar]
 Szabó, Z.; Gretton, A.; Póczos, B.; Sriperumbudur, B. Twostage sampled learning theory on distributions. In Proceeding of The 18th International Conference on Artificial Intelligence and Statistics, San Diego, CA, USA, 9–12 May 2015. [Google Scholar]
 Moon, K.R.; Delouille, V.; Li, J.J.; De Visscher, R.; Watson, F.; Hero, A.O., III. Image patch analysis of sunspots and active regions. II. Clustering via matrix factorization. J. Space Weather Space Clim. 2016, 6, A3. [Google Scholar] [CrossRef]
 Moon, K.R.; Li, J.J.; Delouille, V.; De Visscher, R.; Watson, F.; Hero, A.O., III. Image patch analysis of sunspots and active regions. I. Intrinsic dimension and correlation analysis. J. Space Weather Space Clim. 2016, 6, A2. [Google Scholar] [CrossRef]
 Dhillon, I.S.; Mallela, S.; Kumar, R. A divisive information theoretic feature clustering algorithm for text classification. J. Mach. Learn. Res. 2003, 3, 1265–1287. [Google Scholar]
 Banerjee, A.; Merugu, S.; Dhillon, I.S.; Ghosh, J. Clustering with Bregman divergences. J. Mach. Learn. Res. 2005, 6, 1705–1749. [Google Scholar]
 Lewi, J.; Butera, R.; Paninski, L. Realtime adaptive informationtheoretic optimization of neurophysiology experiments. In Proceedings of the 19th International Conference on Neural Information Processing Systems (NIPS 2006), Vancouver, BC, Canada, 4–9 December 2006; pp. 857–864. [Google Scholar]
 Bruzzone, L.; Roli, F.; Serpico, S.B. An extension of the JeffreysMatusita distance to multiclass cases for feature selection. IEEE Trans. Geosci. Remote Sens. 1995, 33, 1318–1321. [Google Scholar] [CrossRef]
 Guorong, X.; Peiqi, C.; Minhui, W. Bhattacharyya distance feature selection. In Proceedings of the 13th International Conference on Pattern Recognition, Vienna, Austria, 25–29 August 1996; Volume 2, pp. 195–199. [Google Scholar]
 Sakate, D.M.; Kashid, D.N. Variable selection via penalized minimum φdivergence estimation in logistic regression. J. Appl. Stat. 2014, 41, 1233–1246. [Google Scholar] [CrossRef]
 Hild, K.E.; Erdogmus, D.; Principe, J.C. Blind source separation using Renyi’s mutual information. IEEE Signal Process. Lett. 2001, 8, 174–176. [Google Scholar] [CrossRef]
 Mihoko, M.; Eguchi, S. Robust blind source separation by beta divergence. Neural Comput. 2002, 14, 1859–1886. [Google Scholar] [CrossRef] [PubMed]
 Vemuri, B.C.; Liu, M.; Amari, S.; Nielsen, F. Total Bregman divergence and its applications to DTI analysis. IEEE Trans. Med. Imaging 2011, 30, 475–483. [Google Scholar] [CrossRef] [PubMed]
 Hamza, A.B.; Krim, H. Image registration and segmentation by maximizing the JensenRényi divergence. In Proceedings of the 4th International Workshop Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR 2003), Lisbon, Portugal, 7–9 July 2003; pp. 147–163. [Google Scholar]
 Liu, G.; Xia, G.; Yang, W.; Xue, N. SAR image segmentation via nonlocal active contours. In Proceedings of the 2014 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Quebec City, QC, Canada, 13–18 July 2014; pp. 3730–3733. [Google Scholar]
 Korzhik, V.; Fedyanin, I. Steganographic applications of the nearestneighbor approach to KullbackLeibler divergence estimation. In Proceedings of the 2015 Third International Conference on Digital Information, Networking, and Wireless Communications (DINWC), Moscow, Russia, 3–5 February 2015; pp. 133–138. [Google Scholar]
 Basseville, M. Divergence measures for statistical data processing–An annotated bibliography. Signal Process. 2013, 93, 621–633. [Google Scholar] [CrossRef]
 Cichocki, A.; Amari, S. Families of alphabetaand gammadivergences: Flexible and robust measures of similarities. Entropy 2010, 12, 1532–1568. [Google Scholar] [CrossRef]
 Csiszar, I. Informationtype measures of difference of probability distributions and indirect observations. Stud. Sci. Math. Hungar. 1967, 2, 299–318. [Google Scholar]
 Ali, S.M.; Silvey, S.D. A general class of coefficients of divergence of one distribution from another. J. R. Stat. Soc. Ser. B Stat. Methodol. 1966, 28, 131–142. [Google Scholar]
 Kullback, S.; Leibler, R.A. On information and sufficiency. Ann. Math. Stat. 1951, 22, 79–86. [Google Scholar] [CrossRef]
 Rényi, A. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Berkeley, CA, USA, 20 June–30 July 1960; University of California Press: Berkeley, CA, USA, 1961; pp. 547–561. [Google Scholar]
 Hellinger, E. Neue Begründung der Theorie quadratischer Formen von unendlichvielen Veränderlichen. J. Rein. Angew. Math. 1909, 136, 210–271. (In German) [Google Scholar]
 Bhattacharyya, A. On a measure of divergence between two multinomial populations. Indian J. Stat. 1946, 7, 401–406. [Google Scholar]
 Silva, J.F.; Parada, P.A. Shannon entropy convergence results in the countable infinite case. In Proceedings of the 2012 IEEE International Symposium on Information Theory Proceedings (ISIT), Cambridge, MA, USA, 1–6 July 2012; pp. 155–159. [Google Scholar]
 Antos, A.; Kontoyiannis, I. Convergence properties of functional estimates for discrete distributions. Random Struct. Algorithms 2001, 19, 163–193. [Google Scholar] [CrossRef]
 Valiant, G.; Valiant, P. Estimating the unseen: An n/log (n)sample estimator for entropy and support size, shown optimal via new CLTs. In Proceedings of the 43rd Annual ACM Symposium on Theory of Computing, San Jose, CA, USA, 6–8 June 2011; pp. 685–694. [Google Scholar]
 Jiao, J.; Venkat, K.; Han, Y.; Weissman, T. Minimax estimation of functionals of discrete distributions. IEEE Trans. Inf. Theory 2015, 61, 2835–2885. [Google Scholar] [CrossRef] [PubMed]
 Jiao, J.; Venkat, K.; Han, Y.; Weissman, T. Maximum likelihood estimation of functionals of discrete distributions. IEEE Trans. Inf. Theory 2017, 63, 6774–6798. [Google Scholar] [CrossRef]
 Valiant, G.; Valiant, P. The power of linear estimators. In Proceedings of the 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science (FOCS), Palm Springs, CA, USA, 22–25 October 2011; pp. 403–412. [Google Scholar]
 Paninski, L. Estimation of entropy and mutual information. Neural Comput. 2003, 15, 1191–1253. [Google Scholar] [CrossRef]
 Paninski, L. Estimating entropy on m bins given fewer than m samples. IEEE Trans. Inf. Theory 2004, 50, 2200–2203. [Google Scholar] [CrossRef]
 AlbaFernández, M.V.; JiménezGamero, M.D.; ArizaLópez, F.J. Minimum Penalized ϕDivergence Estimation under Model Misspecification. Entropy 2018, 20, 329. [Google Scholar] [CrossRef]
 Ahmed, N.A.; Gokhale, D. Entropy expressions and their estimators for multivariate distributions. IEEE Trans. Inf. Theory 1989, 35, 688–692. [Google Scholar] [CrossRef]
 Misra, N.; Singh, H.; Demchuk, E. Estimation of the entropy of a multivariate normal distribution. J. Multivar. Anal. 2005, 92, 324–342. [Google Scholar] [CrossRef]
 Gupta, M.; Srivastava, S. Parametric Bayesian estimation of differential entropy and relative entropy. Entropy 2010, 12, 818–843. [Google Scholar] [CrossRef]
 Li, S.; Mnatsakanov, R.M.; Andrew, M.E. Knearest neighbor based consistent entropy estimation for hyperspherical distributions. Entropy 2011, 13, 650–667. [Google Scholar] [CrossRef]
 Wang, Q.; Kulkarni, S.R.; Verdú, S. Divergence estimation for multidimensional densities via knearestneighbor distances. IEEE Trans. Inf. Theory 2009, 55, 2392–2405. [Google Scholar] [CrossRef]
 Darbellay, G.A.; Vajda, I. Estimation of the information by an adaptive partitioning of the observation space. IEEE Trans. Inf. Theory 1999, 45, 1315–1321. [Google Scholar] [CrossRef]
 Silva, J.; Narayanan, S.S. Information divergence estimation based on datadependent partitions. J. Stat. Plan. Inference 2010, 140, 3180–3198. [Google Scholar] [CrossRef]
 Le, T.K. Information dependency: Strong consistency of Darbellay–Vajda partition estimators. J. Stat. Plan. Inference 2013, 143, 2089–2100. [Google Scholar] [CrossRef]
 Wang, Q.; Kulkarni, S.R.; Verdú, S. Divergence estimation of continuous distributions based on datadependent partitions. IEEE Trans. Inf. Theory 2005, 51, 3064–3074. [Google Scholar] [CrossRef]
 Hero, A.O., III; Ma, B.; Michel, O.; Gorman, J. Applications of entropic spanning graphs. IEEE Signal Process. Mag. 2002, 19, 85–95. [Google Scholar] [CrossRef]
 Moon, K.R.; Hero, A.O., III. Ensemble estimation of multivariate fdivergence. In Proceedings of the 2014 IEEE International Symposium on Information Theory (ISIT), Honolulu, HI, USA, 29 June–4 July 2014; pp. 356–360. [Google Scholar]
 Moon, K.R.; Sricharan, K.; Greenewald, K.; Hero, A.O., III. Improving convergence of divergence functional ensemble estimators. In Proceedings of the 2016 IEEE International Symposium on Information Theory (ISIT), Barcelona, Spain, 10–15 July 2016; pp. 1133–1137. [Google Scholar]
 Nguyen, X.; Wainwright, M.J.; Jordan, M.I. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Trans. Inf. Theory 2010, 56, 5847–5861. [Google Scholar] [CrossRef]
 Krishnamurthy, A.; Kandasamy, K.; Poczos, B.; Wasserman, L. Nonparametric Estimation of Renyi Divergence and Friends. In Proceedings of the 31st International Conference on Machine Learning, Beijing, China, 21–26 June 2014; pp. 919–927. [Google Scholar]
 Singh, S.; Póczos, B. Generalized exponential concentration inequality for Rényi divergence estimation. In Proceedings of the 31st International Conference on Machine Learning (ICML14), Beijing, China, 21–26 June 2014; pp. 333–341. [Google Scholar]
 Singh, S.; Póczos, B. Exponential Concentration of a Density Functional Estimator. In Proceedings of the 27th International Conference on Neural Information Processing Systems (NIPS 2014), Montreal, QC, Canada, 8–13 December 2014; pp. 3032–3040. [Google Scholar]
 Kandasamy, K.; Krishnamurthy, A.; Poczos, B.; Wasserman, L.; Robins, J. Nonparametric von Mises Estimators for Entropies, Divergences and Mutual Informations. In Proceedings of the 28th International Conference on Neural Information Processing Systems (NIPS 2015), Montreal, QC, Canada, 7–12 December 2015; pp. 397–405. [Google Scholar]
 Härdle, W. Applied Nonparametric Regression; Cambridge University Press: Cambridge, UK, 1990. [Google Scholar]
 Berlinet, A.; Devroye, L.; Györfi, L. Asymptotic normality of L1error in density estimation. Statistics 1995, 26, 329–343. [Google Scholar] [CrossRef]
 Berlinet, A.; Györfi, L.; Dénes, I. Asymptotic normality of relative entropy in multivariate density estimation. Publ. l’Inst. Stat. l’Univ. Paris 1997, 41, 3–27. [Google Scholar]
 Bickel, P.J.; Rosenblatt, M. On some global measures of the deviations of density function estimates. Ann. Stat. 1973, 1, 1071–1095. [Google Scholar] [CrossRef]
 Sricharan, K.; Wei, D.; Hero, A.O., III. Ensemble estimators for multivariate entropy estimation. IEEE Trans. Inf. Theory 2013, 59, 4374–4388. [Google Scholar] [CrossRef] [PubMed]
 Berrett, T.B.; Samworth, R.J.; Yuan, M. Efficient multivariate entropy estimation via knearest neighbour distances. arXiv 2017. [Google Scholar]
 Kozachenko, L.; Leonenko, N.N. Sample estimate of the entropy of a random vector. Probl. Peredachi Inf. 1987, 23, 9–16. [Google Scholar]
 Hansen, B.E.; (University of Wisconsin, Madison, WI, USA). Lecture Notes on Nonparametrics. 2009. [Google Scholar]
 Budka, M.; Gabrys, B.; Musial, K. On accuracy of PDF divergence estimators and their applicability to representative data sampling. Entropy 2011, 13, 1229–1266. [Google Scholar] [CrossRef]
 Efron, B.; Stein, C. The jackknife estimate of variance. Ann. Stat. 1981, 9, 586–596. [Google Scholar] [CrossRef]
 Wisler, A.; Moon, K.; Berisha, V. Direct ensemble estimation of density functionals. In Proceedings of the 2018 International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Calgary, AB, Canada, 15–20 April 2018. [Google Scholar]
 Moon, K.R.; Sricharan, K.; Greenewald, K.; Hero, A.O., III. Nonparametric Ensemble Estimation of Distributional Functionals. arXiv 2016. [Google Scholar]
 Paul, F.; Arkin, Y.; Giladi, A.; Jaitin, D.A.; Kenigsberg, E.; KerenShaul, H.; Winter, D.; LaraAstiaso, D.; Gury, M.; Weiner, A.; et al. Transcriptional heterogeneity and lineage commitment in myeloid progenitors. Cell 2015, 163, 1663–1677. [Google Scholar] [CrossRef] [PubMed]
 Kanehisa, M.; Goto, S. KEGG: Kyoto encyclopedia of genes and genomes. Nucleic Acids Res. 2000, 28, 27–30. [Google Scholar] [CrossRef] [PubMed]
 Kanehisa, M.; Sato, Y.; Kawashima, M.; Furumichi, M.; Tanabe, M. KEGG as a reference resource for gene and protein annotation. Nucleic Acids Res. 2015, 44, D457–D462. [Google Scholar] [CrossRef] [PubMed]
 Kanehisa, M.; Furumichi, M.; Tanabe, M.; Sato, Y.; Morishima, K. KEGG: New perspectives on genomes, pathways, diseases and drugs. Nucleic Acids Res. 2016, 45, D353–D361. [Google Scholar] [CrossRef] [PubMed]
 Van Dijk, D.; Sharma, R.; Nainys, J.; Yim, K.; Kathail, P.; Carr, A.J.; Burdsiak, C.; Moon, K.R.; Chaffer, C.; Pattabiraman, D.; et al. Recovering Gene Interactions from SingleCell Data Using Data Diffusion. Cell 2018, 174, 716–729. [Google Scholar] [CrossRef] [PubMed]
 Moon, K.R.; Sricharan, K.; Hero, A.O., III. Ensemble Estimation of Distributional Functionals via kNearest Neighbors. arXiv 2017. [Google Scholar]
 Durrett, R. Probability: Theory and Examples; Cambridge University Press: Cambridge, UK, 2010. [Google Scholar]
 Gut, A. Probability: A Graduate Course; Springer: Berlin/Heidelberg, Germany, 2012. [Google Scholar]
 Munkres, J. Topology; Prentice Hall: Englewood Cliffs, NJ, USA, 2000. [Google Scholar]
 Evans, L.C. Partial Differential Equations; American Mathematical Society: Providence, RI, USA, 2010. [Google Scholar]
 Gilbarg, D.; Trudinger, N.S. Elliptic Partial Differential Equations of Second Order; Springer: Berlin/Heidelberg, Germany, 2001. [Google Scholar]
Figure 1.
Heat map showing the predicted bias of the divergence functional plugin estimator ${\tilde{\mathbf{G}}}_{h}$ based on Theorem 1 as a function of the dimensions (d) and sample size (N) when $h={N}^{\frac{1}{d+1}}$. Note that the phase transition in the bias as the dimensions (d) increase for a fixed sample size (N); the bias remains small only for relatively small values of $d$. The proposed weighted ensemble estimator EnDive eliminates this phase transition when the densities and the function g are sufficiently smooth.
Figure 2.
The optimal weights from (6) when $d=4$, $N=3100$, $L=50$, and l are uniformly spaced between 1.5 and 3. The lowest values of l are given the highest weight. Thus, the minimum value of bandwidth parameters $\mathcal{L}$ should be sufficiently large to render an adequate estimate.
Figure 3.
(Left) Log–log plot of MSE of the uniform kernel plugin (“Kernel”) and the optimally weighted EnDive estimator for various dimensions and sample sizes. (Right) Plot of the true values being estimated compared to the average values of the same estimators with standard error bars. The proposed weighted ensemble estimator approaches the theoretical rate (see Table 1), performed better than the plugin estimator in terms of MSE and was less biased.
Figure 4.
Log–log plot of the average pointwise squared error between the KDE ${\tilde{\mathbf{f}}}_{1,h}$ and ${f}_{1}$ for various dimensions and sample sizes using the same bandwidth and kernel as the standard plugin estimators in Figure 3. The KDE and the density were compared at 10,000 points sampled from ${f}_{1}$.
Figure 5.
QQplots comparing the quantiles of a standard normal random variable and the quantiles of the centered and scaled EnDive estimator applied to the Kullback–Leibler (KL) divergence when the distributions were the same and different. Quantiles were computed from 10,000 trials. These plots correspond to the same experiments as in Table 2 when N = 100 and N = 1000. The correspondence between quantiles is high for all cases.
Figure 6.
Estimated upper (UB) and lower bounds (LB) on the Bayes error rate (BER) based on estimating the HP divergence between two 10dimensional Gaussian distributions with identity covariance matrices and distances between means of 1 (left) and 3 (right), respectively. Estimates were calculated using EnDive, with error bars indicating the standard deviation from 400 trials. The upper bound was closer, on average, to the true BER when N was small (≈100–300) and the distance between the means was small. The lower bound was closer, on average, in all other cases.
Table 1.
Negative log–log slope of the EnDive mean squared error (MSE) as a function of the sample size for various dimensions. The slope was calculated beginning at ${N}_{start}$. The negative slope was closer to 1 with ${N}_{start}={10}^{2.375}$ than for ${N}_{start}={10}^{2}$ indicating that the asymptotic rate had not yet taken effect at ${N}_{start}={10}^{2}$.
Estimator  $\mathit{d}=5$  $\mathit{d}=10$  $\mathit{d}=15$ 

${N}_{start}={10}^{2}$  $0.85$  $0.84$  $0.80$ 
${N}_{start}={10}^{2.375}$  $0.96$  $0.96$  $0.95$ 
Table 2.
Comparison between quantiles of a standard normal random variable and the quantiles of the centered and scaled EnDive estimator applied to the KL divergence when the distributions were the same and different. Quantiles were computed from 10,000 trials. The parameter $\rho $ gives the correlation coefficient between the quantiles, while $\beta $ is the estimated slope between the quantiles. The correspondence between quantiles was very high for all cases.
N  Same  Different  

$\mathbf{1}\mathbf{}\mathit{\rho}$  $\mathit{\beta}$  $\mathbf{1}\mathbf{}\mathit{\rho}$  $\mathit{\beta}$  
100  $2.35\times {10}^{4}$  $1.014$  $9.97\times {10}^{4}$  $0.993$ 
500  $9.48\times {10}^{5}$  $1.007$  $5.06\times {10}^{4}$  $0.999$ 
1000  $8.27\times {10}^{5}$  $0.996$  $4.30\times {10}^{4}$  $0.988$ 
5000  $8.59\times {10}^{5}$  $0.995$  $4.47\times {10}^{4}$  $1.005$ 
Table 3.
Misclassification rate of a quadratic discriminant analysis classifier (QDA) classifier and estimated upper bounds (UB) and lower bounds (LB) of the pairwise BER between mouse bone marrow cell types using the Henze–Penrose divergence applied to different combinations of genes selected from the KEGG pathways associated with the hematopoietic cell lineage. Results are presented as percentages in the form of mean ± standard deviation. Based on these results, erythrocytes are relatively easy to distinguish from the other two cell types using these gene sets.
Platelets  Erythrocytes  Neutrophils  Macrophages  Random  

Eryth. vs. Mono., LB  $2.8\pm 1.5$  $1.2\pm 0.6$  $0.6\pm 0.6$  $8.5\pm 1.2$  $14.4\pm 8.4$ 
Eryth. vs. Mono., UB  $5.3\pm 2.9$  $2.4\pm 1.3$  $1.2\pm 1.3$  $15.5\pm 1.9$  $23.2\pm 12.3$ 
Eryth. vs. Mono., Prob. Error  $0.9$  $0.4$  $1.3$  $3.4$  $7.2\pm 5.4$ 
Eryth. vs. Baso., LB  $0.5\pm 0.6$  $0.05\pm 0.12$  $0.6\pm 0.5$  $5.1\pm 0.9$  $11.9\pm 5.5$ 
Eryth. vs. Baso., UB  $1.0\pm 1.1$  $0.1\pm 0.2$  $1.1\pm 0.9$  $9.6\pm 1.6$  $20.3\pm 8.8$ 
Eryth. vs. Baso., Prob. Error  $1.2$  $0.3$  $1.9$  $3.6$  $6.8\pm 5.0$ 
Baso. vs. Mono., LB  $31.1\pm 1.8$  $27.8\pm 3.1$  $27.1\pm 2.6$  $31.6\pm 1.3$  $32.1\pm 2.6$ 
Baso. vs. Mono., UB  $42.8\pm 1.4$  $39.9\pm 2.8$  $39.4\pm 2.4$  $43.2\pm 1.0$  $43.5\pm 1.2$ 
Baso. vs. Mono., Prob. Error  $28.8$  $30.9$  $23.9$  $22.4$  $29.7\pm 5.7$ 
© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).