Statistical Estimation of the Kullback–Leibler Divergence

Abstract

Asymptotic unbiasedness and $L^2$-consistency are established, under mild conditions, for the estimates of the Kullback–Leibler divergence between two probability measures in ${\mathbb{R}}^d$, absolutely continuous with respect to (w.r.t.) the Lebesgue measure. These estimates are based on certain k-nearest neighbor statistics for pair of independent identically distributed (i.i.d.) due vector samples. The novelty of results is also in treating mixture models. In particular, they cover mixtures of nondegenerate Gaussian measures. The mentioned asymptotic properties of related estimators for the Shannon entropy and cross-entropy are strengthened. Some applications are indicated.

1. Introduction

The Kullback–Leibler divergence introduced in [1] is used for quantification of similarity of two probability measures. It plays important role in various domains such as statistical inference (see, e.g., [2,3]), metric learning [4,5], machine learning [6,7], computer vision [8,9], network security [10], feature selection and classification [11,12,13], physics [14], biology [15], medicine [16,17], finance [18], among others. It is worth to emphasize that mutual information, widely used in many research directions (see, e.g., [19,20,21,22,23]), is a special case of the Kullback–Leibler divergence for certain measures. Moreover, the Kullback–Leibler divergence itself belongs to a class of f-divergence measures (with $f\left(t\right)=logt$). For comparison of various f-divergence measures see, e.g., [24], their estimates are considered in [25,26].

Let $\mathbb{P}$ and $\mathbb{Q}$ be two probability measures on a measurable space $(S,\mathcal{B})$. The Kullback–Leibler divergence between $\mathbb{P}$ and $\mathbb{Q}$ is defined, according to [1], by way of

$$D\left(\mathbb{P}\right|\left|\mathbb{Q}\right):=\left\{\begin{array}{cc}\underset{\mathit{S}}{\int }log\left(\frac{d\mathbb{P}}{d\mathbb{Q}}\right)d\mathbb{P}\hfill & \mathrm{if}\mathbb{P}\ll \mathbb{Q},\hfill \\ \infty \mathrm{otherwise},\hfill \end{array}\right.$$

(1)

where $\frac{d\mathbb{P}}{d\mathbb{Q}}$ stands for the Radon–Nikodym derivative. The integral in (1) can take values in $[0,\infty ]$. We employ the base e of logarithms since a constant factor is not essential here.

If $(S,\mathcal{B})=({\mathbb{R}}^d,\mathcal{B}\left({\mathbb{R}}^d\right))$, where $d\in \mathbb{N}$, and (absolutely continuous) $\mathbb{P}$ and $\mathbb{Q}$ have densities, $p\left(x\right)$ and $q\left(x\right)$, $x\in {\mathbb{R}}^d$, w.r.t. the Lebesgue measure $\mu$, then (1) can be expressed as

$$D\left(\mathbb{P}\right|\left|\mathbb{Q}\right)=\underset{{\mathbb{R}}^d}{\int }p\left(x\right)log\left(\frac{p\left(x\right)}{q\left(x\right)}\right)dx,$$

(2)

where we write $dx$ instead of $\mu \left(dx\right)$ to simplify notation. One formally sets $0/0:=0$, $a/0:=\infty$ if $a>0$, $log0:=-\infty$, $log(\infty ):=\infty$ and $0log0:=0$. Then $log\left(\frac{p\left(x\right)}{q\left(x\right)}\right)$ is a measurable function with values in $[-\infty ,\infty ]$. So, the right-hand sides of (1) and (2) coincide. Formula (2) is justified by Lemma A1, see Appendix A.

Denote by $S\left(f\right):=\{x\in {\mathbb{R}}^d:f\left(x\right)>0\}$ the support of a (version of) probability density f. The integral in (2) is taken over $S\left(p\right)$ and does not depend on the choice of p and q versions.

The following two functionals are closely related to the Kullback - Leibler divergence. For probability measures $\mathbb{P}$ and $\mathbb{Q}$ on $({\mathbb{R}}^d,\mathcal{B}\left({\mathbb{R}}^d\right))$ having densities $p\left(x\right)$ and $q\left(x\right)$, $x\in {\mathbb{R}}^d$, w.r.t. the Lebesgue measure $\mu$, one can introduce, according to [27], p. 35, entropy H (also called the Shannon differential entropy) and cross-entropy C as follows

$$H\left(\mathbb{P}\right):=-{\int }_{{\mathbb{R}}^d}p\left(x\right)logp\left(x\right)dx,C(\mathbb{P},\mathbb{Q}):=-{\int }_{{\mathbb{R}}^d}p\left(x\right)logq\left(x\right)dx.$$

In view of (2), $D\left(\mathbb{P}\right|\left|\mathbb{Q}\right)=C(\mathbb{P},\mathbb{Q})-H\left(\mathbb{P}\right)$ whenever the right-hand side is well defined.

Usually one constructs statistical estimates of some characteristics of a stochastic model under consideration relying on a collection of observations. In the pioneering paper [28] the estimator of the Shannon differential entropy was proposed, based on the nearest neighbor statistics. In a series of papers this estimate was studied and applied. Moreover, estimators of the Rényi entropy, mutual information and the Kullback–Leibler divergence have appeared (see, e.g., [29,30,31]). However, the authors of [32] indicated the occurrence of gaps in the known proofs concerning the limit behavior of such statistics. Almost all of these flaws refer to the lack of proved correctness of using the (reversed) Fatou lemma (see, e.g., [28], inequality after the statement (21), or [31], inequality (91)) or the generalized Helly–Bray lemma (see, e.g., [30], page 2171). One can find these lemmas in [33], p. 233, and [34], p. 187. Paper [32] has attracted our attention and motivated study of the declared asymptotic properties. Furthermore, we would like to highlight the important role of the papers [28,30,31,32]. Thus, in a recent work [35] the new functionals were introduced to prove asymptotic unbiasedness and $L^2$-consistency of the Kozachenko–Leonenko estimators of the Shannon differential entropy. We used the criterion of uniform integrability, for different families of functions, to avoid employment of the Fatou lemma since it is not clear whether one could indicate the due majorizing functions for those families. The present paper is aimed at extension of our approach to grasp the Kullback–Leibler divergence estimation. Instead of the nearest neighbor statistics we employ the k-nearest neighbor statistics (on order statistics see, e.g., [36]) and also use more general forms of the mentioned functionals.

Note in passing that there exist investigations treating important aspects of the entropy, Kullback–Leibler divergence and mutual information estimation. The mixed models and conditional entropy estimation are studied, e.g., in [37,38]. The central limit theorem (CLT) for the Kozachenko–Leonenko estimates is established in [39]. In [40], deep analysis of efficiency of functional weighted estimates was performed (including CLT). The limit theorems for point processes on manifolds are employed in [41] to analyze behavior of the Shannon and the Rényi entropy estimates. The convergence rates for the Shannon entropy (truncated) estimates are obtained in [42] for one-dimensional case, see also [43] for multidimensional case. A kernel density plug-in estimator of the various divergence functionals is studied in [25]. The principal assumptions of that paper are the following: the densities are smooth and have common bounded support S, they are strictly lower bounded on S, moreover, the set S is smooth with respect to the employed kernel. Ensemble estimation of various divergence functionals is studied in [25]. Profound results for smooth bounded densities are established in recent work [44]. The mutual information estimation by the local Gaussian approximation is developed in [45]. Note that various deep results (including the central limit theorem) were obtained for the Kullback–Leibler estimates under certain conditions imposed on derivatives of unknown densities (see, e.g., the recent papers [25,46]). In a series of papers the authors demand boundedness of densities to prove $L^2$-consistency for the Kozachenko–Leonenko estimates of differential Shannon entropy (see, e.g., [47]).

Our goal is to provide wide conditions for the asymptotic unbiasedness and $L^2$-consistency of the specified Kullback–Leibler divergence estimates without such smoothness and boundedness hypotheses. Furthermore, we do not assume that densities have bounded supports. As a byproduct we obtain new results concerning Shannon differential entropy and cross-entropy.

We employ probabilistic and analytical techniques, namely, weak convergence of probability measures, conditional expectations, regular probability distributions, k-nearest neighbor statistics, probability inequalities, integration by parts in the Lebesgue–Stieltjes integral, analysis of integrals depending on certain parameters and taken over specified domains, criterion of the uniform integrability of various families of functions, slowly varying functions.

The paper is organized as follows. In Section 2, we introduce some notation. In Section 3 we formulate main results, i.e., Theorems 1 and 2. Their proofs are provided in Section 4 and Section 5, respectively. Section 6 contains concluding remarks and perspectives of future research. Proofs of several lemmas are given in Appendix A.

2. Notation

Let X and Y be random vectors taking values in ${\mathbb{R}}^d$ and having distributions ${\mathsf{P}}_X$ and ${\mathsf{P}}_Y$, respectively, (below we will take $\mathbb{P}={\mathsf{P}}_X$ and $\mathbb{Q}={\mathsf{P}}_Y$). Consider random vectors $X_1,X_2,\dots$ and $Y_1,Y_2,\dots$ with values in ${\mathbb{R}}^d$ such that $law\left(X_i\right)=law\left(X\right)$ and $law\left(Y_i\right)=law\left(Y\right)$, $i\in \mathbb{N}$. Assume also that $\{X_i,Y_i,i\in \mathbb{N}\}$ are independent. We are interested in statistical estimation of $D\left({\mathsf{P}}_X\right|\left|{\mathsf{P}}_Y\right)$ constructed by means of observations ${\mathbb{X}}_n:=\{X_1,\dots ,X_n\}$ and ${\mathbb{Y}}_m:=\{Y_1,\dots ,Y_m\}$, $n,m\in \mathbb{N}$. All the random variables under consideration are defined on a probability space $(\mathsf{\Omega },\mathcal{F},\mathsf{P})$, each measure space is assumed complete.

For a finite set $E=\{z_1,\dots ,z_N\}\subset {\mathbb{R}}^d$, where $z_i\ne z_j$$(i\ne j)$, and a vector $v\in {\mathbb{R}}^d$, renumerate points of E as $z_{\left(1\right)}\left(v\right),\dots ,z_{\left(N\right)}\left(v\right)$ in such a way that $\parallel v-z_{\left(1\right)}\parallel \le \dots \le \parallel v-z_{\left(N\right)}\parallel$, $\parallel ·\parallel$ is the Euclidean norm in ${\mathbb{R}}^d$. If there are points $z_{i_1},\dots ,z_{i_s}$ having the same distance from v then we numerate them according to the indexes $i_1,\dots ,i_s$ increase. In other words, for $k=1,\dots ,N$, $z_{\left(k\right)}\left(v\right)$ is the k-nearest neighbor of v in a set E. To indicate that $z_{\left(k\right)}\left(v\right)$ is constructed by means of E we write $z_{\left(k\right)}(v,E)$. Fix $k\in \{1,\dots ,n-1\}$, $l\in \{1,\dots ,m\}$ and (for each $\omega \in \mathsf{\Omega }$) put

$$R_{n,k}\left(i\right):=\parallel X_i-X_{\left(k\right)}(X_i,{\mathbb{X}}_n\backslash \left\{X_i\right\})\parallel ,V_{m,l}\left(i\right):=\parallel X_i-Y_{\left(l\right)}(X_i,{\mathbb{Y}}_m)\parallel ,i=1,\dots ,n.$$

We assume that X and Y have densities $p=\frac{d{\mathsf{P}}_X}{d\mu }$ and $q=\frac{d{\mathsf{P}}_Y}{d\mu }$. Then with probability one all the points in ${\mathbb{X}}_n$ are distinct as well as points of ${\mathbb{Y}}_m$.

Following [31] (see Formula (17) there) introduce an estimate of $D\left({\mathsf{P}}_X\right|\left|{\mathsf{P}}_Y\right)$

$${\tilde{D}}_{n,m}({\mathcal{K}}_n,{\mathcal{L}}_n):=\frac{1}{n}\sum _{i=1}^n\left(\psi \left(k_i\right)-\psi \left(l_i\right)\right)+log\left(\frac{m}{n-1}\right)+\frac{d}{n}\sum _{i=1}^{n}log\left(\frac{V_{m,l_i}\left(i\right)}{R_{n,k_i}\left(i\right)}\right),$$

(3)

where $\psi \left(t\right)=\frac{d}{dt}log\mathsf{\Gamma }\left(t\right)=\frac{{\mathsf{\Gamma }}^{\prime }\left(t\right)}{\mathsf{\Gamma }\left(t\right)}$ is the digamma function, $t>0$, ${\mathcal{K}}_n:={\left\{k_i\right\}}_{i=1}^n$, ${\mathcal{L}}_n:={\left\{l_i\right\}}_{i=1}^n$ are collections of integers and, for some $r\in \mathbb{N}$ and all $i\in \mathbb{N}$, $k_i\le r$, $l_i\le r$. Note that (3) is well-defined for $n\ge {max}_{i=1,\dots ,n}{k}_i+1$, $m\ge {max}_{i=1,\dots ,n}{l}_i$. If $k_i=k$ and $l_i=l$, $i=1,\dots ,n$, then, for $n\ge k+1$ and $m\ge l$, we write

$${\widehat{D}}_{n,m}(k,l):=\psi \left(k\right)-\psi \left(l\right)+log\left(\frac{m}{n-1}\right)+\frac{d}{n}\sum _{i=1}^{n}log\left(\frac{V_{m,l}\left(i\right)}{R_{n,k}\left(i\right)}\right).$$

(4)

If $k=l$ then

$${\widehat{D}}_{n,m}\left(k\right)=log\left(\frac{m}{n-1}\right)+\frac{d}{n}\sum _{i=1}^{n}log\left(\frac{V_{m,k}\left(i\right)}{R_{n,k}\left(i\right)}\right)$$

and we come to formula (5) in [31]. For an intuitive background of the proposed estimates one can address [31] (Introduction, Parts B and C).

We write $B(x,r):=\{y\in {\mathbb{R}}^d:\parallel x-y\parallel \le r\}$ for $x\in {\mathbb{R}}^d$, $r>0$, and $V_d=\mu \left(B(0,1)\right)$ is the volume of the unit ball in ${\mathbb{R}}^d$. Similar to (3) with the same notation and the same conditions for $k_i$ and $l_i$, $i=1,\dots ,n$, one can define the Kozachenko - Leonenko type estimates of $H\left({\mathsf{P}}_X\right)$ and $C({\mathsf{P}}_X,{\mathsf{P}}_Y)$, respectively, by formulas

$${\tilde{H}}_n\left({\mathcal{K}}_n\right):=-\frac{1}{n}\sum _{i=1}^n\psi \left(k_i\right)+log{V}_d+log(n-1)+\frac{d}{n}\sum _{i=1}^{n}log{R}_{n,k_i}\left(i\right),$$

(5)

$${\tilde{C}}_{n,m}\left({\mathcal{L}}_n\right):=-\frac{1}{n}\sum _{i=1}^n\psi \left(l_i\right)+log{V}_d+logm+\frac{d}{n}\sum _{i=1}^{n}log{V}_{m,l_i}\left(i\right).$$

(6)

In [28], an estimate (5) was proposed for $k_i=1$, $i=1,\dots ,n$. If $k_i=k$, $l_i=l$, $i=1,\dots ,n$, $n\ge k+1$ and $m\ge l$, then one has

$${\widehat{H}}_n\left(k\right):=\frac{1}{n}\sum _{i=1}^{n}log\left(\frac{V_d{R}_{n,k}^d\left(i\right)(n-1)}{e^{\psi \left(k\right)}}\right),{\widehat{C}}_{n,m}\left(l\right):=\frac{1}{n}\sum _{i=1}^{n}log\left(\frac{V_d{V}_{m,l}^d\left(i\right)m}{e^{\psi \left(l\right)}}\right).$$

(7)

Remark 1.

All our results are valid for statistics (3). To simplify notation we consider estimates (4) since the study of ${\tilde{D}}_{n,m}({\mathcal{K}}_n,{\mathcal{L}}_n)$ follows the same lines. For the same reason, as in the case of Kullback–Leibler divergence, we will only deal with (7) since (5) and (6) can be analyzed in quite the same way.

Some extra notation is necessary. As in [35], given a probability density f in ${\mathbb{R}}^d$, we consider the following functions of $x\in {\mathbb{R}}^d$, $r>0$ and $R>0$, that is, define integral functionals (depending on parameters)

$$I_f(x,r):=\frac{{\int }_{B(x,r)}f\left(y\right)dy}{r^d{V}_d},$$

(8)

$$M_f(x,R):=\underset{r\in (0,R]}{sup}{I}_f(x,r),m_f(x,R):=\underset{r\in (0,R]}{inf}{I}_f(x,r).$$

(9)

Some properties of function ${\int }_{B(x,r)}f\left(y\right)dy$ are demonstrated in [48]. By virtue of Lemma 2.1 [35], for each probability density f, the function $I_f(x,r)$ introduced above is continuous in $(x,r)$ on ${\mathbb{R}}^d\times (0,\infty )$. Hence on account of Theorem 15.84 [49] the functions $m_f(·,R)$ and $M_f(·,R)$ for any $R>0$ have to be upper semicontinuous and lower semicontinuous, respectively. Therefore, Borel measurability of these nonnegative functions ensues from Proposition 15.82 [49]. On the other hand, the function $m_f(x,·)$ is evidently nonincreasing whereas $M_f(x,·)$ is nondecreasing for each x in $R^d$. Notably, changing ${sup}_{r\in (0,R]}$ to ${sup}_{r\in (0,\infty )}$ transforms the function $M_f(x,R)$ into the famous Hardy–Littlewood maximal function $M_f\left(x\right)$ well-known in Harmonic analysis.

Set $e_{[-1]}:=0$ and $e_{\left[N\right]}:=exp\left\{e_{[N-1]}\right\}$, $N\in {\mathbb{Z}}_{+}$. Introduce a function ${log}_{\left[1\right]}\left(t\right):=logt$, $t>0$. For $N\in \mathbb{N}$, $N>1$, set ${log}_{\left[N\right]}\left(t\right):=log\left({log}_{[N-1]}\left(t\right)\right).$ Evidently, this function (for each $N\in \mathbb{N}$) is defined if $t>e_{[N-2]}$. For $N\in \mathbb{N}$, consider the continuous nondecreasing function $G_N:{\mathbb{R}}_{+}\to {\mathbb{R}}_{+}$, given by formula

$$G_N\left(t\right):=\left\{\begin{array}{cc}0,\hfill & t\in [0,e_{[N-1]}],\hfill \\ t{log}_{\left[N\right]}\left(t\right),\hfill & t\in (e_{[N-1]},\infty ).\hfill \end{array}\right.$$

(10)

In other words we employ the function having the form $tr\left(t\right)$ where a function $r\left(t\right)$, taken as N iterations of $logt$, is slowly varying for large t.

For probability densities $p,q$ in ${\mathbb{R}}^d$, $N\in \mathbb{N}$ and positive constants $\nu ,t,\epsilon ,R$, introduce the functionals taking values in $[0,\infty ]$

$$K_{p,q}(\nu ,N,t):=\underset{x,y\in {\mathbb{R}}^d,\parallel x-y\parallel >t}{\int }{G}_N\left(|log∥x-y∥{|}^{\nu }\right)p\left(x\right)q\left(y\right)dxdy,$$

(11)

$$Q_{p,q}(\epsilon ,R):={\int }_{{\mathbb{R}}^d}{M}_q^{\epsilon }(x,R)p\left(x\right)dx,$$

(12)

$$T_{p,q}(\epsilon ,R):={\int }_{{\mathbb{R}}^d}{m}_q^{-\epsilon }(x,R)p\left(x\right)dx.$$

(13)

Set $K_{p,q}(\nu ,N):=K_{p,q}(\nu ,N,e_{\left[N\right]})$.

Remark 2.

We have stipulated that $1/0:=\infty$ (thus $m_q^{-\epsilon }(x,R):=\infty$ whenever $m_q(x,R)=0$). One can write in (12), (13) the integrals over the support $S\left(p\right)$ instead of integrating over ${\mathbb{R}}^d$, whatever the versions of p and q are taken.

3. Main Results

Theorem 1.

Let, for some positive $\epsilon ,R$ and $N\in \mathbb{N}$, the functionals $K_{p,f}(1,N)$, $Q_{p,f}(\epsilon ,R)$, $T_{p,f}(\epsilon ,R)$ be finite if $f=p$ and $f=q$. Then $D\left({\mathsf{P}}_X\right|\left|{\mathsf{P}}_Y\right)<\infty$ and

$$\underset{n,m\to \infty }{lim}\mathsf{E}{\widehat{D}}_{n,m}(k,l)=D({\mathsf{P}}_X\left|\right|{\mathsf{P}}_Y).$$

(14)

Consider 3 kinds of conditions (labeled A,B,C, possibly with indices, and involving parameters indicated in parentheses) on probability densities.

$(\mathrm{A};p,f,\nu )$ For probability densities $p,f$ in ${\mathbb{R}}^d$ and some positive $\nu$

$$L_{p,f}\left(\nu \right):={\int }_{{\mathbb{R}}^d\times {\mathbb{R}}^d}{|log∥x-y∥|}^{\nu }p\left(x\right)f\left(y\right)dxdy<\infty .$$

(15)

As usual, ${\int }_{A}g\left(z\right)Q\left(dz\right)=0$ whenever $g\left(z\right)=\infty$ (or $-\infty$) for $z\in A$ and $Q\left(A\right)=0$, where Q is a $\sigma$-finite measure on $({\mathbb{R}}^d,\mathcal{B}\left({\mathbb{R}}^d\right))$. Condition (15) with $\nu >1$ is used, e.g., in [28,31,47].

$({\mathrm{B}}_1;f)$ A version of f is upper bounded by a positive number $M\left(f\right)\in (0,\infty )$:

$$f\left(x\right)\le M\left(f\right),x\in {\mathbb{R}}^d.$$

$({\mathrm{C}}_1;f$) A version of f is lower bounded by a positive number $m\left(f\right)\in (0,\infty )$:

$$f\left(x\right)\ge m\left(f\right),x\in S\left(f\right).$$

Corollary 1.

Let, for some $\nu >1$, condition $(\mathrm{A};p,f,\nu )$ be satisfied when $f=p$ and $f=q$. Then the statements of Theorem 1 are true, provided that $({\mathrm{B}}_1;f)$ and $({\mathrm{C}}_1;f)$ are both valid for $f=p$ and $f=q$. Moreover, if the latter assumption involving $({\mathrm{B}}_1;f)$ and $({\mathrm{C}}_1;f)$ holds then conditions of Theorem 1 are satisfied whenever p and q have bounded supports.

Next we formulate conditions to guarantee $L^2$-consistency of estimates (4).

Theorem 2.

Let the requirement $K_{p,f}(1,N)<\infty$ in conditions of Theorem 1 be replaced by $K_{p,f}(2,N)<\infty$, given $f=p$ and $f=q$. Then $D\left({\mathsf{P}}_X\right|\left|{\mathsf{P}}_Y\right)<\infty$ and, for any fixed $k,l\in \mathbb{N}$, the estimates ${\widehat{D}}_{n,m}(k,l)$ are $L^2$-consistent, i.e.,

$$\underset{n,m\to \infty }{lim}\mathsf{E}{\left({\widehat{D}}_{n,m}(k,l)-D({\mathsf{P}}_X\left|\right|{\mathsf{P}}_Y)\right)}^2=0.$$

(16)

Corollary 2.

For some $\nu >2$, let condition $(\mathrm{A};p,f,\nu )$ be satisfied if $f=p$ and $f=q$. Assume that $({\mathrm{B}}_1;f)$ and $({\mathrm{C}}_1;f)$ are both valid for $f=p$ and $f=q$. Then the statements of Theorem 2 are true. Moreover, if the latter assumption involving $({\mathrm{B}}_1;f)$ and $({\mathrm{C}}_1;f)$ holds then conditions of Theorem 2 are satisfied whenever p and q have bounded supports.

Currently we dwell on a modification of condition $({\mathrm{C}}_1;f)$ introduced in [35] that allows us to work with densities that need not have bounded supports.

$({\mathrm{C}}_2;f)$ There exist a version of density f and $R>0$ such that, for some $c>0$,

$$m_f(x,R)\ge cf\left(x\right),x\in {\mathbb{R}}^d.$$

Remark 3.

If, for some positive ε, R and c, condition $({\mathrm{C}}_2;q$) is true and

$${\int }_{{\mathbb{R}}^d}q{\left(x\right)}^{-\epsilon }p\left(x\right)dx<\infty ,$$

(17)

then $T_{p,q}(\epsilon ,R)$ is finite. Hence we could apply, for $f=p$ and $f=q$ in Theorems 1 and 2, condition $({\mathrm{C}}_2;f)$ and presume, for some $\epsilon >0$, validity of (17) and finiteness of ${\int }_{{\mathbb{R}}^d}{p}^{1-\epsilon }\left(x\right)dx$ instead of the corresponding assumptions $T_{p,q}(\epsilon ,R)<\infty$ and $T_{p,p}(\epsilon ,R)<\infty$. An illustrative example to this point is provided with a density having unbounded support.

Corollary 3.

Let X, Y be Gaussian random vectors in ${\mathbb{R}}^d$ with $\mathsf{E}X={\mu }_X$, $\mathsf{E}Y={\mu }_Y$ and nondegenerate covariance matrices ${\mathsf{\Sigma }}_X$ and ${\mathsf{\Sigma }}_Y$, respectively. Then relations (14) and (16) hold where

$$D({\mathsf{P}}_X\left|\right|{\mathsf{P}}_Y)=\frac{1}{2}\left(\mathsf{tr}\left({\mathsf{\Sigma }}_Y^{-1}{\mathsf{\Sigma }}_X\right)+{\left({\mu }_Y-{\mu }_X\right)}^T{\mathsf{\Sigma }}_Y^{-1}\left({\mu }_Y-{\mu }_X\right)-d+log\left(\frac{det{\mathsf{\Sigma }}_Y}{det{\mathsf{\Sigma }}_X}\right)\right).$$

The latter formula can be found, e.g., in [2], p. 147, example 6.3. The proof of Corollary 3 is discussed in Appendix A.

Similarly to condition $({\mathrm{C}}_2;f)$ let us consider the following one.

$({\mathrm{B}}_2;f)$ There exist a version of density f and $R>0$ such that, for some $C>0$,

$$M_f(x,R)\le Cf\left(x\right),x\in S\left(f\right).$$

Remark 4.

If, for some positive ε, R and c, condition $({\mathrm{B}}_2;q)$ is true and

$${\int }_{{\mathbb{R}}^d}q{\left(x\right)}^{\epsilon }p\left(x\right)dx<\infty ,$$

(18)

then obviously $Q_{p,q}(\epsilon ,R)<\infty$. Thus, in Theorems 1 and 2 one can employ, for $f=p$ and $f=q$, condition $({\mathrm{B}}_2;f)$ and exploit, for some $\epsilon >0$, the validity of (18) and finiteness of ${\int }_{{\mathbb{R}}^d}{p}^{1+\epsilon }\left(x\right)dx$ instead of the assumptions $Q_{p,q}(\epsilon ,R)<\infty$ and $Q_{p,p}(\epsilon ,R)<\infty$, respectively.

Remark 5.

D.Evans applied “positive density condition” in Definition 2.1 of [48] assuming the existence of constants $\beta >1$ and $\delta >0$ such that $\frac{r^d}{\beta }\le {\int }_{B(x,r)}q\left(y\right)dy\le \beta r^d$ for all $0\le r\le \delta$ and $x\in {\mathbb{R}}^d$. Consequently $m_q(x,\delta )\ge \frac{1}{\beta V_d}:=m>0$, $x\in {\mathbb{R}}^d$. Then $T_{p,q}(\epsilon ,\delta )\le m^{-\epsilon }{\int }_{{\mathbb{R}}^d}p\left(x\right)dx=m^{-\epsilon }<\infty$ for all $\epsilon >0$. Analogously, $M_q(x,\delta )\le \frac{\beta }{V_d}:=M$, $M>0$, $x\in {\mathbb{R}}^d$, and $Q_{p,q}(\epsilon ,\delta )\le M^{\epsilon }{\int }_{{\mathbb{R}}^d}p\left(x\right)dx=M^{\epsilon }<\infty$ for all $\epsilon >0$. The above mentioned inequalities from Definition 2.1 of [48] are valid, provided that density f is smooth and its support in $R^d$ is a convex closed body, see proof in [50]. Therefore, if p and q are smooth and their supports are compact convex bodies in ${\mathbb{R}}^d$, the relations (14) and (16) are valid.

Moreover, as a byproduct of Theorems 1 and 2, we obtain the new results indicating both the asymptotic unbiasedness and $L^2$-consistency of the estimates (7) for the Shannon differential entropy and cross-entropy.

Theorem 3.

Let $Q_{p,q}(\epsilon ,R)<\infty$ and $T_{p,q}(\epsilon ,R)<\infty$ for some positive ε and R. Then $C({\mathsf{P}}_X,{\mathsf{P}}_Y)$ is finite and the following statements hold for any fixed $l\in \mathbb{N}$.

(1) 

If, for some $N\in \mathbb{N}$, $K_{p,q}(1,N)<\infty$, then $\mathsf{E}{\widehat{C}}_{n,m}\left(l\right)\to C({\mathsf{P}}_X,{\mathsf{P}}_Y),n,m\to \infty .$

(2) 

If, for some $N\in \mathbb{N}$, $K_{p,q}(2,N)<\infty$, then $\mathsf{E}{({\widehat{C}}_{n,m}\left(l\right)-C({\mathsf{P}}_X,{\mathsf{P}}_Y))}^2\to 0,n,m\to \infty .$

In particular, one can employ $L_{p,q}\left(\nu \right)$ with $\nu >1$ instead of $K_{p,q}(1,N)$, and with $\nu >2$ instead of $K_{p,q}(2,N)$, where $N\in \mathbb{N}$.

The first claim of this Theorem follows from the proof of Theorem 1. In a similar way one can infer the second statement from the proof of Theorem 2. If we take $q=p$ in conditions of Theorem 3 then we get the statement concerning the entropy since $C({\mathsf{P}}_X,{\mathsf{P}}_X)=H\left({\mathsf{P}}_X\right)$.

Now we consider the case when p and q are mixtures of some probability densities. Namely,

$$p\left(x\right):=\sum _{i=1}^I{a}_i{p}_i\left(x\right),q\left(x\right):=\sum _{j=1}^J{b}_j{q}_j\left(x\right),$$

(19)

where $p_i\left(x\right)$, $q_j\left(x\right)$ are probability densities (w.r.t. the Lebesgue measure $\mu$), positive weights $a_i$, $b_j$ are such that ${\sum }_{i=1}^I{a}_i=1$, ${\sum }_{j=1}^J{b}_j=1$, $i=1,\dots ,I$, $j=1,\dots ,J$, $x\in {\mathbb{R}}^d$. Some applications of models described by mixtures are treated, e.g., in [51].

Corollary 4.

Let random vectors X and Y have densities of the form (19). Assume that, for some positive ε, R and $N\in \mathbb{N}$, the functionals $K_{f,g}(1,N)$, $Q_{f,g}(\epsilon ,R)$, $T_{f,g}(\epsilon ,R)$ are finite, whenever $f\in \{p_1,\dots ,p_I\}$ and $g\in \{p_1,\dots ,p_I,q_1,\dots ,q_J\}$. Then $D\left({\mathsf{P}}_X\right|\left|{\mathsf{P}}_Y\right)<\infty$ and, for any fixed $k,l\in \mathbb{N}$, (14) holds. Moreover, if the requirement $K_{f,g}(1,N)<\infty$ is replaced by $K_{f,g}(2,N)<\infty$ then (16) is true.

The proof of this Corollary is given in Appendix A. Thus, due to Corollaries 3 and 4 one can guarantee the validity of (14) and (16) for any mixtures of nondegenerate Gaussian densities. Note also that in a similar way we can claim the asymptotic unbaisedness and $L^2$-consistency of estimates (7) for mixtures satisfying conditions of Corollary 4.

Remark 6.

Let us compare our new results with those established in [35]. Developing the approach of [35] to analysis of asymptotic behavior of the Kozachenko–Leonenko estimates of the Shannon differential entropy we encounter new complications due to dealing with k-nearest neighbor statistics for $k\in \mathbb{N}$ (not only for $k=1$). Accordingly, in the framework of the Kullback–Leibler divergence estimation, we propose a new way to bound the function $1-F_{m,l,x}\left(u\right)$ playing the key role in the proofs (see Formula (28)). Furthermore, instead of the function $G\left(t\right)=tlogt$ (for $t>1$), used in [35] for the Shannon entropy estimates, we employ a regularly varying function $G_N\left(t\right)=t{log}_{\left[N\right]}\left(t\right)$ where (for t large enough) ${log}_{\left[N\right]}\left(t\right)$ is the N-fold iteration of the logarithmic function and $N\in \mathbb{N}$ can be large. Whence in the definition of integral functional $K_{p,q}(\nu ,N,t)$ by formula (11) one can take a function $G_N\left(z\right)$ having, for $z>0$, the growth rate close to that of function z. Moreover, this entails a generalization of paper [35] results. Now we invoke convexity of $G_N$ (see Lemma 6) to provide more general conditions for asymptotic unbiasedness and $L^2$-consistency of the Shannon differential entropy as opposed to [35].

4. Proof of Theorem 1

Note that the general structure of this proof, as well as that of Theorem 2, is similar to the one originally proposed in [28] and later used in various papers (see, e.g., [30,31,47]). Nevertheless in order to prove both theorems correctly we employ new ideas and conditions (such as uniform integrability of a family of random variables) in our reasoning.

Remark 7.

In the proof, for certain random variables $\alpha ,{\alpha }_1,{\alpha }_2,\dots$ (depending on some parameters), we will demonstrate that $\mathsf{E}{\alpha }_n\to \mathsf{E}\alpha$, as $n\to \infty$ (and that all these expectations are finite). To this end, for a fixed ${\mathbb{R}}^d$-valued random vector τ and each $x\in A$, where A is a specified subset of ${\mathbb{R}}^d$, we will prove that

$$\mathsf{E}\left({\alpha }_n\right|\tau =x)\to \mathsf{E}\left(\alpha \right|\tau =x),n\to \infty .$$

(20)

It turns out that $\mathsf{E}\left({\alpha }_n\right|\tau =x)=\mathsf{E}\left({\alpha }_{n,x}\right)$ and $\mathsf{E}\left(\alpha \right|\tau =x)=\mathsf{E}{\alpha }_x$, where the auxiliary random variables ${\alpha }_{n,x}$ and ${\alpha }_x$ can be constructed explicitly for each $x\in {\mathbb{R}}^d$. Moreover, it is possible to show that, for each $x\in A$, one has ${\alpha }_{n,x}\stackrel{law}{\to }{\alpha }_x$, $n\to \infty$. Thus, to prove (20) the Fatou lemma is not used, it is not evident whether there exists a random variable majorizing those under consideration. Instead we verify, for each $x\in A$, the uniform integrability (w.r.t. measure $\mathsf{P}$) of a family ${\left({\alpha }_{n,x}\right)}_{n\ge n_0\left(x\right)}$. Here we employ the necessary and sufficient conditions of uniform integrability provided by de la Vallée–Poussin theorem (see, e.g., Theorem 1.3.4 in [52]). After that, to prove the desired relation $\mathsf{E}{\alpha }_n\to \mathsf{E}\alpha$, $n\to \infty$, we have a new task. Namely, we check the uniform integrability of a family ${\left(\mathsf{E}\left({\alpha }_n\right|\tau =x)\right)}_{n\ge k_0}$, where $x\in A$, w.r.t. the measure ${\mathsf{P}}_{\tau }$, i.e., the law of τ, and $k_0$ does not depend of x. Then we can prove that

$$\mathsf{E}{\alpha }_n={\int }_A\mathsf{E}\left({\alpha }_n\right|\tau =x){\mathsf{P}}_{\tau }\left(dx\right)\to {\int }_A\mathsf{E}\left(\alpha \right|\tau =x){\mathsf{P}}_{\tau }\left(dx\right)=\mathsf{E}\alpha ,n\to \infty .$$

Further we will explain a number of nontrivial details concerning the proofs of uniform integrability of various families, the choice of the mentioned random variables (vectors), the set A, $n_0\left(x\right)$ and $k_0$.

The first auxiliary result explains why without loss of generality (w.l.g.) we can consider the same parameters $\epsilon ,R,N$ for different functionals in conditions of Theorems 1 and 2.

Lemma 1.

Let p and q be any probability densities in ${\mathbb{R}}^d$. Then the following statements are valid.

(1) 

If $K_{p,q}({\nu }_0,N_0)<\infty$ for some ${\nu }_0>0$ and $N_0\in \mathbb{N}$ then $K_{p,q}(\nu ,N)<\infty$ for any $\nu \in (0,{\nu }_0]$ and each $N\ge N_0$.

(2) 

If $Q_{p,q}({\epsilon }_1,R_1)<\infty$ for some ${\epsilon }_1>0$ and $R_1>0$ then $Q_{p,q}(\epsilon ,R)<\infty$ for any $\epsilon \in (0,{\epsilon }_1]$ and each $R>0$.

(3) 

If $T_{p,q}({\epsilon }_2,R_2)<\infty$ for some ${\epsilon }_2>0$ and $R_2>0$ then $T_{p,q}(\epsilon ,R)<\infty$ for any $\epsilon \in (0,{\epsilon }_2]$ and each $R>0$.

In particular one can take $q=p$ and the statements of Lemma 1 still remain valid. The proof of Lemma 1 is given in Appendix A.

Remark 8.

The results of Lemma 1 allow us to ensure (14) by demanding the finiteness of the functionals $K_{p,q}(1,N_1)$, $Q_{p,q}({\epsilon }_1,R_1)$, $T_{p,q}({\epsilon }_2,R_2)$, $K_{p,p}(1,N_2)$, $Q_{p,p}({\epsilon }_3,R_3)$, $T_{p,p}({\epsilon }_4,R_4)$, for some ${\epsilon }_i>0,R_j>0$ and $N_j\in \mathbb{N}$, where $i=1,2,3,4$ and $j=1,2$. Moreover, if we assume the finiteness of $K_{p,q}(2,N_3)$ and $K_{p,p}(2,N_4)$, for some $N_3\in \mathbb{N}$, $N_4\in \mathbb{N}$, instead of the finiteness of $K_{p,q}(2,N_1)$ and $K_{p,p}(2,N_2)$ then (16) holds.

According to Remark 2.4 of [35] if, for some positive $\epsilon ,R$, the integrals $Q_{p,q}(\epsilon ,R)$, $T_{p,q}(\epsilon ,R)$, $Q_{p,p}(\epsilon ,R)$, $T_{p,p}(\epsilon ,R)$ are finite then

$${\int }_{{\mathbb{R}}^d}p\left(x\right)|logq\left(x\right)|dx<\infty ,{\int }_{{\mathbb{R}}^d}p\left(x\right)|logp\left(x\right)|dx<\infty .$$

(21)

Therefore $D\left(p\right|\left|q\right)<\infty$ (and thus ${\mathsf{P}}_X\ll {\mathsf{P}}_Y$ in view of Lemma A1).

For $n\in \mathbb{N}$ such that $n>1$, for fixed $k\in \mathbb{N}$ and $m\in \mathbb{N}$, where $1\le k\le n-1$, $1\le l\le m$ and $i=1,\dots ,n$, set ${\varphi }_{m,l}\left(i\right):=m{V}_{m,l}^d\left(i\right)$, ${\zeta }_{n,k}\left(i\right):=(n-1)R_{n,k}^d\left(i\right)$. Then we can rewrite the estimate ${\widehat{D}}_{n,m}(k,l)$ as follows:

$${\widehat{D}}_{n,m}(k,l)=\psi \left(k\right)-\psi \left(l\right)+\frac{1}{n}\sum _{i=1}^n\left(log{\varphi }_{m,l}\left(i\right)-log{\zeta }_{n,k}\left(i\right)\right).$$

(22)

It is sufficient to prove the following two assertions.

Statement 1.

For each fixedl, allmlarge enough and any$i\in \mathbb{N}$, $\mathsf{E}|log{\varphi }_{m,l}\left(i\right)|$is finite.

Moreover,

$$\mathsf{E}\left(\frac{1}{n}\sum _{i=1}^{n}log{\varphi }_{m,l}\left(i\right)\right)=\mathsf{E}log{\varphi }_{m,l}\left(1\right)\to \psi \left(l\right)-log{V}_d-{\int }_{{\mathbb{R}}^d}p\left(x\right)logq\left(x\right)dx,m\to \infty .$$

(23)

Statement 2.

For each fixed k, all nlarge enough and any$i\in \mathbb{N}$, $\mathsf{E}|log{\zeta }_{n,k}\left(i\right)|$is finite.

Moreover,

$$\mathsf{E}\left(\frac{1}{n}\sum _{i=1}^{n}log{\zeta }_{n,k}\left(i\right)\right)=\mathsf{E}log{\zeta }_{n,k}\left(1\right)\to \psi \left(k\right)-log{V}_d-{\int }_{{\mathbb{R}}^d}p\left(x\right)logp\left(x\right)dx,n\to \infty .$$

(24)

Then in view of (2) and (21)–(24)

$$\begin{array}{c}\mathsf{E}{\widehat{D}}_{n,m}(k,l)\to -{\int }_{{\mathbb{R}}^d}p\left(x\right)logq\left(x\right)dx+{\int }_{{\mathbb{R}}^d}p\left(x\right)logp\left(x\right)dx=D({\mathsf{P}}_X\left|\right|{\mathsf{P}}_Y),n,m\to \infty ,\end{array}$$

and Theorem 1 will be proved.

Recall that, as explained in [35], for a nonnegative random variable V (thus $0\le EV\le \infty$) and any random ${\mathbb{R}}^d$-valued vector, one has

$$\mathsf{E}V={\int }_{{\mathbb{R}}^d}\mathsf{E}\left(V\right|X=x){\mathsf{P}}_X\left(dx\right).$$

(25)

This signifies that both sides of (25) coincide, being finite or infinite simultaneously. Let $F(u,\omega )$ be a regular conditional distribution function of a nonnegative random variable $\mathrm{U}$ given X where $u\in \mathbb{R}$ and $\omega \in \mathsf{\Omega }$. Let h be a measurable function such that $h:\mathbb{R}\to [0,\infty )$. It was also explained in [35] that, for ${\mathsf{P}}_X$-almost all $x\in {\mathbb{R}}^d$, it follows (without assuming $Eh\left(\mathrm{U}\right)<\infty$)

$$\mathsf{E}\left(h\left(\mathrm{U}\right)\right|X=x)={\int }_{[0,\infty )}h\left(u\right)dF(u,x).$$

(26)

This means that both sides of (26) are finite or infinite simultaneously and coincide.

By virtue of (25) and (26) one can establish that $\mathsf{E}|log{\varphi }_{m,l}\left(i\right)|<\infty$, for all m large enough, fixed l and for all i, and that (23) holds. To perform this take $\mathrm{U}={\varphi }_{m,l}\left(i\right)$, $X=X_i$, $h\left(u\right)=|logu|$, $u>0$ (we use $h\left(u\right)={log}^{2}u$ in the proof of Theorem 2) and $V=h\left(\mathrm{U}\right)$. If $h:\mathbb{R}\to \mathbb{R}$ and $\mathsf{E}\left|h\right(\mathrm{U}\left)\right|<\infty$ then (26) is true as well. To avoid increasing the volume of this paper we will only examine the evaluation of $\mathsf{E}log{\varphi }_{m,l}\left(i\right)$ as all the steps of the proof will be the same when treating $\mathsf{E}|log{\varphi }_{m,l}\left(i\right)|$.

The proof of Statement 1 is partitioned into 4 steps. The first three demonstrate that there is a measurable $A\subset S\left(p\right)$, depending on p and q versions, such that ${\mathsf{P}}_X(S\left(p\right)\backslash A)=0$ and, for any $x\in A$, $i\in \mathbb{N}$, the following relation holds:

$$\mathsf{E}(log{\varphi }_{m,l}\left(i\right)|X_i=x)=\mathsf{E}(log{\varphi }_{m,l}\left(1\right)|X_1=x)\to \psi \left(l\right)-log{V}_d-logq\left(x\right),m\to \infty .$$

(27)

The last Step 4 justifies the desired result (23). Finally Step 5 validates Statement 2.

Step 1. Here we establish the distribution convergence for the auxiliary random variables. Fix any $i\in \mathbb{N}$ and $l\in \{1,\dots ,m\}$. To simplify notation we do not indicate the dependence of functions on d. For $x\in {\mathbb{R}}^d$ and $u\ge 0$, we identify the asymptotic behavior (as $m\to \infty$) of the function

$$\begin{array}{c}\hfill \begin{array}{c}{F}_{m,l,x}^i\left(u\right):=\mathsf{P}\left({\varphi }_{m,l}\left(i\right)\le u|X_i=x\right)=\mathsf{P}\left(m{V}_{m,l}^d\left(i\right)\le u|X_i=x\right)\\ =1-\mathsf{P}\left(V_{m,l}\left(i\right)>{\left(\frac{u}{m}\right)}^{\frac{1}{d}}|X_i=x\right)=1-\mathsf{P}\left(∥x-Y_{\left(l\right)}(x,{\mathbb{Y}}_m)∥>{\left(\frac{u}{m}\right)}^{\frac{1}{d}}\right)\\ =1-{\displaystyle \sum _{s=0}^{l-1}}\left(\genfrac{}{}{0pt}{}{m}{s}\right){\left(W_{m,x}\left(u\right)\right)}^s{\left(1-W_{m,x}\left(u\right)\right)}^{m-s}:=\mathsf{P}\left({\xi }_{m,l,x}\le u\right),\end{array}\end{array}$$

(28)

where

$$W_{m,x}\left(u\right):={\int }_{B(x,r_m\left(u\right))}q\left(z\right)dz,r_m\left(u\right):={\left(\frac{u}{m}\right)}^{\frac{1}{d}},{\xi }_{m,l,x}:=m{∥x-Y_{\left(l\right)}(x,{\mathbb{Y}}_m)∥}^d.$$

(29)

We take into account in (28) that random vectors $Y_1,\dots ,Y_m,X_i$ are independent and condition that $Y_1,\dots ,Y_m$ have the same law as Y. We also noted that an event $\left\{∥x-Y_{\left(l\right)}(x,{\mathbb{Y}}_m)∥>r_m\left(u\right)\right\}$ is a union of pair-wise disjoint events $A_s$, $s=0,\dots ,l-1$. Here $A_s$ means that exactly s observations among ${\mathbb{Y}}_m$ belong to the ball $B(x,r_m\left(u\right))$ and other $m-s$ are outside this ball (probability that Y belongs to the sphere $\{z\in {\mathbb{R}}^d:\parallel z-x\parallel =r\}$ equals 0 since Y has a density w.r.t. the Lebesgue measure $\mu$). Formulas (28) and (29) show that $F_{m,l,x}^i\left(u\right)$ is the regular conditional distribution function of ${\varphi }_{m,l}\left(i\right)$ given $X_i=x$. Moreover, (28) means that ${\varphi }_{m,l}\left(i\right)$, $i\in \{1,\dots ,n\}$ are identically distributed and we may omit the dependence on i. So, one can write $F_{m,l,x}\left(u\right)$ instead of $F_{m,l,x}^i\left(u\right)$.

According to the Lebesgue differentiation theorem (see, e.g., [49], p. 654) if $q\in L^1\left({\mathbb{R}}^d\right)$, for $\mu$-almost all $x\in {\mathbb{R}}^d$, one has

$$\underset{r\to 0+}{lim}\frac{1}{\mu \left(B\right(x,r\left)\right)}{\int }_{B(x,r)}|q\left(z\right)-q\left(x\right)|dz=0.$$

(30)

Let $\mathsf{\Lambda }\left(q\right)$ denote the set of Lebesgue points of a function q, namely the points in ${\mathbb{R}}^d$ satisfying (30). Evidently it depends on the choice of version within the class of functions in $L^1\left({\mathbb{R}}^d\right)$ equivalent to q, and, for an arbitrary version of q, we have $\mu ({\mathbb{R}}^d\backslash \mathsf{\Lambda }\left(q\right))=0$.

Clearly, for each $u\ge 0$, $r_m\left(u\right)\to 0$ as $m\to \infty$, and $\mu \left(B(x,r_m\left(u\right))\right)=V_d{\left(r_m\left(u\right)\right)}^d=\frac{V_{d}u}{m}$. Therefore by virtue of (30), for any fixed $x\in \mathsf{\Lambda }\left(q\right)$ and $u\ge 0$,

$$W_{m,x}\left(u\right)=\frac{V_{d}u}{m}\left(q\left(x\right)+{\alpha }_m(x,u)\right),$$

where ${\alpha }_m(x,u)\to 0,m\to \infty$. Hence, for $x\in \mathsf{\Lambda }\left(q\right)\cap S\left(q\right)$ (thus $q\left(x\right)>0$), due to (28)

$$\begin{array}{c}\hfill \begin{array}{c}{F}_{m,l,x}\left(u\right)\to 1-{\displaystyle \sum _{s=0}^{l-1}}\frac{{\left(V_{d}q\left(x\right)u\right)}^s}{s!}{e}^{-V_{d}q\left(x\right)u}:=F_{l,x}\left(u\right),m\to \infty .\end{array}\end{array}$$

(31)

Relation (31) means that

$${\xi }_{m,l,x}\stackrel{law}{\to }{\xi }_{l,x},x\in \mathsf{\Lambda }\left(q\right)\cap S\left(q\right),m\to \infty ,$$

where ${\xi }_{l,x}$ has the Gamma distribution $\mathsf{\Gamma }(\alpha ,\lambda )$ with parameters $\alpha =V_{d}q\left(x\right)$ and $\lambda =l$.

For any $x\in S\left(q\right)$, one can assume w.l.g. that the random variables ${\xi }_{l,x}$ and ${\left\{{\xi }_{m,l,x}\right\}}_{m\ge l}$ are defined on a probability space $(\mathsf{\Omega },\mathcal{F},\mathsf{P})$. Indeed, by the Lomnicki–Ulam theorem (see, e.g., [53], p. 93) the independent copies of $Y_1,Y_2,\dots$ and ${\left\{{\xi }_{l,x}\right\}}_{x\in S\left(q\right)}$ exist on a certain probability space. The convergence in distribution of random variables survives under continuous mapping. Thus, for any $x\in \mathsf{\Lambda }\left(q\right)\cap S\left(q\right)$, we see that

$$log{\xi }_{m,l,x}\stackrel{law}{\to }log{\xi }_{l,x},m\to \infty .$$

(32)

We have employed that ${\xi }_{l,x}>0$ a.s. for each $x\in \mathsf{\Lambda }\left(q\right)\cap S\left(q\right)$ and Y has a density, so it follows that $\mathsf{P}({\xi }_{m,l,x}>0)=\mathsf{P}(∥x-Y_{\left(l\right)}(x,{\mathbb{Y}}_m)∥>0)=1$. More precisely, we take strictly positive versions of ${\xi }_{l,x}$ and ${\xi }_{m,l,x}$ for each $x\in \mathsf{\Lambda }\left(q\right)\cap S\left(q\right)$.

Step 2. Now we show that, instead of (27) validity, one can verify the following assertion. For μ-almost every $x\in \mathsf{\Lambda }\left(q\right)\cap S\left(q\right)$

$$\mathsf{E}log{\xi }_{m,l,x}\to \mathsf{E}log{\xi }_{l,x},m\to \infty .$$

(33)

Note that if $\eta \sim \mathsf{\Gamma }(\alpha ,\lambda )$, where $\alpha >0$ and $\lambda >0$, then $\mathsf{E}log\eta =\psi \left(\lambda \right)-log\alpha$, where $\psi$ is a digamma function. Set $\alpha =V_{d}q\left(x\right)$ for $x\in S\left(q\right)$ (then $\alpha >0$) and $\lambda =l$. Hence $\mathsf{E}log{\xi }_{l,x}=\psi \left(l\right)-log\left(V_{d}q\left(x\right)\right)=\psi \left(l\right)-log{V}_d-logq\left(x\right)$. By virtue of (26), for each $x\in {\mathbb{R}}^d$,

$$\begin{array}{c}\mathsf{E}log{\xi }_{m,l,x}={\int }_{(0,\infty )}logud{F}_{m,l,x}\left(u\right)={\int }_{(0,\infty )}logud\mathsf{P}({\varphi }_{m,l}\left(1\right)\le u|X_1=x)\\ =\mathsf{E}(log{\varphi }_{m,l}\left(1\right)|X_1=x).\end{array}$$

Hence, for $x\in \mathsf{\Lambda }\left(q\right)\cap S\left(q\right)$, the relation $\mathsf{E}(log{\varphi }_{m,l}\left(1\right)|X_1=x)\to \psi \left(l\right)-log{V}_d-logq\left(x\right)$ holds if and only if (33) is true.

According to Theorem 3.5 [54] we would have established (33) if relation (32) could be supplemented, for $\mu$-almost all $x\in \mathsf{\Lambda }\left(q\right)\cap S\left(q\right)$, by the condition of uniform integrability of a family ${\{log{\xi }_{m,l,x}\}}_{m\ge m_0\left(x\right)}$. Note that, for each $N\in \mathbb{N}$, a function $G_N\left(t\right)$ introduced by (10) is nondecreasing on $(0,\infty )$ and $\frac{G_N\left(t\right)}{t}\to \infty$, as $t\to \infty$. By the de la Vallée–Poussin theorem (see, e.g., Theorem 1.3.4 [52]), to ensure, for $\mu$-almost every $x\in \mathsf{\Lambda }\left(q\right)\cap S\left(q\right)$, the uniform integrability of ${\{log{\xi }_{m,l,x}\}}_{m\ge m_0\left(x\right)}$, it suffices to prove the following statement. For the indicated x, a positive $C_0\left(x\right)$ and $m_0\left(x\right)\in \mathbb{N}$, one has

$$\underset{m\ge m_0\left(x\right)}{sup}\mathsf{E}{G}_N\left(\right|log{\xi }_{m,l,x}\left|\right)\le C_0\left(x\right)<\infty ,$$

(34)

where $G_N$ appears in conditions of Theorem 1. Moreover, it is possible to find $m_0\in \mathbb{N}$ that does not depend on $x\in {\mathbb{R}}^d$ as we will show further.

Step 3. This step is devoted to proving validity of (34). It is convenient to divide this step into its own parts (3a), (3b), etc. For any $N\in \mathbb{N}$, set

$$g_N\left(t\right)=\left\{\begin{array}{cc}-\frac{1}{t}\left({log}_{\left[N\right]}(-logt)+\frac{1}{{\prod }_{j=1}^{N-1}{log}_{\left[j\right]}(-logt)}\right),\hfill & t\in \left(0,\frac{1}{e_{\left[N\right]}}\right],\hfill \\ 0,\hfill & t\in \left(\frac{1}{e_{\left[N\right]}},e_{\left[N\right]}\right],\hfill \\ \frac{1}{t}\left({log}_{\left[N\right]}(logt)+\frac{1}{{\prod }_{j=1}^{N-1}{log}_{\left[j\right]}(logt)}\right),\hfill & t\in \left(e_{\left[N\right]},\infty \right),\hfill \end{array}\right.$$

where the product over empty set (when $N=1$) is equal to 1.

The proof of the following result is placed at Appendix A.

Lemma 2.

Let $F\left(u\right),u\in \mathbb{R}$, be a distribution function such that $F\left(0\right)=0$. Then, for each $N\in \mathbb{N}$, one has

$$\left(1\right){\int }_{\left(0,\frac{1}{e_{\left[N\right]}}\right]}{G}_N\left(\right|logu\left|\right)dF\left(u\right)={\int }_{\left(0,\frac{1}{e_{\left[N\right]}}\right]}F\left(u\right)(-g_N\left(u\right))du,$$

$$\left(2\right){\int }_{\left(e_{\left[N\right]},\infty \right)}{G}_N\left(\right|logu\left|\right)dF\left(u\right)={\int }_{\left(e_{\left[N\right]},\infty \right)}(1-F\left(u\right))g_N\left(u\right)du.$$

Fix N appearing in conditions of Theorem 1. Observe that, for $u\in \left(\frac{1}{e_{\left[N\right]}},e_{\left[N\right]}\right]$, one has $G_N\left(\right|logu\left|\right)=0$. Therefore, according to Lemma 2, for $x\in \mathsf{\Lambda }\left(q\right)\cap S\left(q\right)$ and $m\ge l$, we get $\mathsf{E}{G}_N\left(\right|log{\xi }_{m,l,x}\left|\right):=I_1(m,x)+I_2(m,x)$ where

$$\begin{array}{c}{I}_1(m,x):={\int }_{\left(0,\frac{1}{e_{\left[N\right]}}\right]}{F}_{m,l,x}\left(u\right)(-g_N\left(u\right))du,I_2(m,x):={\int }_{(e_{\left[N\right]},\infty )}(1-F_{m,l,x}\left(u\right))g_N\left(u\right)du.\end{array}$$

For convenience sake we write $I_1(m,x)$ and $I_2(m,x)$ without indicating their dependence on $N,l$ and d (these parameters are fixed).

Part (3a). We provide bounds for$I_1(m,x)$. Take $R>0$ appearing in conditions of Theorem 1 and any $u\in \left(0,\frac{1}{e_{\left[N\right]}}\right]$. Introduce $m_1:=max\left\{⌈\frac{1}{e_{\left[N\right]}{R}^d}⌉,l\right\}$, where, for $a\in \mathbb{R}$, $\lceil a\rceil :=inf\{m\in \mathbb{Z}:m\ge a\}$. Then $r_m\left(u\right)={\left(\frac{u}{m}\right)}^{1/d}\le {\left(\frac{1}{e_{\left[N\right]}m}\right)}^{1/d}\le R$ if $m\ge m_1$. Note also that we can consider only $m\ge l$ everywhere below, because the size of sample ${\mathbb{Y}}_m$ is not less than the number of neighbors l (see, e.g., (28)). Thus, for $R>0$, $u\in \left(0,\frac{1}{e_{\left[N\right]}}\right]$, $x\in {\mathbb{R}}^d$ and $m\ge m_1$,

$$\begin{array}{c}\frac{W_{m,x}\left(u\right)}{\mu \left(B(x,r_m\left(u\right))\right)}=\frac{{\int }_{B(x,r_m\left(u\right))}q\left(y\right)dy}{r_m^d\left(u\right)V_d}\le \underset{r\in (0,R]}{sup}\frac{{\int }_{B(x,r)}q\left(y\right)dy}{r^d{V}_d}=M_q(x,R),\end{array}$$

and we arrive at the inequality

$$W_{m,x}\left(u\right)\le M_q(x,R)\mu \left(B(x,r_m\left(u\right))\right)=\frac{M_q(x,R)V_{d}u}{m}.$$

(35)

If $\gamma \in (0,1]$ and $t\in [0,1]$ then, for all $m\ge 1$, invoking the Bernoulli inequality, one has

$$1-{(1-t)}^m\le {\left(mt\right)}^{\gamma }.$$

(36)

Recall that we assume $Q_{p,q}(\epsilon ,R)<\infty$ for some $\epsilon >0$, $R>0$. By virtue of Lemma 1 one can take $\epsilon <1$. So, due to (36) and since $W_{m,x}\left(u\right)\in [0,1]$ for all $x\in {\mathbb{R}}^d$, $u>0$ and $m\ge l$, we get

$$1-{(1-W_{m,x}\left(u\right))}^m\le {\left(m{W}_{m,x}\left(u\right)\right)}^{\epsilon }.$$

(37)

Thus in view of (28), (35) and (37) we have established that, for all $x\in \mathsf{\Lambda }\left(q\right)\cap S\left(q\right)$, $u\in (0,\frac{1}{e_{\left[N\right]}}]$ and $m\ge m_1$,

$$\begin{array}{c}\hfill \begin{array}{c}{F}_{m,l,x}\left(u\right)=1-{\displaystyle \sum _{s=0}^{l-1}}\left(\genfrac{}{}{0pt}{}{m}{s}\right){\left(W_{m,x}\left(u\right)\right)}^s{\left(1-W_{m,x}\left(u\right)\right)}^{m-s}\\ \le 1-{(1-W_{m,x}\left(u\right))}^m\le {\left(m\frac{M_q(x,R)V_{d}u}{m}\right)}^{\epsilon }={\left(M_q(x,R)\right)}^{\epsilon }{V}_d^{\epsilon }{u}^{\epsilon }.\end{array}\end{array}$$

(38)

Therefore, for any $x\in \mathsf{\Lambda }\left(q\right)\cap S\left(q\right)$ and $m\ge m_1$, one can write

$$\begin{array}{c}\hfill \begin{array}{c}{I}_1(m,x)\le {\left(M_q(x,R)\right)}^{\epsilon }{V}_d^{\epsilon }{\int }_{\left(0,\frac{1}{e_{\left[N\right]}}\right]}{u}^{\epsilon }(-g_N\left(u\right))du\\ \le {\left(M_q(x,R)\right)}^{\epsilon }{V}_d^{\epsilon }{\int }_{\left(0,\frac{1}{e_{\left[N\right]}}\right]}\frac{{log}_{\left[N\right]}(-logu)+1}{u^{1-\epsilon }}du=U_1(\epsilon ,N,d){\left(M_q(x,R)\right)}^{\epsilon },\end{array}\end{array}$$

(39)

where $U_1(\epsilon ,N,d):=V_d^{\epsilon }{L}_N\left(\epsilon \right)$, $L_N\left(\epsilon \right):={\int }_{[e_{\left[N\right]},\infty )}({log}_{\left[N\right]}\left(t\right)+1)e^{-\epsilon t}dt<\infty$. We took into account that $(-g_N\left(u\right))\le \frac{1}{u}({log}_{\left[N\right]}(-logu)+1)$ whenever $u\in \left(0,\frac{1}{e_{\left[N\right]}}\right]$.

Part (3b).We give bounds for$I_2(m,x)$. Since $g_N\left(u\right)\le \frac{{log}_{[N+1]}\left(u\right)+1}{u}$ if $u\in (e_{\left[N\right]},\infty )$, we can write, for $m\ge max\{e_{\left[N\right]}^2,l\}$,

$$\begin{array}{c}\hfill \begin{array}{c}{I}_2(m,x)\le {\int }_{(e_{\left[N\right]},\sqrt{m}]}(1-F_{m,l,x}\left(u\right))\frac{{log}_{[N+1]}\left(u\right)+1}{u}du\\ +{\int }_{(\sqrt{m},m^2]}(1-F_{m,l,x}\left(u\right))\frac{{log}_{[N+1]}\left(u\right)+1}{u}du+{\int }_{\left(m^2,\infty \right)}(1-F_{m,l,x}\left(u\right))g_N\left(u\right)du\\ :=J_1(m,x)+J_2(m,x)+J_3(m,x).\end{array}\end{array}$$

Evidently,

$$\begin{array}{c}\hfill \begin{array}{c}1-F_{m,l,x}\left(u\right)={\displaystyle \sum _{r=m-l+1}^m}\left(\genfrac{}{}{0pt}{}{m}{r}\right){\left(P_{m,x}\left(u\right)\right)}^r{\left(1-P_{m,x}\left(u\right)\right)}^{m-r}=\mathsf{P}(Z\ge m-l+1),\end{array}\end{array}$$

where $P_{m,x}\left(u\right)=1-W_{m,x}\left(u\right)$ and $Z\sim \mathsf{Bin}(m,P_{m,x}\left(u\right))$.

By Markov’s inequality $\mathsf{P}(Z\ge t)\le e^{-\lambda t}\mathsf{E}{e}^{\lambda Z}$ for any $\lambda >0$ and $t>0$. One has

$$\begin{array}{c}\hfill \begin{array}{c}\mathsf{E}{e}^{\lambda Z}={\displaystyle \sum _{j=0}^m}{e}^{\lambda j}\left(\genfrac{}{}{0pt}{}{m}{j}\right){\left(P_{m,x}\left(u\right)\right)}^j{\left(1-P_{m,x}\left(u\right)\right)}^{m-j}\\ ={\displaystyle \sum _{j=0}^m}\left(\genfrac{}{}{0pt}{}{m}{j}\right){\left(P_{m,x}\left(u\right)e^{\lambda }\right)}^j{\left(1-P_{m,x}\left(u\right)\right)}^{m-j}={\left(1-P_{m,x}\left(u\right)+e^{\lambda }{P}_{m,x}\left(u\right)\right)}^m.\end{array}\end{array}$$

Consequently, for each $\lambda >0$,

$$\begin{array}{c}\hfill \begin{array}{c}1-F_{m,l,x}\left(u\right)\le e^{-\lambda (m-l+1)}{\left(1-P_{m,x}\left(u\right)+e^{\lambda }{P}_{m,x}\left(u\right)\right)}^m\\ =e^{-\lambda (m-l+1)}{\left(W_{m,x}\left(u\right)+e^{\lambda }(1-W_{m,x}\left(u\right))\right)}^m=e^{\lambda (l-1)}{\left(1-\left(1-\frac{1}{e^{\lambda }}\right)W_{m,x}\left(u\right)\right)}^m.\end{array}\end{array}$$

(40)

To simplify bounds we take $\lambda =1$ and set $S_1=S_1\left(l\right):=e^{l-1}$, $S_2:=1-\frac{1}{e}$ (recall that l is fixed). Thus, $S_1\ge 1$ and $S_2<1$. Therefore,

$$1-F_{m,l,x}\left(u\right)\le S_1{\left(1-S_2{W}_{m,x}\left(u\right)\right)}^m\le S_{1}exp\left\{-S_{2}m{W}_{m,x}\left(u\right)\right\},$$

(41)

where we have used simple inequality $1-t\le e^{-t}$, $t\in [0,1]$.

For $R>0$ appearing in conditions of the Theorem and any $u\in \left(e_{\left[N\right]},\sqrt{m}\right]$, one can choose $m_2:=max\left\{⌈\frac{1}{R^{2d}}⌉,⌈e_{\left[N\right]}^2⌉,l\right\}$ such that if $m\ge m_2$ then $r_m\left(u\right)={\left(\frac{u}{m}\right)}^{1/d}\le {\left(\frac{1}{\sqrt{m}}\right)}^{1/d}\le R.$ Due to (29) and (41), for $u\in (e_{\left[N\right]},\sqrt{m}]$ and $m\ge m_2$, one has

$$\begin{array}{c}\hfill \begin{array}{c}1-F_{m,l,x}\left(u\right)\le S_{1}exp\left\{-S_{2}m\frac{V_{d}u}{m}\frac{W_{m,x}\left(u\right)}{\frac{V_{d}u}{m}}\right\}\\ =S_{1}exp\left\{-S_2{V}_{d}u\frac{{\int }_{B(x,r_m\left(u\right))}q\left(z\right)dz}{\mu \left(B(x,r_m\left(u\right))\right)}\right\}\le S_{1}exp\left\{-S_2{V}_{d}u{m}_q(x,R)\right\},\end{array}\end{array}$$

(42)

by definition of $m_f$ (for $f=q$) in (9). Now we use the following Lemma 3.2 of [35].

Lemma 3.

For a version of a density q and each $R>0$, one has $\mu (S\left(q\right)\backslash D_q\left(R\right))=0$ where $D_q\left(R\right):=\{x\in S\left(q\right):m_q(x,R)>0\}$ and $m_q(·,R)$ is defined according to (9).

It is easily seen that, for any $t>0$ and each $\delta \in (0,e]$, one has $e^{-t}\le t^{-\delta }$. Thus, for $x\in D_q\left(R\right)$, $m\ge m_2$, $u\in (e_{\left[N\right]},\sqrt{m}]$ and $\epsilon >0$, we deduce from conditions of the Theorem (in view of Lemma 1 one can suppose that $\epsilon \in (0,e]$) that

$$1-F_{m,l,x}\left(u\right)\le S_1{\left(S_2{V}_{d}u{m}_q(x,R)\right)}^{-\epsilon }.$$

(43)

We also took into account that $m_q(x,R)>0$ for $x\in D_q\left(R\right)$ and applied relation (42). Thus, for all $x\in \mathsf{\Lambda }\left(q\right)\cap S\left(q\right)\cap D_q\left(R\right)$ and any $m\ge m_2$,

$$\begin{array}{c}\hfill \begin{array}{c}{J}_1(m,x)\le \frac{S_1}{{\left(S_2{V}_d\right)}^{\epsilon }{\left(m_q(x,R)\right)}^{\epsilon }}{\int }_{(e_{\left[N\right]},\infty )}\frac{{log}_{[N+1]}\left(u\right)+1}{u^{1+\epsilon }}du\\ =U_2(\epsilon ,N,d,l){\left(m_q(x,R)\right)}^{-\epsilon },\end{array}\end{array}$$

(44)

where $U_2(\epsilon ,N,d,l):=S_1\left(l\right)L_N\left(\epsilon \right){\left(S_2{V}_d\right)}^{-\epsilon }$.

Part (3c). We provide the bound for$J_2(m,x)$. For all $x\in \mathsf{\Lambda }\left(q\right)\cap S\left(q\right)\cap D_q\left(R\right)$ and any $m\ge m_2$, in view of (43), it holds $1-F_{m,l,x}\left(\sqrt{m}\right)\le S_1{\left(S_2{V}_d{m}_q(x,R)\sqrt{m}\right)}^{-\epsilon }$. Hence (as $m_2\ge 2$)

$$\begin{array}{c}\hfill \begin{array}{c}{J}_2(m,x)\le {\int }_{\left(\sqrt{m},m^2\right]}(1-F_{m,l,x}\left(u\right))\frac{{log}_{[N+1]}\left(u\right)+1}{u}du\\ \le \left(1-F_{m,l,x}\left(\sqrt{m}\right)\right){\int }_{\left(\sqrt{m},m^2\right]}\left({log}_{[N+1]}\left(u\right)+1\right)dlogu\\ \le S_1{\left(S_2{V}_d\right)}^{-\epsilon }{\left(m_q(x,R)\right)}^{-\epsilon }{m}^{-\frac{\epsilon }{2}}\left({log}_{\left[N\right]}(2logm)+1\right)\frac{3}{2}logm.\end{array}\end{array}$$

Then, for all $x\in \mathsf{\Lambda }\left(q\right)\cap S\left(q\right)\cap D_q\left(R\right)$ and any $m\ge m_2$,

$$J_2(m,x)\le U_3(m,\epsilon ,N,d,l){\left(m_q(x,R)\right)}^{-\epsilon },$$

(45)

where $U_3(m,\epsilon ,N,d,l):=\frac{3}{2}{S}_1\left(l\right){\left(S_2{V}_d\right)}^{-\epsilon }{m}^{-\frac{\epsilon }{2}}logm\left({log}_{\left[N\right]}(2logm)+1\right)\to 0$, $m\to \infty$.

Part (3d).To indicate bounds for$J_3(m,x)$we employ several auxiliary results.

Lemma 4.

For each $N\in \mathbb{N}$ and any $\nu >0$, there are $a:=a(d,\nu )\ge 0,b:=b(N,d,\nu )\ge 0$ such that, for arbitrary $x,y\in {\mathbb{R}}^d$,

$$G_N\left(|log{\parallel x-y\parallel }^d{|}^{\nu }\right)\le a{G}_N\left({|log\parallel x-y\parallel |}^{\nu }\right)+b.$$

The proof is given in Appendix A.

On the one hand, by (29), for any $w\ge 0$, we get

$$W_{m,x}\left(mw\right)={\int }_{B(x,w^{1/d})}q\left(z\right)dz=W_{1,x}\left(w\right).$$

On the other hand, by (28), one has $F_{1,1,x}\left(w\right)=1-\left(1-W_{1,x}\left(w\right)\right)=W_{1,x}\left(w\right)$. Consequently, for any $m\in \mathbb{N}$, $w\ge 0$ and all $x\in {\mathbb{R}}^d$,

$$W_{m,x}\left(mw\right)=F_{1,1,x}\left(w\right).$$

(46)

Moreover, $F_{1,1,x}\left(w\right)=\mathsf{P}({∥Y-x∥}^d\le w)$. So, ${\xi }_{1,1,x}\stackrel{law}{=}{∥Y-x∥}^d$. Thus, due to Lemmas 2 and 4 (for $\nu =1$)

$$\begin{array}{c}\hfill \begin{array}{c}{\int }_{\left(e_{\left[N\right]},\infty \right)}(1-F_{1,1,x}\left(w\right))g_N\left(w\right)dw={\int }_{\left(e_{\left[N\right]},\infty \right)}{G}_N(logw)d{F}_{1,1,x}\left(w\right)\\ =\mathsf{E}\left[G_N\left(log{\xi }_{1,1,x}\right)\mathbb{I}\left\{{\xi }_{1,1,x}>e_{\left[N\right]}\right\}\right]=\mathsf{E}[G_N{(log\parallel Y-x\parallel }^d{\left)\mathbb{I}\right\{\parallel Y-x\parallel }^d>e_{\left[N\right]}\left\}\right]\\ ={\int }_{y\in {\mathbb{R}}^d,∥x-y∥>{\left(e_{\left[N\right]}\right)}^{1/d}}{G}_N(log{∥x-y∥}^d)q\left(y\right)dy\\ \le a(d,1){\int }_{y\in {\mathbb{R}}^d,∥x-y∥>{\left(e_{\left[N\right]}\right)}^{1/d}}{G}_N\left(\right|log∥x-y∥\left|\right)q\left(y\right)dy+b(N,d,1)\\ =a(d,1){\int }_{y\in {\mathbb{R}}^d,∥x-y∥>e_{\left[N\right]}}{G}_N(log∥x-y∥)q\left(y\right)dy+b(N,d,1),\end{array}\end{array}$$

(47)

since $G_N\left(t\right)=0$ for $t\in [0,e_{[N-1]}]$, $N\in \mathbb{N}$.

Now we will estimate $1-F_{m,l,x}\left(u\right)$ in a way different from (40). Fix any $\delta >0$. Note that, for all $m\ge (l-1)\left(1+\frac{1}{\delta }\right)$ and $s\in \{0,\dots ,l-1\}$, it holds $\frac{m}{m-s}\le \frac{m}{m-l+1}\le 1+\delta$. Then, for all $x\in {\mathbb{R}}^d$, $u\ge 0$ and $m\ge max\{l,(l-1)\left(1+\frac{1}{\delta }\right)\}$, in view of (28) one can write

$$\begin{array}{c}\hfill \begin{array}{c}1-F_{m,l,x}\left(u\right)=\left(1-W_{m,x}\left(u\right)\right){\displaystyle \sum _{s=0}^{l-1}}\left(\genfrac{}{}{0pt}{}{m-1}{s}\right)\frac{m}{m-s}{\left(W_{m,x}\left(u\right)\right)}^s{\left(1-W_{m,x}\left(u\right)\right)}^{(m-1)-s}\\ \le (1+\delta )\left(1-W_{m,x}\left(u\right)\right){\displaystyle \sum _{s=0}^{l-1}}\left(\genfrac{}{}{0pt}{}{m-1}{s}\right){\left(W_{m,x}\left(u\right)\right)}^s{\left(1-W_{m,x}\left(u\right)\right)}^{(m-1)-s}\end{array}\end{array}$$

$$\le (1+\delta )\left(1-W_{m,x}\left(u\right)\right).$$

(48)

We are going to employ the following statement as well.

Lemma 5.

For each $N\in \mathbb{N}$, a function ${log}_{\left[N\right]}\left(t\right)$, $t>e_{[N-1]}$, is slowly varying at infinity.

The proof is elementary and thus is omitted.

Part (3e). Now we are ready to get the bound for$J_3(m,x)$. Set $u=mw$. Then one has

$$\begin{array}{c}\hfill \begin{array}{c}{J}_3(m,x)={\int }_{\left(m^2,\infty \right)}(1-F_{m,l,x}\left(u\right))\frac{1}{u}\left({log}_{\left[N\right]}(logu)+\frac{1}{{\prod }_{j=1}^{N-1}{log}_{\left[j\right]}(logu)}\right)du\\ ={\int }_{\left(m,\infty \right)}(1-F_{m,l,x}\left(mw\right))\frac{1}{w}\left({log}_{[N+1]}\left(mw\right)+\frac{1}{{\prod }_{j=2}^N{log}_{\left[j\right]}\left(mw\right)}\right)dw.\end{array}\end{array}$$

Given $w>m$, Lemma 5 implies that ${log}_{[N+1]}\left(mw\right)\le {log}_{[N+1]}\left(w^2\right)={log}_{\left[N\right]}(2logw)\le 2{log}_{[N+1]}\left(w\right)$ for w large enough, namely for all $w\ge W$, where $W=W\left(N\right)$. Take $\delta >0$ and set $m_3:=max\left\{l,⌈(l-1)\left(1+\frac{1}{\delta }\right)⌉,⌈W\left(N\right)⌉,⌈e_{\left[N\right]}⌉\right\}$. Let further $m\ge m_3$. Then

$$\begin{array}{c}\hfill \begin{array}{c}{J}_3(m,x)\le 2{\int }_{\left(m,\infty \right)}(1-F_{m,l,x}\left(mw\right))\frac{1}{w}\left({log}_{[N+1]}\left(w\right)+\frac{1}{{\prod }_{j=2}^N{log}_{\left[j\right]}\left(w\right)}\right)dw.\end{array}\end{array}$$

By virtue of (46) and (48) one has

$$1-F_{m,l,x}\left(mw\right)\le (1+\delta )\left(1-W_{m,x}\left(mw\right)\right)=(1+\delta )\left(1-F_{1,1,x}\left(w\right)\right).$$

(49)

Hence it can be seen that

$$\begin{array}{c}\hfill \begin{array}{c}{J}_3(m,x)\le 2(1+\delta ){\int }_{\left(m,\infty \right)}(1-F_{1,1,x}\left(w\right))g_{N_1}\left(w\right)dw.\end{array}\end{array}$$

(50)

Introduce

$$R_N\left(x\right):={\int }_{y\in {\mathbb{R}}^d,∥x-y∥>e_{\left[N\right]}}{G}_N(log∥x-y∥)q\left(y\right)dy,A_p\left(G_N\right):=\{x\in S\left(p\right):R_N\left(x\right)<\infty \}.$$

Let us note that (1) ${\mathsf{P}}_X(S\left(p\right)\backslash A_p\left(G_N\right))=0$ as $K_{p,q}(1,N)<\infty$;

(2) ${\mathsf{P}}_X(S\left(p\right)\backslash S\left(q\right))=0$ as ${\mathsf{P}}_X\ll {\mathsf{P}}_Y$ (see Lemma A1);

(3) $\mu \left(S\left(q\right)\backslash (\mathsf{\Lambda }\left(q\right)\cap D_q\left(R\right))\right)=0$ due to Lemma 3.

Since ${\mathsf{P}}_X\ll \mu$ we conclude that ${\mathsf{P}}_X\left(S\left(q\right)\backslash (\mathsf{\Lambda }\left(q\right)\cap D_q\left(R\right))\right)=0$. Hence, one has ${\mathsf{P}}_X\left(S\left(p\right)\backslash (\mathsf{\Lambda }\left(q\right)\cap D_q\left(R\right))\right)=0$ in view of 2) and because $B\backslash C\subset (B\backslash A)\cup (A\backslash C)$ for any $A,B,C\subset {\mathbb{R}}^d$. Set further $A:=\mathsf{\Lambda }\left(q\right)\cap S\left(q\right)\cap D_q\left(R\right)\cap S\left(p\right)\cap A_p\left(G_N\right)$. It follows from (1), (2) and (3) that ${\mathsf{P}}_X(S\left(p\right)\backslash A)=0$, so ${\mathsf{P}}_X\left(A\right)=1$. We are going to consider only $x\in A$.

Then, by virtue of (47) and (50), for all $m\ge m_3$ and $x\in A$, we come to the inequality

$$J_3(m,x)\le 2(1+\delta )\left(a(d,1)R_N\left(x\right)+b(N,d,1)\right)=A(\delta ,d)R_N\left(x\right)+B(\delta ,d,N),$$

(51)

where $A(\delta ,d):=2(1+\delta )a(d,1)$, $B(\delta ,d,N):=2(1+\delta )b(N,d,1)$.

Part (3f). Here we get the upper bound for$\mathsf{E}{G}_N\left(\right|log{\xi }_{m,l,x}\left|\right)$. For $m\ge max\{m_1,m_2,m_3\}$ and each $x\in A$, taking into account (39), (44), (45) and (51) we can claim that

$$\begin{array}{c}\hfill \begin{array}{c}\mathsf{E}{G}_N\left(\right|log{\xi }_{m,l,x}\left|\right)\le I_1(m,x)+J_1(m,x)+J_2(m,x)+J_3(m,x)\\ \le U_1(\epsilon ,N,d){\left(M_q(x,R)\right)}^{\epsilon }+U_2(\epsilon ,N,d,l){\left(m_q(x,R)\right)}^{-\epsilon }\\ +U_3(m,\epsilon ,N,d,l){\left(m_q(x,R)\right)}^{-\epsilon }+\left(A(\delta ,d)R_N\left(x\right)+B(\delta ,d,N)\right).\end{array}\end{array}$$

(52)

For any $\kappa >0$, one can take $m_4=m_4(\kappa ,\epsilon ,N,d,l)\in \mathbb{N}$ such that $U_3(m,\epsilon ,N,d,l)\le \kappa$ if $m\ge m_4$. Then by virtue of (52), for each $x\in A$ and $m\ge m_0:=max\{m_1,m_2,m_3,m_4\}$,

$$\begin{array}{c}\hfill \begin{array}{c}\mathsf{E}{G}_N\left(\right|log{\xi }_{m,l,x}\left|\right)\le U_1(\epsilon ,N,d){\left(M_q(x,R)\right)}^{\epsilon }\\ +\left(U_2(\epsilon ,N,d,l)+\kappa \right){\left(m_q(x,R)\right)}^{-\epsilon }+\left(A(\delta ,d)R_N\left(x\right)+B(\delta ,d,N)\right):=C_0\left(x\right)<\infty .\end{array}\end{array}$$

(53)

Hence, for each $x\in A$, we have established uniform integrability of the family ${\left\{log{\xi }_{m,l,x}\right\}}_{m\ge m_0}$.

Step 4. Now we verify (23). It was checked, for each $x\in A$ (thus, for ${\mathsf{P}}_X$-almost every x belonging to $S\left(p\right)$) that $\mathsf{E}(log{\varphi }_{m,l}\left(1\right)|X_1=x)\to \psi \left(l\right)-log{V}_d-logq\left(x\right)$, $m\to \infty$. Set $Z_{m,l}\left(x\right):=\mathsf{E}(log{\varphi }_{m,l}\left(1\right)|X_1=x)=\mathsf{E}log{\xi }_{m,l,x}$. Consider $x\in A$ and take any $m\ge m_0$. We use the following property of $G_N$ which is shown in Appendix A.

Lemma 6.

For each $N\in \mathbb{N}$, a function $G_N$ is convex on ${\mathbb{R}}_{+}$.

By the Jensen inequality a function $G_N$ is nondecreasing and convex.

$$G_N\left(\right|Z_{m,l}\left(x\right)\left|\right)=G_N\left(\right|\mathsf{E}log{\xi }_{m,l,x}\left|\right)\le G_N\left(\mathsf{E}\right|log{\xi }_{m,l,x}\left|\right)\le \mathsf{E}{G}_N\left(\right|log{\xi }_{m,l,x}\left|\right).$$

Relation (53) guarantees that, for all $m\ge m_0$,

$$\begin{array}{c}{\int }_{{\mathbb{R}}^d}{G}_N\left(\right|Z_{m,l}\left(x\right)\left|\right)p\left(x\right)dx\le U_1(\epsilon ,N,d)Q_{p,q}(\epsilon ,R)\\ +\left(U_2(\epsilon ,N,d,l)+\kappa \right)T_{p,q}(\epsilon ,R)+A(\delta ,d)K_{p,q}(1,N)+B(\delta ,d,N)<\infty .\end{array}$$

Now we know that the family ${\left\{Z_{m,l}\left(x\right)\right\}}_{m\ge m_0}$, $x\in A$, is uniformly integrable w.r.t. measure ${\mathsf{P}}_X$. Thus, for $i\in \mathbb{N}$,

$$\begin{array}{c}\mathsf{E}log{\varphi }_{m,l}\left(i\right)={\int }_{{\mathbb{R}}^d}\mathsf{E}(log{\varphi }_{m,l}\left(1\right)|X_1=x){\mathsf{P}}_{X_1}\left(dx\right)={\int }_{{\mathbb{R}}^d}{Z}_{m,l}\left(x\right)p\left(x\right)dx\\ \to \psi \left(l\right)-log{V}_d-{\int }_{{\mathbb{R}}^d}p\left(x\right)logq\left(x\right)dx,m\to \infty ,\end{array}$$

and we come to relation (23) establishing Statement 1.

Step 5. Here we prove Statement 2. Similar to $F_{m,l,x}\left(u\right)$, one can introduce, for $n,k\in \mathbb{N}$, $n\ge k+1$, $x\in {\mathbb{R}}^d$ and $u\ge 0$, the following function

$$\begin{array}{c}\hfill \begin{array}{c}{\tilde{F}}_{n,k,x}\left(u\right):=\mathsf{P}\left({\zeta }_{n,k}\left(i\right)\le u|X_i=x\right)=1-\mathsf{P}\left(∥x-X_{\left(k\right)}(x,{\mathbb{X}}_n\backslash \left\{x\right\})∥>r_{n-1}\left(u\right)\right)\\ =1-{\displaystyle \sum _{s=0}^{k-1}}\left(\genfrac{}{}{0pt}{}{n-1}{s}\right){\left(V_{n-1,x}\left(u\right)\right)}^s{\left(1-V_{n-1,x}\left(u\right)\right)}^{n-1-s}:=\mathsf{P}\left({\tilde{\xi }}_{n,k,x}\le u\right),\end{array}\end{array}$$

(54)

where $r_n\left(u\right)$ was defined in (29), and

$$V_{n,x}\left(u\right):={\int }_{B(x,r_n\left(u\right))}p\left(z\right)dz,{\tilde{\xi }}_{n,k,x}:=(n-1){∥x-X_{\left(k\right)}(x,{\mathbb{X}}_n\backslash \left\{x\right\})∥}^d.$$

(55)

Formulas (54) and (55) show that ${\tilde{F}}_{n,k,x}\left(u\right)$ is the regular conditional distribution function of ${\zeta }_{n,k}\left(i\right)$ given $X_i=x$. Moreover, for any fixed $u\ge 0$ and $x\in \mathsf{\Lambda }\left(p\right)\cap S\left(p\right)$ (thus $p\left(x\right)>0$),

$$\begin{array}{c}\hfill \begin{array}{c}{\tilde{F}}_{n,k,x}\left(u\right)\to 1-{\displaystyle \sum _{s=0}^{k-1}}\frac{{\left(V_{d}p\left(x\right)u\right)}^s}{s!}{e}^{-V_{d}p\left(x\right)u}:={\tilde{F}}_{k,x}\left(u\right),n\to \infty .\end{array}\end{array}$$

Hence, ${\tilde{\xi }}_{n,k,x}\stackrel{law}{\to }{\tilde{\xi }}_{k,x}$, $x\in \mathsf{\Lambda }\left(p\right)\cap S\left(p\right)$, $n\to \infty$. Set ${\tilde{A}}_p\left(G_N\right):=\{x\in S\left(p\right):{\tilde{R}}_N\left(x\right)<\infty \}$, where $N\in \mathbb{N}$ and

$${\tilde{R}}_N\left(x\right):={\int }_{y\in {\mathbb{R}}^d,∥x-y∥>e_{\left[N\right]}}{G}_N(log∥x-y∥)p\left(y\right)dy.$$

Take $\tilde{A}:=\mathsf{\Lambda }\left(p\right)\cap S\left(p\right)\cap D_p\left(R\right)\cap {\tilde{A}}_p\left(G_N\right)$. Then ${\mathsf{P}}_X\left(\tilde{A}\right)=1$ and, for $x\in \tilde{A}$, one can verify that $\mathsf{E}{G}_N\left(\right|log{\tilde{\xi }}_{n,k,x}\left|\right)\le {\tilde{C}}_0\left(x\right)<\infty$, for all $n\ge n_0$, and therefore $\mathsf{E}log{\tilde{\xi }}_{n,k,x}\to \mathsf{E}log{\tilde{\xi }}_{k,x}$ as $n\to \infty$. Thus, $\mathsf{E}(log{\zeta }_{n,k}\left(1\right)|X_1=x)\to \psi \left(k\right)-log{V}_d-logp\left(x\right)$, $n\to \infty$. Set ${\tilde{Z}}_{n,k}\left(x\right):=\mathsf{E}(log{\zeta }_{n,k}\left(1\right)|X_1=x)$. One can see that, for all $n\ge n_0$, ${\int }_{{\mathbb{R}}^d}{G}_N\left(\right|{\tilde{Z}}_{n,k}\left(x\right)\left|\right)p\left(x\right)dx<\infty$. Hence similar to Steps 1–4 we come to relation (24).

So, (14) holds and the proof of Theorem 1 is complete.

5. Proof of Theorem 2

We will follow the general scheme described in Remark 7. However now this scheme is more involved.

First of all note that, in view of Lemma 1, the finiteness of $K_{p,q}(2,N)$ and $K_{p,p}(2,N)$ implies the finiteness of $K_{p,q}(1,N)$ and $K_{p,p}(1,N)$, respectively. Thus, the conditions of Theorem 2 entail validity of Theorem 1 statements. Consequently under the conditions of Theorem 2, for n and m large enough, one can claim that ${\widehat{D}}_{n,m}(k,l)\in L^1\left(\mathsf{\Omega }\right)$ and $\mathsf{E}{\widehat{D}}_{n,m}(k,l)\to D({\mathsf{P}}_X\left|\right|{\mathsf{P}}_Y)$, as $n,m\to \infty$.

We will show that ${\widehat{D}}_{n,m}(k,l)\in L^2\left(\mathsf{\Omega }\right)$ for all n and m large enough. Then one can write

$$\mathsf{E}{\left({\widehat{D}}_{n,m}(k,l)-D({\mathsf{P}}_X\left|\right|{\mathsf{P}}_Y)\right)}^2=\mathsf{var}\left({\widehat{D}}_{n,m}(k,l)\right)+{\left(\mathsf{E}{\widehat{D}}_{n,m}(k,l)-D({\mathsf{P}}_X\left|\right|{\mathsf{P}}_Y)\right)}^2.$$

Therefore to prove (16) we will demonstrate that $\mathsf{var}\left({\widehat{D}}_{n,m}(k,l)\right)\to 0$, $n,m\to \infty$.

Due to (28) the random variables $log{\varphi }_{m,l}\left(1\right),\dots ,log{\varphi }_{m,l}\left(n\right)$ are identically distributed (and $log{\zeta }_{n,k}\left(1\right)$, $\dots ,log{\zeta }_{n,k}\left(n\right)$ are identically distributed as well). The variables ${\varphi }_{m,l}\left(i\right)$ and ${\zeta }_{n,k}\left(i\right)$ are the same as in (22). We will demonstrate that $log{\varphi }_{m,l}\left(1\right)$ and $log{\zeta }_{n,k}\left(1\right)$ belong to $L^2\left(\mathsf{\Omega }\right)$. Hence (22) yields

$$\begin{array}{c}\hfill \begin{array}{c}\mathsf{var}\left({\widehat{D}}_{n,m}(k,l)\right)=\frac{1}{n^2}{\displaystyle \sum _{i,j=1}^n}\mathsf{cov}\left(log{\varphi }_{m,l}\left(i\right)-log{\zeta }_{n,k}\left(i\right),log{\varphi }_{m,l}\left(j\right)-log{\zeta }_{n,k}\left(j\right)\right)\\ =\frac{1}{n}\mathsf{var}\left(log{\varphi }_{m,l}\left(1\right)\right)+\frac{2}{n^2}{\displaystyle \sum _{1\le i<j\le n}}\mathsf{cov}\left(log{\varphi }_{m,l}\left(i\right),log{\varphi }_{m,l}\left(j\right)\right)+\frac{1}{n}\mathsf{var}\left(log{\zeta }_{n,k}\left(1\right)\right)\\ +\frac{2}{n^2}{\displaystyle \sum _{1\le i<j\le n}}\mathsf{cov}\left(log{\zeta }_{n,k}\left(i\right),log{\zeta }_{n,k}\left(j\right)\right)-\frac{2}{n^2}{\displaystyle \sum _{i,j=1}^n}\mathsf{cov}\left(log{\varphi }_{m,l}\left(i\right),log{\zeta }_{n,k}\left(j\right)\right).\end{array}\end{array}$$

(56)

We mainly follow the notation employed in the above proof of Theorem 1, except the possibly different choice of the sets $A\subset {\mathbb{R}}^d$, $\tilde{A}\subset {\mathbb{R}}^d$, positive $U_j,C_j\left(x\right),{\tilde{C}}_j\left(x\right)$ and integers $m_j,n_j$, where $j\in {\mathbb{Z}}_{+}$ and $x\in {\mathbb{R}}^d$. The following Theorem 2 proof is also subdivided in 5 parts. Steps 1–3 deal with the demonstration of relation $\frac{1}{n}\mathsf{var}(log{\varphi }_{m,l}\left(1\right))\to 0$ as $n,m\to \infty$. Step 4 validates the relation $\frac{2}{n^2}{\sum }_{1\le i<j\le n}\mathsf{cov}(log{\varphi }_{m,l}\left(i\right),log{\varphi }_{m,l}\left(j\right))\to 0$ as $n,m\to \infty$. At Step 5 we establish that

$$\frac{2}{n^2}\sum _{1\le i<j\le n}\mathsf{cov}(log{\zeta }_{n,k}\left(i\right),log{\zeta }_{n,k}\left(j\right))\to 0,n\to \infty ,$$

This step is rather involved. Step 6 justifies the desired statement $\mathsf{var}\left({\widehat{D}}_{n,m}(k,l)\right)\to 0$, $n,m\to \infty$.

Step 1. We study$\mathsf{E}{log}^2\left({\varphi }_{m,l}\left(1\right)\right)$, as$m\to \infty$. For $x\in {\mathbb{R}}^d$ and $N\in \mathbb{N}$, introduce

$$R_{N,2}\left(x\right):={\int }_{∥x-y∥\ge e_{\left[N\right]}}{G}_N\left({log}^2∥x-y∥\right)q\left(y\right)dy.$$

(57)

Set $A_{p,2}\left(G_N\right):=\{x\in S\left(p\right):R_{N,2}\left(x\right)<\infty \}$. Then ${\mathsf{P}}_X(S\left(p\right)\backslash A_{p,2}\left(G_N\right))=0$ since $K_{p,q}(2,N)<\infty$. Consider

$$A:=\mathsf{\Lambda }\left(q\right)\cap S\left(q\right)\cap D_q\left(R\right)\cap S\left(p\right)\cap A_{p,2}\left(G_N\right),$$

(58)

where the first four sets appeared in Theorem 1 proof, R and N are indicated in conditions of Theorem 2. It is easily seen that ${\mathsf{P}}_X\left(A\right)=1$. The reasoning is exactly the same as in the proof of Theorem 1.

Recall that, for each $x\in A$, one has $log{\xi }_{m,l,x}\stackrel{law}{\to }log{\xi }_{l,x},m\to \infty$, where ${\xi }_{m,l,x}:=m{∥x-Y_{\left(l\right)}(x,{\mathbb{Y}}_m)∥}^d$ and ${\xi }_{l,x}$ has $\mathsf{\Gamma }(V_{d}q\left(x\right),l)$ distribution. Convergence in law of random variables is maintained by continuous transformations. Thus, for each $x\in A$, we get

$${log}^2{\xi }_{m,l,x}\stackrel{law}{\to }{log}^2{\xi }_{l,x},m\to \infty .$$

(59)

For any $x\in A$, according to (28),

$$\begin{array}{c}\hfill \begin{array}{c}\mathsf{E}{log}^2{\xi }_{m,l,x}={\int }_{(0,\infty )}{log}^{2}ud{F}_{m,l,x}\left(u\right)={\int }_{(0,\infty )}{log}^{2}ud\mathsf{P}({\varphi }_{m,l}\left(1\right)\le u|X_1=x)\\ =\mathsf{E}\left({log}^2{\varphi }_{m,l}\left(1\right)\right|X_1=x).\end{array}\end{array}$$

(60)

Note that if $\eta \sim \mathsf{\Gamma }(\alpha ,\lambda )$, where $\alpha >0$ and $\lambda >0$, then it is not difficult to verify that

$$\begin{array}{c}\hfill \begin{array}{c}\mathsf{E}{log}^2\eta =\frac{{\mathsf{\Gamma }}^{\prime \prime }\left(\lambda \right)}{\mathsf{\Gamma }\left(\lambda \right)}-2\psi \left(\lambda \right)log\alpha +{log}^2\alpha .\end{array}\end{array}$$

Since ${\xi }_{l,x}\sim \mathsf{\Gamma }(V_{d}q\left(x\right),l)$, for $x\in S\left(q\right)$, one has

$$\begin{array}{c}\hfill \begin{array}{c}\mathsf{E}{log}^2{\xi }_{l,x}=\frac{{\mathsf{\Gamma }}^{\prime \prime }\left(l\right)}{\mathsf{\Gamma }\left(l\right)}-2\psi \left(l\right)log\left(V_{d}q\left(x\right)\right)+{log}^2\left(V_{d}q\left(x\right)\right)={log}^{2}q\left(x\right)+h_{1}logq\left(x\right)+h_2,\end{array}\end{array}$$

(61)

where $h_1:=h_1(l,d)$ and $h_2:=h_2(l,d)$ depend only on fixed l and d.

We prove now that, for $x\in A$, one has

$$\mathsf{E}\left({log}^2{\varphi }_{m,l}\left(1\right)\right|X_1=x)\to {log}^{2}q\left(x\right)+h_{1}logq\left(x\right)+h_2,m\to \infty .$$

(62)

Taking into account (60) and (61) we can claim that relation (62) is equivalent to the following one: $\mathsf{E}{log}^2{\xi }_{m,l,x}\to \mathsf{E}{log}^2{\xi }_{l,x}$, $m\to \infty$. So, in view of (59) to prove (62) it is sufficient to show that, for each $x\in A$, a family ${\left\{{log}^2{\xi }_{m,l,x}\right\}}_{m\ge m_0\left(x\right)}$ is uniformly integrable for some $m_0\left(x\right)\in \mathbb{N}$. Then, following Theorem 1 proof, one can certify that, for all $x\in A$ and some nonnegative $C_0\left(x\right)$,

$$\underset{m\ge m_0\left(x\right)}{sup}\mathsf{E}{G}_N\left({log}^2{\xi }_{m,l,x}\right)\le C_0\left(x\right)<\infty .$$

(63)

Step 2. Now we will prove (63). For each $N\in \mathbb{N}$, introduce $\rho \left(N\right):=exp\left\{\sqrt{e_{[N-1]}}\right\}$ and

$$h_N\left(t\right):=\left\{\begin{array}{cc}0,\hfill & t\in \left(\frac{1}{\rho \left(N\right)},\rho \left(N\right)\right],\hfill \\ \frac{2logt}{t}\left({log}_{\left[N\right]}\left({log}^{2}t\right)+\frac{1}{{\prod }_{j=1}^{N-1}{log}_{\left[j\right]}\left({log}^{2}t\right)}\right),\hfill & t\in \left(0,\frac{1}{\rho \left(N\right)}\right]\cup \left(\rho \left(N\right),\infty \right).\hfill \end{array}\right.$$

As usual, a product over an empty set (if $N=1$) equals to 1. To show (63) we refer to the next lemma.

Lemma 7.

Let $F\left(u\right),u\in \mathbb{R}$, be a distribution function such that $F\left(0\right)=0$. Fix an arbitrary $N\in \mathbb{N}$. Then

$$\left(1\right){\int }_{\left(0,\frac{1}{\rho \left(N\right)}\right]}{G}_N\left({log}^{2}u\right)dF\left(u\right)={\int }_{\left(0,\frac{1}{\rho \left(N\right)}\right]}F\left(u\right)(-h_N\left(u\right))du,$$

$$\left(2\right){\int }_{\left(\rho \left(N\right),\infty \right)}{G}_N\left({log}^{2}u\right)dF\left(u\right)={\int }_{\left(\rho \left(N\right),\infty \right)}(1-F\left(u\right))h_N\left(u\right)du.$$

The proof of this lemma is omitted, being quite similar to one of Lemma 2. By Lemma 7 and since $G_N\left({log}^{2}u\right)=0$, for $u\in \left(\frac{1}{\rho \left(N\right)},\rho \left(N\right)\right]$, one has

$$\begin{array}{c}\mathsf{E}{G}_N\left({log}^2{\xi }_{m,l,x}\right)={\int }_{\left(0,\frac{1}{\rho \left(N\right)}\right]}{F}_{m,l,x}\left(u\right)(-h_N\left(u\right))du+{\int }_{\left(\rho \left(N\right),\infty \right)}(1-F_{m,l,x}\left(u\right))h_N\left(u\right)du\\ :=I_1(m,x)+I_2(m,x).\end{array}$$

To simplify notation we do not indicate the dependence of $I_i(m,x)$ ($i=1,2$) on fixed N, l and d.

For clarity, further implementation of Step 2 is divided into several parts.

Part (2a).At first we consider$I_1(m,x)$. As in Theorem 1 proof, for fixed $R>0$ and $\epsilon >0$ appearing in the conditions of Theorem 2, an inequality $F_{m,l,x}\left(u\right)\le {\left(M_q(x,R)\right)}^{\epsilon }{V}_d^{\epsilon }{u}^{\epsilon }$ holds for any $x\in A$, $u\in \left(0,\frac{1}{\rho \left(N\right)}\right]$ and $m\ge m_1:=max\left\{⌈\frac{1}{\rho \left(N\right)R^d}⌉,l\right\}$. Taking into account that $0\le (-h_N\left(u\right))\le \frac{(-2logu)\left({log}_{\left[N\right]}\left({log}^{2}u\right)+1\right)}{u}$ if $u\in \left(0,\frac{1}{\rho \left(N\right)}\right]$, we get, for $m\ge m_1$,

$$\begin{array}{c}\hfill \begin{array}{c}{I}_1(m,x)\le {\left(M_q(x,R)\right)}^{\epsilon }{V}_d^{\epsilon }{\int }_{\left(0,\frac{1}{\rho \left(N\right)}\right]}\frac{(-2logu)\left({log}_{\left[N\right]}\left({log}^{2}u\right)+1\right)}{u^{1-\epsilon }}du\\ =U_1(\epsilon ,N,d){\left(M_q(x,R)\right)}^{\epsilon }.\end{array}\end{array}$$

(64)

Here $U_1(\epsilon ,N,d):=V_d^{\epsilon }{L}_{N,2}\left(\epsilon \right)$, $L_{N,2}\left(\epsilon \right):={\int }_{\left[\sqrt{e_{[N-1]}},\infty \right)}2t\left({log}_{\left[N\right]}\left(t^2\right)+1\right)e^{-\epsilon t}dt<\infty$ for each $\epsilon >0$ and any $N\in \mathbb{N}$.

Part (2b).Consider$I_2(m,x)$. Following the previous theorem proof we at first observe that $h_N\left(u\right)\le \frac{2logu}{u}\left({log}_{\left[N\right]}\left({log}^{2}u\right)+1\right)$ for $u\in \left(\rho \right(N),\infty )$. So, for all $m\ge max\{{\rho }^2\left(N\right),l\}$,

$$\begin{array}{c}\hfill \begin{array}{c}{I}_2(m,x)\le {\int }_{(\rho \left(N\right),\sqrt{m}]}(1-F_{m,l,x}\left(u\right))\frac{2logu\left({log}_{\left[N\right]}\left({log}^{2}u\right)+1\right)}{u}du\\ +{\int }_{(\sqrt{m},m^2]}(1-F_{m,l,x}\left(u\right))\frac{2logu\left({log}_{\left[N\right]}\left({log}^{2}u\right)+1\right)}{u}du\\ +{\int }_{(m^2,\infty )}(1-F_{m,l,x}\left(u\right))h_N\left(u\right)du:=J_1(m,x)+J_2(m,x)+J_3(m,x),\end{array}\end{array}$$

where we do not indicate the dependence of $J_j(m,x)$ ($j=1,2,3$) on N, l and d.

For $R>0$ and $\epsilon >0$ appearing in the conditions of Theorem 2, one can show (see Theorem 1 proof), that inequality

$$1-F_{m,l,x}\left(u\right)\le S_1{\left(S_2{V}_{d}u{m}_q(x,R)\right)}^{-\epsilon }$$

(65)

holds for any $x\in A$, $u\in \left(\rho \left(N\right),\sqrt{m}\right]$ and all $m\ge m_2:=max\left\{⌈\frac{1}{R^{2d}}⌉,⌈{\rho }^2\left(N\right)⌉,l\right\}$. Here $S_1:=S_1\left(l\right)$ and $S_2$ are the same as in the proof of Theorem 1. For all $x\in A$ and $m\ge m_2$, we come to the relations

$$\begin{array}{c}\hfill \begin{array}{c}{J}_1(m,x)\le \frac{S_1}{{\left(S_2{V}_d\right)}^{\epsilon }{\left(m_q(x,R)\right)}^{\epsilon }}{\int }_{\left(\rho \right(N),\infty )}\frac{2logu\left({log}_{\left[N\right]}\left({log}^{2}u\right)+1\right)}{u^{1+\epsilon }}du\\ =U_2(\epsilon ,N,d,l){\left(m_q(x,R)\right)}^{-\epsilon },\end{array}\end{array}$$

(66)

where $U_2(\epsilon ,N,d,l):=2S_1\left(l\right)L_{N,2}\left(\epsilon \right){\left(S_2{V}_d\right)}^{-\epsilon }$.

Part (2c). Let us consider$J_2(m,x)$. Take $\delta >0$. Then, due to (65), for all $x\in A$ and any $m\ge m_2$,

$$\begin{array}{c}\hfill \begin{array}{c}{J}_2(m,x)\le 2\left(1-F_{m,l,x}\left(\sqrt{m}\right)\right){\int }_{\left(\sqrt{m},m^2\right]}logu\left({log}_{\left[N\right]}\left({log}^{2}u\right)+1\right)dlogu\\ \le 4S_1{\left(S_2{V}_d\right)}^{-\epsilon }{m}^{-\frac{\epsilon }{2}}{\left(m_q(x,R)\right)}^{-\epsilon }\left({log}_{\left[N\right]}\left(4{log}^{2}m\right)+1\right){log}^{2}m\\ =U_3(m,\epsilon ,N,d,l){\left(m_q(x,R)\right)}^{-\epsilon },\end{array}\end{array}$$

(67)

where $U_3(m,\epsilon ,N,d,l):=4S_1{\left(S_2{V}_d\right)}^{-\epsilon }{m}^{-\frac{\epsilon }{2}}\left({log}^{2}m\right)\left({log}_{\left[N\right]}\left(4{log}^{2}m\right)+1\right)\to 0$, $m\to \infty$.

Part (2d). Now we turn to$J_3(m,x)$. Take $u=mw$. Then $J_3(m,x)$ has the form

$$\begin{array}{c}\hfill \begin{array}{c}{\int }_{\left(m,\infty \right)}(1-F_{m,l,x}\left(mw\right))\frac{2log\left(mw\right)}{w}\left({log}_{\left[N\right]}\left({log}^2\left(mw\right)\right)+\frac{1}{{\prod }_{j=1}^{N-1}{log}_{\left[j\right]}\left({log}^2\left(mw\right)\right)}\right)dw.\end{array}\end{array}$$

Due to Lemma 5 there exists $T\left(N\right)>\rho \left(N\right)$ such that

$${log}_{\left[N\right]}\left({log}^2\left(w^2\right)\right)={log}_{\left[N\right]}\left(4{log}^{2}w\right)\le 2{log}_{\left[N\right]}\left({log}^{2}w\right),w\ge T\left(N\right).$$

(68)

Pick some $\delta >0$ and set $m_3:=max\left\{l,⌈(l-1)\left(1+\frac{1}{\delta }\right)⌉,⌈T\left(N\right)⌉,⌈\rho \left(N\right)⌉\right\}$, where $T\left(N\right)$ was introduced in (68). Consider $m\ge m_3$. In view of Lemma 4 (for $\nu =2$), (49), (68) and since $w>m$,

$$\begin{array}{c}\hfill \begin{array}{c}{J}_3(m,x)\le {\int }_{\left(m,\infty \right)}(1-F_{m,l,x}\left(mw\right))\frac{2log\left(w^2\right)}{w}\left({log}_{\left[N\right]}\left({log}^2\left(w^2\right)\right)+\frac{1}{{\prod }_{j=1}^{N-1}{log}_{\left[j\right]}\left({log}^{2}w\right)}\right)dw\\ \le 4(1+\delta ){\int }_{\left(m,\infty \right)}(1-F_{1,1,x}\left(w\right))\frac{2logw}{w}\left({log}_{\left[N\right]}\left({log}^{2}w\right)+\frac{1}{{\prod }_{j=1}^{N-1}{log}_{\left[j\right]}\left({log}^{2}w\right)}\right)dw\\ =4(1+\delta ){\int }_{\left(m,\infty \right)}(1-F_{1,1,x}\left(w\right))h_N\left(w\right)dw\le 4(1+\delta ){\int }_{\left(\rho \left(N\right),\infty \right)}(1-F_{1,1,x}\left(w\right))h_N\left(w\right)dw\\ =4(1+\delta ){\int }_{\left(\rho \left(N\right),\infty \right)}{G}_N\left({log}^{2}w\right)d{F}_{1,1,x}\left(w\right)=4(1+\delta )\mathsf{E}\left[G_N\left({log}^2{\xi }_{1,1,x}\right)\mathbb{I}\{{\xi }_{1,1,x}>\rho \left(N\right)\}\right]\\ =4(1+\delta )\mathsf{E}[G_N\left({(log{∥Y-x∥}^d)}^2\right){\mathbb{I}\{\parallel Y-x\parallel }^d>\rho \left(N\right)\left\}\right]\\ =4(1+\delta ){\int }_{y\in {\mathbb{R}}^d,\parallel x-y\parallel >{\left(\rho \left(N\right)\right)}^{1/d}}{G}_N{\left(\right(log\parallel x-y\parallel }^d{)}^2)q\left(y\right)dy\\ \le 4(1+\delta )\left(a(d,2){\int }_{y\in {\mathbb{R}}^d,\parallel x-y\parallel >{\left(\rho \left(N\right)\right)}^{1/d}}{G}_N({log}^2\parallel x-y\parallel )q\left(y\right)dy+b(N,d,2)\right)\end{array}\end{array}$$

$$\begin{array}{c}\hfill \begin{array}{c}\le 4(1+\delta )\left(a(d,2)\left(R_{N,2}\left(x\right)+G_N\left(e_{[N-1]}^2\right)\right)+b(N,d,2)\right)\\ =A(\delta ,d)R_{N,2}\left(x\right)+B(\delta ,d,N),\end{array}\end{array}$$

(69)

$A(\delta ,d):=4(1+\delta )a(d,2)$, $B(\delta ,d,N):=4(1+\delta )\left(a(d,2)G_N\left(e_{[N-1]}^2\right)+b(N,d,2)\right)$, $R_{N,2}\left(x\right)$ is defined in (57). Here we have also used, for any $N\in \mathbb{N}$, $\nu ,t,u>0$, $t<u$, the following estimates

$$K_{p,q}(\nu ,N,u)\le K_{p,q}(\nu ,N,t)\le K_{p,q}(\nu ,N,u)+max\{G_N{\left(\right|logt|}^{\nu }),G_N{\left(\right|logu|}^{\nu }\left)\right\}.$$

Part (2e). Examine$\mathsf{E}{G}_N\left({log}^2{\xi }_{m,l,x}\right)$. Thus, for each $x\in A$ and $m\ge max\{m_1,m_2,m_3\}$, taking into account (64), (66), (67) and (69), we can claim that

$$\begin{array}{c}\hfill \begin{array}{c}\mathsf{E}{G}_N\left({log}^2{\xi }_{m,l,x}\right)\le I_1(m,x)+J_1(m,x)+J_2(m,x)+J_3(m,x)\\ \le U_1(\epsilon ,N,d){\left(M_q(x,R)\right)}^{\epsilon }+U_2(\epsilon ,N,d,l){\left(m_q(x,R)\right)}^{-\epsilon }+U_3(m,\epsilon ,N,d,l){\left(m_q(x,R)\right)}^{-\epsilon }\end{array}\end{array}$$

$$+A(\delta ,d)R_{N,2}\left(x\right)+B(\delta ,d,N).$$

(70)

Moreover, for any $\kappa >0$, one can choose $m_4:=m_4(\kappa ,\epsilon ,N,d,l)\in \mathbb{N}$ such that, for $m\ge m_4$, it holds $U_3(m,\epsilon ,N,d,l)\le \kappa$. Then by (70), for each $x\in A$ and $m\ge m_0:=max\{m_1,m_2,m_3,m_4\}$,

$$\begin{array}{c}\hfill \begin{array}{c}\mathsf{E}{G}_N\left({log}^2{\xi }_{m,l,x}\right)\le U_1(\epsilon ,N,d){\left(M_q(x,R)\right)}^{\epsilon }\\ +\left(U_2(\epsilon ,N,d,l)+\kappa \right){\left(m_q(x,R)\right)}^{-\epsilon }+A(\delta ,d)R_{N,2}\left(x\right)+B(\delta ,d,N):=C_0\left(x\right)<\infty .\end{array}\end{array}$$

(71)

Hence we have proved the uniform integrability of the family ${\left\{{log}^2{\xi }_{m,l,x}\right\}}_{m\ge m_0}$ for each $x\in A$. Therefore, for any $x\in A$ (thus for ${\mathsf{P}}_X$-almost every $x\in S\left(p\right)$), relation (62) holds.

Step 3. Now we can return to$\mathsf{E}{log}^2{\varphi }_{m,l}\left(1\right)$. Set ${\Delta }_{m,l}\left(x\right):=\mathsf{E}\left({log}^2{\varphi }_{m,l}\left(1\right)\right|X_1=x)=\mathsf{E}{log}^2{\xi }_{m,l,x}$. Consider $x\in A$ and take any $m\ge m_0$. A function $G_N$ is nondecreasing and convex according to Lemma 6. Due to the Jensen inequality

$$\begin{array}{c}{G}_N\left({\Delta }_{m,l}\left(x\right)\right)=G_N\left(\mathsf{E}{log}^2{\xi }_{m,l,x}\right)\le \mathsf{E}{G}_N\left({log}^2{\xi }_{m,l,x}\right).\end{array}$$

(72)

Relation (72) guarantees that, for each $x\in A$ and all $m\ge m_0$,

$$\begin{array}{c}{\int }_{{\mathbb{R}}^d}{G}_N\left({\Delta }_{m,l}\left(x\right)\right)p\left(x\right)dx\le U_1(\epsilon ,N,d)Q_{p,q}(\epsilon ,R)+\left(U_2(\epsilon ,N,d,l)+\kappa \right)T_{p,q}(\epsilon ,R)\\ +A(\delta ,d)K_{p,q}(2,N)+B(\delta ,d,N)<\infty .\end{array}$$

Uniform integrability of the family ${\left\{{\Delta }_{m,l}(·)\right\}}_{m\ge m_0}$ (w.r.t measure ${\mathsf{P}}_X$) is thus established. Hence one can claim that

$$\begin{array}{c}\mathsf{E}{log}^2{\varphi }_{m,l}\left(1\right)\to {\int }_{{\mathbb{R}}^d}p\left(x\right){log}^{2}q\left(x\right)dx+h_1{\int }_{{\mathbb{R}}^d}p\left(x\right)logq\left(x\right)dx+h_2,m\to \infty .\end{array}$$

It is easily seen that finiteness of integrals $Q_{p,q}(\epsilon ,R)$, $T_{p,q}(\epsilon ,R)$ implies that

$${\int }_{{\mathbb{R}}^d}p\left(x\right){log}^{2}q\left(x\right)dx<\infty ,{\int }_{{\mathbb{R}}^d}p\left(x\right)|logq\left(x\right)|dx<\infty .$$

Thus, $\mathsf{E}{log}^2{\varphi }_{m,l}\left(1\right)\to {\tau }_2<\infty$ and $\mathsf{var}\left(log{\varphi }_{m,l}\left(1\right)\right)=\mathsf{E}{log}^2{\varphi }_{m,l}\left(1\right)-{\left(\mathsf{E}log{\varphi }_{m,l}\left(1\right)\right)}^2\to {\tau }_2-{\tau }_1^2<\infty$, $m\to \infty$, where ${\tau }_1:=\psi \left(l\right)-log{V}_d-{\int }_{{\mathbb{R}}^d}p\left(x\right)logq\left(x\right)dx$ according to (23). Consequently, $\frac{1}{n}\mathsf{var}\left(log{\varphi }_{m,l}\left(1\right)\right)\to 0$ as $n,m\to \infty$.

Step 4. Now we consider$\mathsf{cov}(log{\varphi }_{m,l}\left(i\right),log{\varphi }_{m,l}\left(j\right))$for$i\ne j$, where$i,j\in \{1,\dots ,n\}$. For $x,y\in {\mathbb{R}}^d$, define conditional distribution function

$${\mathsf{\Phi }}_{m,l,x,y}^{i,j}(u,w):=\mathsf{P}({\varphi }_{m,l}\left(i\right)\le u,{\varphi }_{m,l}\left(j\right)\le w|X_i=x,X_j=y),u,w\ge 0.$$

For $x,y\in {\mathbb{R}}^d$, $u,w\ge 0$, $i\ne j$,

$$\begin{array}{c}\hfill \begin{array}{c}{\mathsf{\Phi }}_{m,l,x,y}^{i,j}(u,w)=1-\mathsf{P}({\varphi }_{m,l}\left(i\right)>u|X_i=x,X_j=y)\\ -\mathsf{P}({\varphi }_{m,l}\left(j\right)>w|X_i=x,X_j=y)+\mathsf{P}({\varphi }_{m,l}\left(i\right)>u,{\varphi }_{m,l}\left(j\right)>w|X_i=x,X_j=y)\\ =1-\mathsf{P}\left(∥x-Y_{\left(l\right)}(x,{\mathbb{Y}}_m)∥>r_m\left(u\right)\right)-\mathsf{P}\left(∥y-Y_{\left(l\right)}(y,{\mathbb{Y}}_m)∥>r_m\left(w\right)\right)\\ +\mathsf{P}\left(∥x-Y_{\left(l\right)}(x,{\mathbb{Y}}_m)∥>r_m\left(u\right),∥y-Y_{\left(l\right)}(y,{\mathbb{Y}}_m)∥>r_m\left(w\right)\right).\end{array}\end{array}$$

(73)

Here $r_m\left(a\right)={\left(\frac{a}{m}\right)}^{\frac{1}{d}}$ for all $a\ge 0$, as previously. One can write ${\mathsf{\Phi }}_{m,l,x,y}(u,w)$ instead of ${\mathsf{\Phi }}_{m,l,x,y}^{i,j}(u,w)$, because the right-hand side of (73) does not depend on i and j.

Set $A_1:=\left\{(x,y):x\in A,y\in A,x\ne y\right\}$ and $A_2:=\left\{(x,y):x\in A,y\in A,x=y\right\}$, where A is introduced in (58). Evidently, $\left({\mathsf{P}}_X\otimes {\mathsf{P}}_X\right)\left(A_1\right)=1$ and $\left({\mathsf{P}}_X\otimes {\mathsf{P}}_X\right)\left(A_2\right)=0$. Consider $(x,y)\in A_1$. Obviously, for any $a>0$, $r_m\left(a\right)\to 0$, as $m\to \infty$. For $(x,y)\in A_1$ we take $m_5=m_5(u,w,∥x-y∥):=1+⌈{\left(\frac{2}{∥x-y∥}\right)}^{d}max\left\{u,w\right\}⌉$. Then $r_m\left(u\right)<\frac{∥x-y∥}{2}$ and $r_m\left(w\right)<\frac{∥x-y∥}{2}$ for all $m\ge m_5$. Thus, $B(x,r_m\left(u\right))\cap B(y,r_m\left(w\right))=⌀$ if $m\ge m_5$. Consequently, for $m\ge m_6(u,w,∥x-y∥):=max\left\{m_5,2(l-1)\right\}$,

$$\begin{array}{c}\hfill \begin{array}{c}\mathsf{P}\left(∥x-Y_{\left(l\right)}(x,{\mathbb{Y}}_m)∥>r_m\left(u\right),∥y-Y_{\left(l\right)}(y,{\mathbb{Y}}_m)∥>r_m\left(w\right)\right)\\ ={\displaystyle \sum _{s_1=0}^{l-1}}{\displaystyle \sum _{s_2=0}^{l-1}}\frac{m!}{s_1!s_2!(m-s_1-s_2)!}{\left(W_{m,x}\left(u\right)\right)}^{s_1}{\left(W_{m,y}\left(w\right)\right)}^{s_2}{\left(1-W_{m,x}\left(u\right)-W_{m,y}\left(w\right)\right)}^{m-s_1-s_2}.\end{array}\end{array}$$

(74)

In view of (28), (73) and (74), one has for ${\mathsf{\Phi }}_{m,l,x,y}(u,w)$ the following representation

$$\begin{array}{c}\hfill \begin{array}{c}1-{\displaystyle \sum _{s_1=0}^{l-1}}\left(\genfrac{}{}{0pt}{}{m}{s_1}\right){\left(W_{m,x}\left(u\right)\right)}^{s_1}{\left(1-W_{m,x}\left(u\right)\right)}^{m-s_1}-{\displaystyle \sum _{s_2=0}^{l-1}}\left(\genfrac{}{}{0pt}{}{m}{s_2}\right){\left(W_{m,y}\left(w\right)\right)}^{s_2}{\left(1-W_{m,y}\left(w\right)\right)}^{m-s_2}\\ +{\displaystyle \sum _{s_1=0}^{l-1}}{\displaystyle \sum _{s_2=0}^{l-1}}\frac{m!}{s_1!s_2!(m-s_1-s_2)!}{\left(W_{m,x}\left(u\right)\right)}^{s_1}{\left(W_{m,y}\left(w\right)\right)}^{s_2}{\left(1-W_{m,x}\left(u\right)-W_{m,y}\left(w\right)\right)}^{m-s_1-s_2}.\end{array}\end{array}$$

(75)

For any fixed $(x,y)\in A_1$ and $u,w\ge 0$, we get, as $m\to \infty$,

$$\begin{array}{c}\hfill \begin{array}{c}\frac{m!}{s_1!s_2!(m-s_1-s_2)!}{\left(W_{m,x}\left(u\right)\right)}^{s_1}{\left(W_{m,y}\left(w\right)\right)}^{s_2}\to \frac{{\left(V_{d}uq\left(x\right)\right)}^{s_1}}{s_1!}\frac{{\left(V_{d}wq\left(y\right)\right)}^{s_2}}{s_2!},\\ {\left(1-W_{m,x}\left(u\right)-W_{m,y}\left(w\right)\right)}^{m-s_1-s_2}\to e^{-V_d\left(uq\left(x\right)+wq\left(y\right)\right)}.\end{array}\end{array}$$

(76)

Then, according to (31), (75) and (76), for all fixed $u,w\ge 0$, $(x,y)\in A_1$, one has

$$\begin{array}{c}\hfill \begin{array}{c}{\mathsf{\Phi }}_{m,l,x,y}(u,w)\to 1-{\displaystyle \sum _{s_1=0}^{l-1}}\frac{{\left(V_{d}uq\left(x\right)\right)}^{s_1}}{s_1!}{e}^{-V_{d}uq\left(x\right)}-{\displaystyle \sum _{s_2=0}^{l-1}}\frac{{\left(V_{d}wq\left(y\right)\right)}^{s_2}}{s_2!}{e}^{-V_{d}wq\left(y\right)}\\ +{\displaystyle \sum _{s_1=0}^{l-1}}{\displaystyle \sum _{s_2=0}^{l-1}}\frac{{\left(V_{d}uq\left(x\right)\right)}^{s_1}}{s_1!}\frac{{\left(V_{d}wq\left(y\right)\right)}^{s_2}}{s_2!}{e}^{-V_d\left(uq\left(x\right)+wq\left(y\right)\right)}\\ =\left(1-{\displaystyle \sum _{s_1=0}^{l-1}}\frac{{\left(V_{d}uq\left(x\right)\right)}^{s_1}}{s_1!}{e}^{-V_{d}uq\left(x\right)}\right)\left(1-{\displaystyle \sum _{s_2=0}^{l-1}}\frac{{\left(V_{d}wq\left(y\right)\right)}^{s_2}}{s_2!}{e}^{-V_{d}wq\left(y\right)}\right)\\ =F_{l,x}\left(u\right)F_{l,y}\left(w\right):={\mathsf{\Phi }}_{l,x,y}(u,w),m\to \infty .\end{array}\end{array}$$

Thus, ${\mathsf{\Phi }}_{l,x,y}(·,·)$ is identified as a distribution function of a vector ${\eta }_{l,x,y}:=({\xi }_{l,x},{\xi }_{l,y})$ having independent components such that ${\xi }_{l,x}\sim \mathsf{\Gamma }(V_{d}q\left(x\right),l)$, ${\xi }_{l,y}\sim \mathsf{\Gamma }(V_{d}q\left(y\right),l)$. Observe also that ${\mathsf{\Phi }}_{m,l,x,y}(·,·)$ is a distribution function of a random vector ${\eta }_{m,l,x,y}:=({\xi }_{m,l,x},{\xi }_{m,l,y})$. Consequently, we have shown that ${\eta }_{m,l,x,y}\stackrel{law}{\to }{\eta }_{l,x,y}$ as $m\to \infty$. Hence, for any $(x,y)\in A_1$,

$$log{\xi }_{m,l,x}log{\xi }_{m,l,y}\stackrel{law}{\to }log{\xi }_{l,x}log{\xi }_{l,y},m\to \infty .$$

Here we take strictly positive versions of random variables under consideration. Note that, for all $i,j\in \mathbb{N}$, $i\ne j$,

$$\begin{array}{c}\hfill \begin{array}{c}\mathsf{E}(log{\xi }_{m,l,x}log{\xi }_{m,l,y})={\int }_{(0,\infty )\times (0,\infty )}logulogwd{\mathsf{\Phi }}_{m,l,x,y}(u,w)\\ =\mathsf{E}\left(log{\varphi }_{m,l}\left(i\right)log{\varphi }_{m,l}\left(j\right)|X_i=x,X_j=y\right).\end{array}\end{array}$$

(77)

One has $\mathsf{E}(log{\xi }_{l,x}log{\xi }_{l,y})=\mathsf{E}log{\xi }_{l,x}\mathsf{E}log{\xi }_{l,y}=a_{l,d}\left(x\right)a_{l,d}\left(y\right)$ because ${\xi }_{l,x}$ and ${\xi }_{l,y}$ are independent, here $a_{l,d}\left(z\right):=\psi \left(l\right)-log{V}_d-logq\left(z\right)$, $z\in {\mathbb{R}}^d$.

Now we intend to verify that, for any $(x,y)\in A_1$,

$$\begin{array}{c}\hfill \begin{array}{c}\mathsf{E}\left(log{\varphi }_{m,l}\left(1\right)log{\varphi }_{m,l}\left(2\right)|X_1=x,X_2=y\right)\to a_{l,d}\left(x\right)a_{l,d}\left(y\right).\end{array}\end{array}$$

(78)

Equivalently, one can prove that $\mathsf{E}(log{\xi }_{m,l,x}log{\xi }_{m,l,y})\to \mathsf{E}(log{\xi }_{l,x}log{\xi }_{l,y})$ for each $(x,y)\in A_1$, as $m\to \infty$.

Part (4a). We will prove the uniform integrability of a family${\{log{\xi }_{m,l,x}log{\xi }_{m,l,y}\}}_{m\ge m_0}$for$(x,y)\in A_1$. The convex function $G_N(·)$ is nondecreasing. Thus, following the proof of Step 2 for any $(x,y)\in A_1$, one can find $m_0$ (the same as in the proof of Step 2 such that, for all $m\ge m_0$,

$$\begin{array}{c}\hfill \begin{array}{c}\mathsf{E}{G}_N\left(\right|log{\xi }_{m,l,x}log{\xi }_{m,l,y}\left|\right)\le \mathsf{E}{G}_N\left(\frac{1}{2}{log}^2{\xi }_{m,l,x}+\frac{1}{2}{log}^2{\xi }_{m,l,y}\right)\\ \le \frac{1}{2}\left(\mathsf{E}{G}_N\left({log}^2{\xi }_{m,l,x}\right)+\mathsf{E}{G}_N\left({log}^2{\xi }_{m,l,y}\right)\right)\le \frac{U_1}{2}\left({\left(M_q(x,R)\right)}^{\epsilon }+{\left(M_q(y,R)\right)}^{\epsilon }\right)\\ +\frac{U_2+\kappa }{2}\left({\left(m_q(x,R)\right)}^{-\epsilon }+{\left(m_q(y,R)\right)}^{-\epsilon }\right)+\frac{A}{2}\left(R_{N,2}\left(x\right)+R_{N,2}\left(y\right)\right)+B:={\tilde{C}}_0(x,y).\end{array}\end{array}$$

(79)

Here we used (71). It is essential that $U_1,U_2,\kappa ,A,B$ do not depend on x and y. Hence, for any $(x,y)\in A_1$, a family ${\{log{\xi }_{m,l,x}log{\xi }_{m,l,y}\}}_{m\ge m_0}$ is uniformly integrable. So, we establish (78) for $(x,y)\in A_1$.

Part (4b). We return to$\mathsf{cov}(log{\varphi }_{m,l}\left(i\right),log{\varphi }_{m,l}\left(j\right))$for$i\ne j$, $i,j\in \{1,\dots ,n\}$. Set $T_{m,l}(x,y):=\mathsf{E}\left(log{\varphi }_{m,l}\left(1\right)log{\varphi }_{m,l}\left(2\right)|X_1=x,X_2=y\right)$ where $(x,y)\in A_1$. Then (78) means that $T_{m,l}(x,y)=\mathsf{E}(log{\xi }_{m,l,x}log{\xi }_{m,l,y})\to a_{l,d}\left(x\right)a_{l,d}\left(y\right)$ for any $(x,y)\in A_1$, as $m\to \infty$. Note that

$$\begin{array}{c}\hfill \begin{array}{c}{G}_N\left(\right|T_{m,l}(x,y)\left|\right)=G_N\left(\right|\mathsf{E}log{\xi }_{m,l,x}log{\xi }_{m,l,y}\left|\right)\\ \le G_N\left(\mathsf{E}\right|log{\xi }_{m,l,x}log{\xi }_{m,l,y}\left|\right)\le \mathsf{E}{G}_N\left(\right|log{\xi }_{m,l,x}log{\xi }_{m,l,y}\left|\right).\end{array}\end{array}$$

(80)

As $\left({\mathsf{P}}_X\otimes {\mathsf{P}}_X\right)\left(A_1\right)=1$, one can conclude due to (79) and (80) that, for all $m\ge m_0$,

$$\begin{array}{c}{\int }_{{\mathbb{R}}^d\times {\mathbb{R}}^d}{G}_N\left(\right|T_{m,l}(x,y)\left|\right)p\left(x\right)p\left(y\right)dxdy={\int }_{(x,y)\in A_1}{G}_N\left(\right|T_{m,l}(x,y)\left|\right)p\left(x\right)p\left(y\right)dxdy\\ \le U_1{\int }_{{\mathbb{R}}^d}{M}_q^{\epsilon }(x,R)p\left(x\right)dx+\left(U_2+\kappa \right){\int }_{{\mathbb{R}}^d}{m}_q^{-\epsilon }(x,R)p\left(x\right)dx+A{\int }_{{\mathbb{R}}^d}{R}_{N,2}\left(x\right)p\left(x\right)dx+B\\ =U_1{Q}_{p,q}(\epsilon ,R)+(U_2+\kappa )T_{p,q}(\epsilon ,R)+A{K}_{p,q}(2,N)+B<\infty .\end{array}$$

Therefore, for $(x,y)\in A_1$, the family ${\left\{T_{m,l}(x,y)\right\}}_{m\ge m_0}$ is uniformly integrable w.r.t. ${\mathsf{P}}_X\otimes {\mathsf{P}}_X$. Consequently,

$$\begin{array}{c}\hfill \begin{array}{c}{\int }_{{\mathbb{R}}^d\times {\mathbb{R}}^d}{T}_{m,l}(x,y)p\left(x\right)p\left(y\right)dxdy\to {\int }_{{\mathbb{R}}^d\times {\mathbb{R}}^d}{a}_{l,d}\left(x\right)a_{l,d}\left(y\right)p\left(x\right)p\left(y\right)dxdy,m\to \infty .\end{array}\end{array}$$

Thus

$$\mathsf{E}log{\varphi }_{m,l}\left(1\right)log{\varphi }_{m,l}\left(2\right)\to {\left(\psi \left(l\right)-log{V}_d-{\int }_{{\mathbb{R}}^d}logq\left(x\right)p\left(x\right)dx\right)}^2,m\to \infty .$$

(81)

On the other hand, taking also into account (23), we come to the relation

$$\mathsf{E}log{\varphi }_{m,l}\left(1\right)\mathsf{E}log{\varphi }_{m,l}\left(2\right)\to {\left(\psi \left(l\right)-log{V}_d-{\int }_{{\mathbb{R}}^d}logq\left(x\right)p\left(x\right)dx\right)}^2.$$

(82)

Hence (81) and (82) imply that

$$\frac{2}{n^2}\sum _{1\le i<j\le n}\mathsf{cov}\left(log{\varphi }_{m,l}\left(i\right),log{\varphi }_{m,l}\left(j\right)\right)=\frac{n-1}{n}\mathsf{cov}(log{\varphi }_{m,l}\left(1\right),log{\varphi }_{m,l}\left(2\right))\to 0,n,m\to \infty .$$

Step 5. Now we consider$\mathsf{cov}(log{\zeta }_{n,k}\left(i\right),log{\zeta }_{n,k}\left(j\right))$ for $i\ne j$, where $i,j\in \{1,\dots ,n\}$.

Similarly to Step 4, for $x,y\in {\mathbb{R}}^d$ and $u,w\ge 0$, introduce a conditional distribution function

$${\tilde{\mathsf{\Phi }}}_{n,k,x,y}^{i,j}(u,w):=\mathsf{P}({\zeta }_{n,k}\left(i\right)\le u,{\zeta }_{n,k}\left(j\right)\le w|X_i=x,X_j=y)$$

$$=\mathsf{P}\left(∥x-X_{\left(k\right)}(x,{\mathbb{X}}_n^{i,j}\cup \left\{y\right\})∥\le r_{n-1}\left(u\right),∥y-X_{\left(k\right)}(y,{\mathbb{X}}_n^{i,j}\cup \left\{x\right\})∥\le r_{n-1}\left(w\right)\right)$$

$$:=\mathsf{P}({\tilde{\eta }}_{n,k,x}^{y,i,j}\le u,{\tilde{\eta }}_{n,k,y}^{x,i,j}\le w),u,w\ge 0,$$

where ${\mathbb{X}}_n^{i,j}={\mathbb{X}}_n\backslash \{X_i,X_j\}$, ${\tilde{\eta }}_{n,k,x}^{y,i,j}:=(n-1){∥x-X_{\left(k\right)}(x,{\mathbb{X}}_n^{i,j}\cup \left\{y\right\})∥}^d$. We write ${\tilde{\mathsf{\Phi }}}_{n,k,x,y}$$(u,w)$, ${\tilde{\eta }}_{n,k,x}^y$ and ${\tilde{\eta }}_{n,k,y}^x$ instead of ${\tilde{\mathsf{\Phi }}}_{n,k,x,y}^{i,j}(u,w)$, ${\tilde{\eta }}_{n,k,x}^{y,i,j}$, ${\tilde{\eta }}_{n,k,y}^{x,i,j}$, respectively, (because $X_1,X_2,\dots$ are i.i.d. random vectors). Moreover, ${\tilde{\mathsf{\Phi }}}_{n,k,x,y}(u,w)$ is the distribution function of a random vector ${\tilde{\eta }}_{n,k,x,y}:=({\tilde{\eta }}_{n,k,x}^y,{\tilde{\eta }}_{n,k,y}^x)$ and the regular conditional distribution function of a random vector $({\zeta }_{n,k}\left(i\right),{\zeta }_{n,k}\left(j\right))$ given $(X_i,X_j)=(x,y)$. One has

$$\begin{array}{c}\hfill \begin{array}{c}{\tilde{\mathsf{\Phi }}}_{n,k,x,y}(u,w)=1-\mathsf{P}\left(∥x-X_{\left(k\right)}(x,{\mathbb{X}}_n^{i,j}\cup \left\{y\right\})∥>r_{n-1}\left(u\right)\right)\\ -\mathsf{P}\left(∥y-X_{\left(k\right)}(y,{\mathbb{X}}_n^{i,j}\cup \left\{x\right\})∥>r_{n-1}\left(w\right)\right)\\ +\mathsf{P}\left(∥x-X_{\left(k\right)}(x,{\mathbb{X}}_n^{i,j}\cup \left\{y\right\})∥>r_{n-1}\left(u\right),∥y-X_{\left(k\right)}(y,{\mathbb{X}}_n^{i,j}\cup \left\{x\right\})∥>r_{n-1}\left(w\right)\right).\end{array}\end{array}$$

Introduce

$${\tilde{R}}_{N,2}\left(x\right):={\int }_{∥x-y∥\ge e_{\left[N\right]}}{G}_N\left({log}^2∥x-y∥\right)p\left(y\right)dy,$$

${\tilde{A}}_{p,2}\left(G_N\right):=\{x\in S\left(p\right):{\tilde{R}}_{N,2}\left(x\right)<\infty \}$ and $\tilde{A}:=\mathsf{\Lambda }\left(p\right)\cap S\left(p\right)\cap D_p\left(R\right)\cap {\tilde{A}}_{p,2}\left(G_N\right)$, where the first three sets appeared in Theorem 1 proof (Step 5) Then ${\mathsf{P}}_X(S\left(p\right)\backslash {\tilde{A}}_{p,2}\left(G_N\right))=0$ since $K_{p,p}(2,N)<\infty$. It is easily seen that ${\mathsf{P}}_X\left(\tilde{A}\right)=1$.

Take ${\tilde{A}}_1:=\left\{(x,y):x\in \tilde{A},y\in \tilde{A},x\ne y\right\}$ and ${\tilde{A}}_2:=\left\{(x,y):x\in \tilde{A},y\in \tilde{A},x=y\right\}$. Evidently, $\left({\mathsf{P}}_X\otimes {\mathsf{P}}_X\right)\left({\tilde{A}}_1\right)=1$ and $\left({\mathsf{P}}_X\otimes {\mathsf{P}}_X\right)\left({\tilde{A}}_2\right)=0$. For any $a>0$, $r_m\left(a\right)\to 0$, as $m\to \infty$. Hence, for $(x,y)\in {\tilde{A}}_1$, one can find ${\tilde{n}}_5={\tilde{n}}_5(u,w,∥x-y∥)=2+⌈{\left(\frac{2}{∥x-y∥}\right)}^{d}max\left\{u,w\right\}⌉$ such that $r_{n-1}\left(u\right)<\frac{∥x-y∥}{2}$, $r_{n-1}\left(w\right)<\frac{∥x-y∥}{2}$ if $n\ge {\tilde{n}}_5$. Then $B(x,r_{n-1}\left(u\right))\cap B(y,r_{n-1}\left(w\right))=⌀$ if $n\ge {\tilde{n}}_5(u,w,∥x-y∥)$. Thus, for $n\ge {\tilde{n}}_6:=max\left\{{\tilde{n}}_5,2k\right\}$, one has

$${\tilde{\mathsf{\Phi }}}_{n,k,x,y}(u,w)=1-{\displaystyle \sum _{s_1=0}^{k-1}}\left(\genfrac{}{}{0pt}{}{n-2}{s_1}\right){\left(V_{n-1,x}\left(u\right)\right)}^{s_1}{\left(1-V_{n-1,x}\left(u\right)\right)}^{n-2-s_1}$$

$$\begin{array}{c}\hfill \begin{array}{c}-{\displaystyle \sum _{s_2=0}^{k-1}}\left(\genfrac{}{}{0pt}{}{n-2}{s_2}\right){\left(V_{n-1,y}\left(w\right)\right)}^{s_2}{\left(1-V_{n-1,y}\left(w\right)\right)}^{n-2-s_2}\end{array}\end{array}$$

$$+{\displaystyle \sum _{s_1=0}^{k-1}}{\displaystyle \sum _{s_2=0}^{k-1}}\frac{(n-2)!}{s_1!s_2!(n-2-s_1-s_2)!}{\left(V_{n-1,x}\left(u\right)\right)}^{s_1}{\left(V_{n-1,y}\left(w\right)\right)}^{s_2}{\left(1-V_{n-1,x}\left(u\right)-V_{n-1,y}\left(w\right)\right)}^{n-2-s_1-s_2}.$$

Therefore, for each fixed $(x,y)\in {\tilde{A}}_1$, $u,w\ge 0$, we get, as $n\to \infty$,

$$\begin{array}{c}\hfill \begin{array}{c}{\tilde{\mathsf{\Phi }}}_{n,k,x,y}(u,w)\to 1-{\displaystyle \sum _{s_1=0}^{k-1}}\frac{{\left(V_{d}up\left(x\right)\right)}^{s_1}}{s_1!}{e}^{-V_{d}up\left(x\right)}-{\displaystyle \sum _{s_2=0}^{k-1}}\frac{{\left(V_{d}wp\left(y\right)\right)}^{s_2}}{s_2!}{e}^{-V_{d}wp\left(y\right)}\\ +{\displaystyle \sum _{s_1=0}^{k-1}}{\displaystyle \sum _{s_2=0}^{k-1}}\frac{{\left(V_{d}up\left(x\right)\right)}^{s_1}}{s_1!}\frac{{\left(V_{d}wp\left(y\right)\right)}^{s_2}}{s_2!}{e}^{-V_d\left(up\left(x\right)+wp\left(y\right)\right)}\\ =\left(1-{\displaystyle \sum _{s_1=0}^{k-1}}\frac{{\left(V_{d}up\left(x\right)\right)}^{s_1}}{s_1!}{e}^{-V_{d}up\left(x\right)}\right)\left(1-{\displaystyle \sum _{s_2=0}^{k-1}}\frac{{\left(V_{d}wp\left(y\right)\right)}^{s_2}}{s_2!}{e}^{-V_{d}wp\left(y\right)}\right)={\tilde{F}}_{k,x}\left(u\right){\tilde{F}}_{k,y}\left(w\right)\\ :={\tilde{\mathsf{\Phi }}}_{k,x,y}(u,w).\end{array}\end{array}$$

Here ${\tilde{\mathsf{\Phi }}}_{k,x,y}(·,·)$ denotes the distribution function of a vector ${\tilde{\eta }}_{k,x,y}:=({\tilde{\xi }}_{k,x},{\tilde{\xi }}_{k,y})$. The components of ${\tilde{\eta }}_{k,x,y}$ are independent, ${\tilde{\xi }}_{k,x}\sim \mathsf{\Gamma }(V_{d}p\left(x\right),k)$ and ${\tilde{\xi }}_{k,y}\sim \mathsf{\Gamma }(V_{d}p\left(y\right),k)$. Consequently, for each fixed $(x,y)\in {\tilde{A}}_1$, we have shown that ${\tilde{\eta }}_{n,k,x,y}\stackrel{law}{\to }{\tilde{\eta }}_{k,x,y}$ as $n\to \infty$. Therefore, for such $(x,y)$,

$$log{\tilde{\eta }}_{n,k,x}^{y}log{\tilde{\eta }}_{n,k,y}^x\stackrel{law}{\to }log{\tilde{\xi }}_{k,x}log{\tilde{\xi }}_{k,y},n\to \infty .$$

(83)

Here we take strictly positive versions of the random variables under consideration. In a way similar to (77), for $i,j\in \{1,\dots ,n\}$, $i\ne j$, we write

$$\begin{array}{c}\hfill \begin{array}{c}\mathsf{E}(log{\tilde{\eta }}_{n,k,x}^{y}log{\tilde{\eta }}_{n,k,y}^x)={\int }_{(0,\infty )\times (0,\infty )}logulogwd{\tilde{\mathsf{\Phi }}}_{n,k,x,y}(u,w)\\ =\mathsf{E}\left(log{\zeta }_{n,k}\left(i\right)log{\zeta }_{n,k}\left(j\right)|X_i=x,X_j=y\right).\end{array}\end{array}$$

Since ${\tilde{\xi }}_{k,x}$ and ${\tilde{\xi }}_{k,y}$ are independent, write $\mathsf{E}(log{\tilde{\xi }}_{k,x}log{\tilde{\xi }}_{k,y})=\mathsf{E}log{\tilde{\xi }}_{k,x}\mathsf{E}log{\tilde{\xi }}_{k,y}=b_{k,d}\left(x\right)b_{k,d}\left(y\right)$, where $b_{k,d}\left(z\right):=\psi \left(k\right)-log{V}_d-logp\left(z\right)$, $z\in {\mathbb{R}}^d$.

For any fixed $M>0$, consider ${\tilde{A}}_{1,M}:=\left\{(x,y)\in {\tilde{A}}_1:∥x-y∥>M\right\}$. Now our aim is to verify that, for each $(x,y)\in {\tilde{A}}_{1,M}$,

$$\begin{array}{c}\hfill \begin{array}{c}\mathsf{E}\left(log{\zeta }_{n,k}\left(1\right)log{\zeta }_{n,k}\left(2\right)|X_1=x,X_2=y\right)\to b_{k,d}\left(x\right)b_{k,d}\left(y\right).\end{array}\end{array}$$

(84)

Equivalently, we can prove, for each $(x,y)\in {\tilde{A}}_{1,M}$, that

$$\mathsf{E}log{\tilde{\eta }}_{n,k,x}^{y}log{\tilde{\eta }}_{n,k,y}^x\to \mathsf{E}log{\tilde{\xi }}_{k,x}log{\tilde{\xi }}_{k,y},n\to \infty .$$

(85)

The idea that we consider only $(x,y)\in {\tilde{A}}_{1,M}$ is principal for the further proof.

Part (5a). We are going to establish that, for$(x,y)\in {\tilde{A}}_{1,M}$, a family$\{log{\tilde{\eta }}_{n,k,x}^y$$log{\tilde{\eta }}_{n,k,y}^x{\}}_{n\ge {\tilde{n}}_0}$is uniformly integrable, where${\tilde{n}}_0\in \mathbb{N}$is independent of$x,y$, but might depend onM. Then, due to (83), the relation (85) would be valid for such $(x,y)$ as well. As we have seen, the function $G_N(·)$ is nondecreasing and convex. Hence

$$\begin{array}{c}\hfill \begin{array}{c}\mathsf{E}{G}_N\left(\right|log{\tilde{\eta }}_{n,k,x}^{y}log{\tilde{\eta }}_{n,k,y}^x\left|\right)\le \frac{1}{2}\left(\mathsf{E}{G}_N\left({log}^2{\tilde{\eta }}_{n,k,x}^y\right)+\mathsf{E}{G}_N\left({log}^2{\tilde{\eta }}_{n,k,y}^x\right)\right).\end{array}\end{array}$$

(86)

Let us consider, for instance, $\mathsf{E}{G}_N\left({log}^2{\tilde{\eta }}_{n,k,x}^y\right)$. Alike Step 2 we can write

$$\begin{array}{c}\hfill \begin{array}{c}\mathsf{E}{G}_N\left({log}^2{\tilde{\eta }}_{n,k,x}^y\right)={\int }_{\left(0,\frac{1}{\rho \left(N\right)}\right]}{\tilde{F}}_{n,k,x}^y\left(u\right)(-h_N\left(u\right))du+{\int }_{\left(\rho \left(N\right),\infty \right)}(1-{\tilde{F}}_{n,k,x}^y\left(u\right))h_N\left(u\right)du\\ :=I_1(n,x,y)+I_2(n,x,y),\end{array}\end{array}$$

where

$$\begin{array}{c}\hfill \begin{array}{c}{\tilde{F}}_{n,k,x}^y\left(u\right):=\mathsf{P}\left({\tilde{\eta }}_{n,k,x}^y\le u\right)=1-\mathsf{P}\left(∥x-X_{\left(k\right)}(x,{\mathbb{X}}_n^{i,j}\cup \left\{y\right\})∥>r_{n-1}\left(u\right)\right)\\ =\mathbb{I}\left\{∥x-y∥>r_{n-1}\left(u\right)\right\}\left(1-{\displaystyle \sum _{s=0}^{k-1}}\left(\genfrac{}{}{0pt}{}{n-2}{s}\right){\left(V_{n-1,x}\left(u\right)\right)}^s{\left(1-V_{n-1,x}\left(u\right)\right)}^{n-2-s}\right)\\ +\mathbb{I}\left\{∥x-y∥\le r_{n-1}\left(u\right)\right\}\left(1-{\displaystyle \sum _{s=0}^{k-2}}\left(\genfrac{}{}{0pt}{}{n-2}{s}\right){\left(V_{n-1,x}\left(u\right)\right)}^s{\left(1-V_{n-1,x}\left(u\right)\right)}^{n-2-s}\right),\end{array}\end{array}$$

(87)

As usual a sum over empty set is equal to 0 (for $k=1$).

If $u\in \left(0,\frac{1}{\rho \left(N\right)}\right]$, where $\rho \left(N\right):=exp\left\{\sqrt{e_{[N-1]}}\right\}$ and $n\ge {\tilde{n}}_1:=⌈\frac{1}{\rho \left(N\right)M^d}⌉+1$, then $r_{n-1}\left(u\right)\le M$. Thus, $r_{n-1}\left(u\right)<∥x-y∥$ if $(x,y)\in {\tilde{A}}_{1,M}$. In view of (87) and similarly to (38), one has

$${\tilde{F}}_{n,k,x}^y\left(u\right)\le {\left(\frac{n-2}{n-1}\right)}^{\epsilon }{\left(M_p(x,R)V_{d}u\right)}^{\epsilon }\le {\left(M_p(x,R)\right)}^{\epsilon }{V}_d^{\epsilon }{u}^{\epsilon }$$

for $(x,y)\in {\tilde{A}}_{1,M}$, $u\in \left(0,\frac{1}{\rho \left(N\right)}\right]$, $n\ge max\{{\tilde{n}}_1\left(M\right),{\tilde{n}}_2\left(R\right)\}$, here ${\tilde{n}}_2\left(R\right):=max\{⌈\frac{1}{\rho \left(N\right)R^d}⌉,$$k+1\}$. So, $I_1(n,x,y)\le U_1(\epsilon ,N,d){\left(M_p(x,R)\right)}^{\epsilon }$ for $(x,y)\in {\tilde{A}}_{1,M}$ and $n\ge max\{{\tilde{n}}_1\left(M\right),$${\tilde{n}}_2\left(R\right)\}$. Moreover, for all $u\ge 0$, in view of (87) it holds

$$1-{\tilde{F}}_{n,k,x}^y\left(u\right)\le \sum _{s=0}^{k-1}\left(\genfrac{}{}{0pt}{}{n-2}{s}\right){\left(V_{n-1,x}\left(u\right)\right)}^s{\left(1-V_{n-1,x}\left(u\right)\right)}^{n-2-s}.$$

The same reasoning as was used in Theorem 1 proof (Step 3, Part (3b)) leads to the inequalities

$$\begin{array}{c}\hfill \begin{array}{c}1-{\tilde{F}}_{n,k,x}^y\left(u\right)\le S_1\left(k\right){\left(1-S_2{V}_{n-1,x}\left(u\right)\right)}^{n-2}\le S_{1}exp\left\{-S_2(n-2)V_{n-1,x}\left(u\right)\right\}\\ \le S_{1}exp\left\{-\frac{n-2}{n-1}{S}_2{V}_{d}u{m}_p(x,R)\right\}\le S_1{\left(\frac{S_2}{2}{V}_{d}u{m}_p(x,R)\right)}^{-\epsilon }\end{array}\end{array}$$

for all $n\ge max\left\{{\tilde{n}}_3\left(R\right),3\right\}$. Then similarly to (70), the relation

$$\begin{array}{c}\hfill \begin{array}{c}\mathsf{E}{G}_N\left({log}^2{\tilde{\eta }}_{n,k,x}^y\right)\le U_1{\left(M_p(x,R)\right)}^{\epsilon }+\left({\tilde{U}}_2+\kappa \right){\left(m_p(x,R)\right)}^{-\epsilon }+A{\tilde{R}}_{N,2}\left(x\right)+B:={\tilde{C}}_1\left(x\right)<\infty \end{array}\end{array}$$

is valid for all $(x,y)\in {\tilde{A}}_{1,M}$ and $n\ge {\tilde{n}}_0\left(M\right):=max\left\{{\tilde{n}}_1\left(M\right),{\tilde{n}}_2,{\tilde{n}}_3,{\tilde{n}}_4\left(\kappa \right),3\right\}$. Here $U_1,{\tilde{U}}_2,\kappa ,A,B$ do not depend on x and y. The term $\mathsf{E}{G}_N\left({log}^2{\tilde{\eta }}_{n,k,y}^x\right)$ can be treated in the above manner. Thus, in view of (86), one has

$$\begin{array}{c}\hfill \begin{array}{c}\mathsf{E}{G}_N\left(\right|log{\tilde{\eta }}_{n,k,x}^{y}log{\tilde{\eta }}_{n,k,y}^x\left|\right)\le \frac{U_1}{2}\left({\left(M_p(x,R)\right)}^{\epsilon }+{\left(M_p(y,R)\right)}^{\epsilon }\right)\\ +\frac{U_2+\kappa }{2}\left({\left(m_p(x,R)\right)}^{-\epsilon }+{\left(m_p(y,R)\right)}^{-\epsilon }\right)+\frac{A}{2}\left({\tilde{R}}_{N,2}\left(x\right)+{\tilde{R}}_{N,2}\left(y\right)\right)+B:={\tilde{C}}_1(x,y).\end{array}\end{array}$$

(88)

Therefore, for any $(x,y)\in {\tilde{A}}_{1,M}$, a family ${\{log{\tilde{\eta }}_{n,k,x}^{y}log{\tilde{\eta }}_{n,k,y}^x\}}_{n\ge {\tilde{n}}_0}$ is uniformly integrable. Thus, we come to (84) for $(x,y)\in {\tilde{A}}_{1,M}$.

Part (5b). Now we return to upper bound for$\mathsf{cov}\left(log{\zeta }_{n,k}\left(1\right),log{\zeta }_{n,k}\left(2\right)\right)$. Set

$${\tilde{T}}_{n,k}(x,y):=\mathsf{E}\left(log{\zeta }_{n,k}\left(1\right)log{\zeta }_{n,k}\left(2\right)|X_1=x,X_2=y\right)=\mathsf{E}log{\tilde{\eta }}_{n,k,x}^{y}log{\tilde{\eta }}_{n,k,y}^x$$

for all $(x,y)\in {\tilde{A}}_1$. Validity of (84) is equivalent to the following relation: for any $(x,y)\in {\tilde{A}}_{1,M}$, ${\tilde{T}}_{n,k}(x,y)\to b_{k,d}\left(x\right)b_{k,d}\left(y\right)$, as $n\to \infty$. Take any $(x,y)\in {\tilde{A}}_1$. For each $M>0$, it was shown that

$$\begin{array}{c}\hfill \begin{array}{c}{\tilde{T}}_{n,k}(x,y)\mathbb{I}\{∥x-y∥>M\}\to b_{k,d}\left(x\right)b_{k,d}\left(y\right)\mathbb{I}\{∥x-y∥>M\},n\to \infty .\end{array}\end{array}$$

Note that

$$\begin{array}{c}\hfill \begin{array}{c}{G}_N\left(\right|{\tilde{T}}_{n,k}(x,y)|\mathbb{I}\{∥x-y∥>M\})\le G_N\left(\right|{\tilde{T}}_{n,k}(x,y)\left|\right)=G_N\left(\right|\mathsf{E}log{\tilde{\eta }}_{n,k,x}^{y}log{\tilde{\eta }}_{n,k,y}^x\left|\right)\\ \le G_N\left(\mathsf{E}\right|log{\tilde{\eta }}_{n,k,x}^{y}log{\tilde{\eta }}_{n,k,y}^x\left|\right)\le \mathsf{E}{G}_N\left(\right|log{\tilde{\eta }}_{n,k,x}^{y}log{\tilde{\eta }}_{n,k,y}^x\left|\right).\end{array}\end{array}$$

(89)

Due to (88) and (89) one can conclude that, for all $n\ge {\tilde{n}}_0$,

$$\begin{array}{c}{\int }_{{\mathbb{R}}^d\times {\mathbb{R}}^d}{G}_N\left(\right|{\tilde{T}}_{n,k}(x,y)\left|\mathbb{I}\{∥x-y∥>M\}\right)p\left(x\right)p\left(y\right)dxdy\\ \le U_1{\int }_{{\mathbb{R}}^d}{M}_p^{\epsilon }(x,R)p\left(x\right)dx+\left({\tilde{U}}_2+\kappa \right){\int }_{{\mathbb{R}}^d}{m}_p^{-\epsilon }(x,R)p\left(x\right)dx+A{\int }_{{\mathbb{R}}^d}{\tilde{R}}_{N,2}\left(x\right)p\left(x\right)dx+B\\ =U_1{Q}_{p,p}(\epsilon ,R)+({\tilde{U}}_2+\kappa )T_{p,p}(\epsilon ,R)+A{K}_{p,p}(2,N)+B<\infty .\end{array}$$

Therefore, for $(x,y)\in {\tilde{A}}_1$, the family ${\left\{{\tilde{T}}_{n,k}(x,y)\mathbb{I}\{∥x-y∥>M\}\right\}}_{n\ge {\tilde{n}}_0}$ is uniformly integrable w.r.t. ${\mathsf{P}}_X\otimes {\mathsf{P}}_X$. Hence, by virtue of (84), for each $M>0$,

$$\begin{array}{c}\hfill \begin{array}{c}{\int }_{D\left(M\right)}{\tilde{T}}_{n,k}(x,y)p\left(x\right)p\left(y\right)dxdy\to {\int }_{D\left(M\right)}{b}_{k,d}\left(x\right)b_{k,d}\left(y\right)p\left(x\right)p\left(y\right)dxdy,n\to \infty ,\end{array}\end{array}$$

(90)

where $D\left(M\right):=\{x,y\in {\mathbb{R}}^d,∥x-y∥>M\}$. Now we turn to the case $∥x-y∥\le M$. One has ${\bigcap }_{s=1}^{\infty }\left\{∥X_1-X_2∥\le \frac{1}{s}\right\}=\left\{X_1=X_2\right\}$ and $\mathsf{P}\left(X_1=X_2\right)=0$ as $X_1$ and $X_2$ are independent and have a density $p\left(x\right)$ w.r.t. the Lebesgue measure $\mu$. Then in view of continuity of a probability measure it holds that $\mathsf{P}\left(∥X_1-X_2∥\le M\right)\to 0$, as $M\to 0.$ Taking into account that, for an integrable function h, ${\int }_{C}hd\mathsf{P}\to 0$ as $\mathsf{P}\left(C\right)\to 0$, we get

$$\mathsf{E}(log{\zeta }_{n,k}\left(1\right)log{\zeta }_{n,k}\left(2\right)\mathbb{I}\{∥X_1-X_2∥\le M\})\to 0,M\to 0,$$

since $\mathsf{E}log{\zeta }_{n,k}\left(1\right)log{\zeta }_{n,k}\left(2\right)\le \frac{1}{2}\left(\mathsf{E}{log}^2{\zeta }_{n,k}\left(1\right)+\mathsf{E}{log}^2{\zeta }_{n,k}\left(2\right)\right)<\infty$ (the proof is similar to the establishing that $\mathsf{E}{log}^2{\varphi }_{m,l}\left(1\right)<\infty$). Thus, for any $\gamma >0$, one can find $M_1=M_1\left(\gamma \right)>0$ such that, for all $M\in (0,M_1]$ and $n\ge {\tilde{n}}_0$,

$$\left|{\int }_{{\mathbb{R}}^{2d}\backslash D\left(M\right)}{\tilde{T}}_{n,k}(x,y)p\left(x\right)p\left(y\right)dxdy\right|=|\mathsf{E}log{\zeta }_{n,k}\left(1\right)log{\zeta }_{n,k}\left(2\right)\mathbb{I}\{∥X_1-X_2∥\le M\}|<\frac{\gamma }{3}.$$

(91)

Also there exists $M_2=M_2\left(\gamma \right)>0$ such that, for all $M\in (0,M_2]$,

$$\left|{\int }_{{\mathbb{R}}^{2d}\backslash D\left(M\right)}{b}_{k,d}\left(x\right)b_{k,d}\left(y\right)p\left(x\right)p\left(y\right)dxdy\right|<\frac{\gamma }{3}.$$

(92)

Take $M\left(\gamma \right):=min\{M_1\left(\gamma \right),M_2\left(\gamma \right)\}$. Due to (90) there is ${\tilde{n}}_7(M\left(\gamma \right),\gamma )$ such that $n\ge max\{{\tilde{n}}_0,{\tilde{n}}_7(M\left(\gamma \right),\gamma )\}$ entails the following inequality

$$|{\int }_{D\left(M\right)}{\tilde{T}}_{n,k}(x,y)p\left(x\right)p\left(y\right)dxdy-{\int }_{D\left(M\right)}{b}_{k,d}\left(x\right)b_{k,d}\left(y\right)p\left(x\right)p\left(y\right)dxdy|<\frac{\gamma }{3}.$$

(93)

So, in view of (91)–(93), for any $\gamma >0$, there is $M\left(\gamma \right)>0$ such that, for all n large enough, i.e., $n\ge max\{{\tilde{n}}_0,{\tilde{n}}_7(M\left(\gamma \right),\gamma )\}$, one has

$$\begin{array}{c}|{\int }_{{\mathbb{R}}^d\times {\mathbb{R}}^d}{\tilde{T}}_{n,k}(x,y)p\left(x\right)p\left(y\right)dxdy-{\int }_{{\mathbb{R}}^d\times {\mathbb{R}}^d}{b}_{k,d}\left(x\right)b_{k,d}\left(y\right)p\left(x\right)p\left(y\right)dxdy|<\gamma .\end{array}$$

(94)

By virtue of the formula

$$\begin{array}{c}{\int }_{{\mathbb{R}}^d\times {\mathbb{R}}^d}{b}_{k,d}\left(x\right)b_{k,d}\left(y\right)p\left(x\right)p\left(y\right)dxdy={\left(\psi \left(k\right)-log{V}_d-{\int }_{{\mathbb{R}}^d}(logp\left(x\right))p\left(x\right)dx\right)}^2,\end{array}$$

and taking into account (94) we deduce the limit relation, for $n\to \infty$,

$$\begin{array}{c}\mathsf{E}log{\zeta }_{n,k}\left(1\right)log{\zeta }_{n,k}\left(2\right)\to {\left(\psi \left(k\right)-log{V}_d-{\int }_{{\mathbb{R}}^d}(logp\left(x\right))p\left(x\right)dx\right)}^2.\end{array}$$

Moreover, in view of (24) (see Step 5 of Theorem 1 proof), it follows that

$$\begin{array}{c}\mathsf{E}log{\zeta }_{n,k}\left(1\right)\mathsf{E}log{\zeta }_{n,k}\left(2\right)\to {\left(\psi \left(k\right)-log{V}_d-{\int }_{{\mathbb{R}}^d}(logp\left(x\right))p\left(x\right)dx\right)}^2.\end{array}$$

Therefore,

$$\frac{2}{n^2}\sum _{1\le i<j\le n}\mathsf{cov}\left(log{\zeta }_{n,k}\left(i\right),log{\zeta }_{n,k}\left(j\right)\right)=\frac{n-1}{n}\mathsf{cov}(log{\zeta }_{n,k}\left(1\right),log{\zeta }_{n,k}\left(2\right))\to 0,n\to \infty .$$

Step 6. Here we complete the analysis of summands in (56). Reasoning as at Steps 1–3 shows that $\frac{1}{n^2}{\sum }_{i=1}^n\mathsf{var}\left(log{\zeta }_{n,k}\left(i\right)\right)$$=\frac{1}{n}\mathsf{var}\left(log{\zeta }_{n,k}\left(1\right)\right)\to 0$ since

$$\mathsf{var}(log{\zeta }_{n,k}\left(i\right))=\mathsf{var}(log{\zeta }_{n,k}\left(1\right))\to v_k<\infty$$

for each $i\in \mathbb{N}$, as $n\to \infty$. It remains to prove that $\frac{2}{n^2}{\sum }_{i,j=1}^n\mathsf{cov}(log$${\varphi }_{m,l}\left(i\right),$$log{\zeta }_{n,k}\left(j\right))\to 0$, as $n,m\to \infty$.

For $i=1,\dots ,n$, one has $|\mathsf{cov}\left(log{\varphi }_{m,l}\left(i\right),log{\zeta }_{n,k}\left(i\right)\right)|\le {\left(\mathsf{var}(log{\varphi }_{m,l}\left(1\right))\mathsf{var}(log{\zeta }_{n,k}\left(1\right))\right)}^{\frac{1}{2}}$$<\infty$ for all $n,m$ large enough. So, it suffices to show that

$$\frac{1}{n^2}\sum _{i,j=1,\dots ,n;i\ne j}\mathsf{cov}(log{\varphi }_{m,l}\left(i\right),log{\zeta }_{n,k}\left(j\right))\to 0,n,m\to \infty .$$

For $i,j=1,\dots ,n$, $i\ne j$, $u,w\ge 0$, $x,y\in {\mathbb{R}}^d$, let us introduce a conditional distribution function

$$\begin{array}{c}\hfill \begin{array}{c}\mathsf{P}\left({\varphi }_{m,l}\left(i\right)\le u,{\zeta }_{n,k}\left(j\right)\le w|X_i=x,X_j=y\right)\\ =\mathsf{P}\left(∥x-Y_{\left(l\right)}(x,{\mathbb{Y}}_m)∥\le r_m\left(u\right),∥y-X_{\left(k\right)}(y,{\mathbb{X}}_n^{i,j}\cup \left\{x\right\})∥\le r_{n-1}\left(w\right)\right)\\ =\mathsf{P}\left(∥x-Y_{\left(l\right)}(x,{\mathbb{Y}}_m)∥\le r_m\left(u\right)\right)\mathsf{P}\left(∥y-X_{\left(k\right)}(y,{\mathbb{X}}_n^{i,j}\cup \left\{x\right\})∥\le r_{n-1}\left(w\right)\right)\\ =\left(1-{\displaystyle \sum _{s_1=0}^{l-1}}\left(\genfrac{}{}{0pt}{}{m}{s_1}\right){\left(W_{m,x}\left(u\right)\right)}^{s_1}{(1-W_{m,x}\left(u\right))}^{m-s_1}\right)\\ ·(\mathbb{I}\left\{∥x-y∥>r_{n-1}\left(w\right)\right\}\left(1-{\displaystyle \sum _{s=0}^{k-1}}\left(\genfrac{}{}{0pt}{}{n-2}{s}\right){\left(V_{n-1,y}\left(w\right)\right)}^s{\left(1-V_{n-1,y}\left(w\right)\right)}^{n-2-s}\right)\\ +\mathbb{I}\left\{∥x-y∥\le r_{n-1}\left(w\right)\right\}\left(1-{\displaystyle \sum _{s=0}^{k-2}}\left(\genfrac{}{}{0pt}{}{n-2}{s}\right){\left(V_{n-1,y}\left(w\right)\right)}^s{\left(1-V_{n-1,y}\left(w\right)\right)}^{n-2-s}\right)).\end{array}\end{array}$$

We used that $\{{\mathbb{X}}_n,{\mathbb{Y}}_m\}$ is a collection of independent vectors. Now we combine the estimates obtained at Steps 4 and 5 of Theorem 2 proof to verify that, for $i,j\in \{1,\dots ,n\}$ and $i\ne j$, $\mathsf{cov}\left(log{\varphi }_{m,l}\left(i\right),log{\zeta }_{n,k}\left(j\right)\right)=\mathsf{cov}\left(log{\varphi }_{m,l}\left(1\right),log{\zeta }_{n,k}\left(2\right)\right)\to 0$, $n,m\to \infty$.

Thus, we have established that $\mathsf{var}\left({\widehat{D}}_{n,m}(k,l)\right)\to 0$ as $n,m\to \infty$, hence (16) holds. The proof of Theorem 2 is complete.

6. Conclusions

The aim of this paper is to provide wide conditions ensuring the asymptotic unbiasedness and mean square consistency for statistical estimates of the Kullback–Leibler divergence proposed in [31]. We do not impose restrictions on the smoothness of the densities under consideration and do not assume that the densities have bounded supports. Thus, in particular one can apply our results to various mixtures of distributions, for instance, to mixture of any nondegenerate normal laws in ${\mathbb{R}}^d$ (Corollary 4). As a byproduct we relax conditions in our recent analysis of the Kozachenko - Leonenko type estimators for the Shannon differential entropy [55] and use these conditions in estimating the cross-entropy as well. Observe that the integral functional $K_{p,q}$ appearing in Theorems 1–3 involves the function $G_N\left(t\right)$ which is close to a function t when parameter N is large enough. Thus, we impose essentially less restrictive condition than one requiring a function $G\left(t\right)=t^{1+\nu }$ for some $\nu >0$ instead of $G_N\left(t\right)$. Even for the latter choice of G our results provide the first valid proof without appealing to the Fatou lemma (the long standing problem to obtain correct proofs was discussed in Introduction). An interesting and hard problem for future research is to find the class of functions $\phi :{\mathbb{R}}_{+}\to {\mathbb{R}}_{+}$ such that one can replace $G_N\left(t\right)$ in expression of $K_{p,q}$ by $G\left(t\right)=t\phi \left(t\right)$, where $\phi \left(t\right)\to \infty$, as $t\to \infty$, and keep the validity of established theorems. Here one can see the analogy with investigation of fluctuations of sums of random variables or Brownian motion by G.H.Hardy, H.D.Steinhaus, A.Ya.Khinchin, A.N.Kolmogorov, I.G.Petrovski, W.Feller and other researchers. The increasing precision on the way of description of the upper and lower functions has led to the law of the iterated logarithm and its generalizations. Another deep problem is to provide sharp conditions of CLT validity for estimates of the Kullback - Leibler divergence.

Beside pure theoretical aspects the estimates of entropy and related functionals have diverse applications. In [5], the estimates of the Kullback - Leibler divergence are applied to the change-point detection in time series. That issue is important, e.g., in analysis of stochastic financial models. Moreover, it is interesting to study the spatial variant of this problem. Namely, in [55,56] statistical estimates of entropy and scan-statistics (see, e.g., [57]) were employed for identification of inhomogeneities of fiber materials. In [58], the Kullback–Leibler divergence estimators are used to identify multivariate spatial clusters in the Bernoulli model. A modification of the latter paper idea can also be applied to analysis of the fiber structures. Such structures in ${\mathbb{R}}^d$ can be modeled by spatial point stochastic process to specify the locations of the centers of fibers (segments). A certain law on the unit sphere of ${\mathbb{R}}^d$ can be used to model their directions. The length of fibers can be fixed or follow some distribution on ${\mathbb{R}}_{+}$. Since various scan domains could contain random number of observations the development of present results will be applied along with the theory of random sums of random variables. The latter theory (see, e.g., [59]) is essential in this case. Moreover, we intend to employ the studied estimators in the feature selection theory, actively used in Genome-wide association studies (GWAS), see, e.g., [16,17,22]. In this regard statistical estimates of the mutual information were proposed, see, e.g., [12]. Note also an important problem of stability analysis of constructing, by means of statistical estimates of the mutual information, a sub-collection of relevant (in a sense) factors determining a random response. The above mentioned applications will be considered in separate publications, supplemented with computer simulations and illustrative graphs.