k-Nearest Neighbor Based Consistent Entropy Estimation for k-Nearest Neighbor Based Consistent Entropy Estimation for Hyperspherical Distributions Hyperspherical Distributions

: A consistent entropy estimator for hyperspherical data is proposed based on the k -nearest neighbor (knn) approach. The asymptotic unbiasedness and consistency of the estimator are proved. Moreover, cross entropy and Kullback-Leibler (KL) divergence estimators are also discussed. Simulation studies are conducted to assess the performance of the estimators for models including uniform and von Mises-Fisher distributions. The proposed knn entropy estimator is compared with the moment based counterpart via simulations. The results show that these two methods are comparable


Introduction
The Shannon (or differential) entropy of a continuously distributed random variable (r.v.) X with probability density function (pdf ) f is widely used in probability theory and information theory as a measure of uncertainty.It is defined as the negative mean of the logarithm of the density function, i.e., k-Nearest neighbor (knn) density estimators were proposed by Mack and Rosenblatt [1].Penrose and Yukich [2] studied the laws of large numbers for k-nearest neighbor distances.The nearest neighbor entropy estimators when X ∈ R p were studied by Kozachenko and Leonenko [3].Singh et al. [4] and Leonenko et al. [5] extended these estimators using k-nearest neighbors.Mnatsakanov et al. [6] studied knn entropy estimators for variable rather than fixed k.Eggermontet et al. [7] studied the kernel entropy estimator for univariate smooth distributions.Li et al. [8] studied parametric and nonparametric entropy estimators for univariate multimodal circular distributions.Neeraj et al. [9] studied knn estimators of circular distributions for the data from the Cartesian product, that is, [0, 2π) p .Recently, Mnatsakanov et al. [10] proposed an entropy estimator for hyperspherical data based on the moment-recovery (MR) approach (see also Section 4.3).
In this paper, we propose k-nearest neighbor entropy, cross-entropy and KL-divergence estimators for hyperspherical random vectors defined on a unit p-hypersphere S p−1 centered at the origin in p-dimensional Euclidean space.Formally, The surface area S p of the hypersphere is well known: , where Γ is the gamma function.For a part of the hypersphere, the area of a cap with solid angle φ relative to its pole is given by Li [11] (cf. Gray [12]): where sgn is the sign function, and I x (α, β) is the regularized incomplete beta function.
For a random vector from the unit circle S 1 , the von Mises distribution vM(μ, κ) is the most widely used model: where T is the transpose operator, ||μ|| = 1 and κ ≥ 0 are the mean direction vector and concentration parameters, and I 0 is the zero-order modified Bessel function of the first kind.Note that the von Mises distribution has a single mode.The multimodal extension to the von Mises distribution is the so-called generalized von Mises model.Its properties are studied by Yfantis and Borgman [13] and Gatto and Jammalamadaka [14].
The generalization of von Mises distribution onto S p−1 is the von Mises-Fisher distribution (also known as Langevin distribution) vMF p (μ, κ) with pdf, where the normalization constant is and I ν (x) is the ν-order modified Bessel function of the first kind.See Mardia and Jupp [15] (p.167) for details.
Since von Mises-Fisher distributions are members of the exponential family, by differentiating the cumulant generating function, one can obtain the mean and variance of μ T X: where A p (κ) = I p/2 (κ)/I p/2−1 (κ), and /κA p (κ). See Watamori [16] for details.Thus the entropy of f p is: and Spherical distributions have been used to model the orientation distribution functions (ODF) in HARDI (High Angular Resolution Diffusion Imaging).Knutsson [17] proposed a mapping from (p = 3) orientation to a continuous and distance preserving vector space (p = 5).Rieger and Vilet [18] generalized the orientation in any p-dimensional spaces.McGraw et al. [19] used vMF 3 mixture to model the 3-D ODF and Bhalerao and Westin [20] applied vMF 5 mixture to 5-D ODF in the mapped space.Entropy of the ODF is proposed as a measure of anisotropy ( Özarslan et al. [21], Leow et al. [22]).
McGraw et al. [19] used Rényi entropy for the vMF 3 mixture since it has a closed form.Leow et al. [22] proposed an exponential isotropy measure based on the Shannon entropy.In addition, KL-divergence can be used to measure the closeness of two ODF's.A nonparametric entropy estimator based on knn approach for hyperspherical data provides an easy way to compute the entropy related quantities.
In Section 2, we will propose the knn based entropy estimator for hyperspherical data.The unbiasedness and consistency are proved in this section.In Section 3, the knn estimator is extended to estimate cross entropy and KL-divergence.In Section 4, we present simulation studies using uniform hyperspherical distributions and aforementioned vMF probability models.In addition, the knn entropy estimator is compared with the MR approach proposed in Mnatsakanov et al. [10].We conclude this study in Section 5.

Construction of knn Entropy Estimators
Let X ∈ S p−1 be a random vector having pdf f and X 1 , X 2 , . . ., X n be a set of i.i.d.random vectors drawn from f .To measure the nearness of two vectors x and y, we define a distance measure as the angle between them: φ = arccos(x T y) and denote the distance between X i and its k-th nearest neighbor in the set of n random vectors by φ i := φ n,k,i .
With the distance measure defined above and without loss of generality, the naïve k-nearest neighbor density estimate at X i is thus, where S(φ i ) is the cap area as expressed by (3).Let L n,i be the natural logarithm of the density estimate at X i , and thus we construct a similar k-nearest neighbor entropy estimator (cf.Singh et al. [4]): where ψ(k) = Γ (k) Γ(k) is the digamma function.In the sequel, we shall prove the asymptotic unbiasedness and consistency of H n (f ).

Unbiasedness of H n
To prove the asymptotic unbiasedness, we first introduce the following lemma: Lemma 2.1.For a fixed integer k < n, the asymptotic conditional mean of L n,i given Proof.∀ ∈ R, consider the conditional probability Equation (11) implies that there are at most k samples falling within the cap C i centered at Define a random variable L i having the conditional cumulative density function, By taking derivative w.r.t., we obtain the conditional pdf of L i : The conditional mean of L i is and Moreover, η k,x is a gamma r.v. with the shape parameter k and the rate parameter f (x).
Theorem 2.3.If a pdf f satisfies the following conditions: for some > 0, (A 1 ) : then the estimator proposed in (9) is asymptotically unbiased.
Proof.According to Corollary 2.2 and condition (A 2 ), we can show (see ( 16)-( 22)) that for almost all values of x ∈ S p−1 , there exists a positive constant C such that ] < C for all sufficiently large n.Hence, applying the moment convergence theorem [23] (p.186), it follows that for almost all values of x ∈ S p−1 .In addition, using Fatou's lemma and condition (A 1 ), we have that lim sup where C is a constant.Therefore, To show (i), one can follow the arguments similar to those used in the proof of Theorem 1 in [24].Indeed, we can first establish Namely, we justify that (i) is valid when n = 2 and k = 1.But the inequality (ii) follows immediately from the condition (A 2 ) and Here φ 1,2 = arccos(X T 1 X 2 ) and 1(•) is the indicator function.Now let us denote the distribution function of η n,k,x by φ} with the pole x and base radius sin φ.Note also that the functions S(φ) (see (3)) and φ n (u) = S −1 ( u n ) are both increasing functions.Now, one can see (cf.(66) in [24]): where It is easy to see that for sufficiently large n and almost all x ∈ S p−1 : and (cf. ( 89) and (85) in [24], respectively).Finally, let us show that I 3 → 0 as n → ∞.For each x with f (x) > 0, if we choose a δ ∈ (0, f(x)), then for all sufficiently large n, Using arguments similar to those used in (69)-( 72) from [24], we have The integral in (19) after changing the variable, t = 2u n , takes the form ).The first integral in the right side of ( 20) is bounded as follows: while for the second one, we have where Combination of ( 15)-( 22) and (ii) yields (i).
Remark.Note that where B(•, •) is the beta function.Hence, in the conditions (A j ), j = 2, 4, 6 and 8, the difference ) can be replaced by 1 − x T y.

Consistency of H n
Lemma 2.4.Under the following conditions: for some > 0, (A 3 ) : the asymptotic variance of L n,i is finite and equals V f [ln f (X)] + ψ 1 (k), where ψ 1 (k) is the trigamma function.
Proof.The conditions (A 3 ) and (A 4 ), and the argument similar to the one used in the proof of Theorem 2.3, yields Similarly to ( 14), we have Since After some algebra, it can be shown that Lemma 2.5.For a fixed integer k < n, L n,i are asymptotically pairwise independent.
Proof.For a pair of random variables L n,i and L n,j with i = j and X i = X j , following the similar argument for Lemma 2.1, C i and C j shrink as n increases.Thus, it is safe to assume that C i and C j are disjoint for large n, and L n,i and L n,j are independent.Hence Lemma 2.5 follows.
Theorem 2.6.Under the conditions (A 1 ) through (A 4 ), the variance of H n (f ) decreases with sample size n, that is and H n (f ) is a consistent estimator of H(f ).
Theorem 2.6 can be established by using Theorem 2.3 and Lemmas 2.4 and 2.5, and For a finite sample, the variance of H n (f ) can be approximated by See the illustration in Figure 1.The simulation was done with sample size n = 1000 and the number of simulations was N = 10, 000.Since ψ 1 (k) is a decreasing function, the variance of H n (f ) decreases when k increases.

Estimation of Cross Entropy
The definition of cross entropy between continuous pdf 's f and g is, Given a random sample of size n from f , {X 1 , X 2 , . . ., X n }, and a random sample of size m from g, {Y 1 , Y 2 , . . ., Y m }, on a hypersphere, denote the knn density estimator of g by g m .Similarly to (7), where ϕ i is the distance from X i to its k-th nearest neighbor in {Y 1 , Y 2 , . . ., Y m }.Analogously to the entropy estimator (9), the cross entropy can be estimated by: Under the conditions (A 1 )-(A 4 ), for a fixed integer k < min(n, m), one can show that H n,m (f, g) is asymptotically unbiased.Moreover, by similar reasoning applied for H n (f ), one can show that For example, when both f and g are vMF with the same mean direction and different concentration parameters, κ 1 and κ 2 , respectively, the approximate variance will be 1 n [κ 2 2 A p (κ 1 ) + ψ 1 (k)].Figure 2 shows the approximated and simulated variance of the knn estimators for cross entropy are close to each other and both decrease with k.The simulation is done with sample size n = m = 1000 and the number of simulations was N = 10, 000.

Estimation of KL-Divergence
KL-divergence is also known as relative entropy.It is used to measure the similarity of two distributions.Wang et al. [24] studied the knn estimator of KL-divergence for distributions defined on R p .Here we propose the knn estimator of KL-divergence of continuous distribution f from g defined on a hypersphere.The KL-divergence is defined as: Equation ( 30) can also be expressed as where g m (X i ) is defined as in (28).Besides, for finite samples, the variance of the estimator, When f and g are vMF as mentioned above, with concentration parameter κ 1 and κ 2 , respectively, we have: and So the approximate variance is Figure 3 shows the approximated and simulated variance of the knn estimators for KL-divergence.The approximation for von Mises-Fisher distribution is not as good as the one for uniform distributions.This could be due to the modality of von Mises-Fisher distributions or the finitude of sample sizes.The larger the sample size, the closer the approximation is to the true value.In summary, we have Corollary 3.1.(1) Under conditions (A 1 ), (A 2 ) and for some > 0, (A 5 ) : for a fixed integer k < min(n, m), the knn estimator of KL-divergence given in (31) is asymptotically unbiased.
To prove the last two corollaries, one can follow the similar steps proposed in Wang et al. [24].

Simulation Study
To demonstrate the proposed knn entropy estimators and assess their performance for finite samples, we conducted simulations for the uniform distribution and von Mises-Fisher distributions with the p-coordinate unit vector, e p , as the common mean direction for p = 3 and 10.For each distribution, we drew samples of size n = 100, 500 and 1000.All simulations were repeated N = 10, 000 times.Bias, standard deviation (SD) and root mean squared error (RMSE) were calculated.

Bias and Standard Deviation
Figures 4-9 show simulated bias and standard deviation of the proposed entropy, cross-entropy and KL-divergence estimators along different k.The pattern for the standard deviation is clear.It decreases sharply then slowly as k increases.This is consistent with the variance approximations described in Sections 2 and 3.The pattern for bias is diverse.For uniform distributions, the bias term is very small.When the underlying distribution has a mode, for example, vMF models used in the current simulations, the relation between bias and k becomes complex and the bias term can be larger for larger k values.

Convergence
To validate the consistency, we conducted simulations of different sample size n from 10 to 100,000 for the distribution models used above.Figures 10 and 11 shows the estimates and theoretical values of entropy, cross-entropy and KL-divergence for different sample sizes with k = 1 and k = ln n + 0.5 = 2-12, respectively.The proposed estimators converge to the corresponding theoretical values quickly.Thus the consistency of these estimators are verified.The choice of k is an open problem for knn based estimation approaches.These figures show that using lager k, e.g., the logarithm of n, for lager n, is giving a slightly better preference.

Comparison with the Moment-Recovered Construction
Another entropy estimator for hyperspherical data was developed recently by Mnatsakanov et al. [10] using MR approach.We call this estimator the MR entropy estimator and denote it by H (MR) n (f ): where P n,t (X i ) is the estimated probability of the cap {y ∈ S p−1 : y T X i ≥ t} defined by the revolution axis X i and t is the distance from the cap base to the origin and acts as a tuning parameter.Namely, (see Mnatsakanov et al. [10]), Via simulation study, the empirical comparison between H n (f ) and H (MR) n (f ) was done for the uniform and vMF distributions.The results are presented in Table 1.The values of k and t listed in the table are the optimal ones in the sense of minimizing RMSE.Z-tests and F-tests (at α = 0.05) were performed to compare the bias, standard deviation (variance) and RMSE (MSE) between the knn estimators and corresponding MR estimators.In general, for uniform distributions, there are no significant difference for biases.Among other comparisons, the differences are significant.Specifically, knn achieves slightly smaller bias and RMSE values than those of the MR method.The standard deviations of knn method are also smaller for the uniform distribution but larger for vMF distributions than those based on MR approach.

Discussion and Conclusions
In this paper, the knn based estimators for entropy, cross-entropy and Kullback-Leibler divergence are proposed for distributions on hyperspheres.Asymptotic properties such as unbiasedness and consistency are proved and validated by simulation studies using uniform and von Mises-Fisher distribution models.The variances of these estimators decrease with k.For uniform distributions, variance is dominant and bias is negligible.When the underlying distributions are modal, the bias can be large if k is large.In general, we conclude that the behavior of knn and MR entropy estimators have similar performance in terms of root mean square error.

Figure 1 .
Figure 1.Variances of H n (f ) by simulation and approximation.

Figure 2 .
Figure 2. Variances of H n,m by simulation and approximation.

Figure 3 .
Figure 3. Variances of KL n,m by simulation and approximation.

Figure 4 .
Figure 4. |Bias| (dashed line) and standard deviation (solid line) of entropy estimate H n for uniform distributions.

Figure 5 .
Figure 5. |Bias| (dashed line) and standard deviation (solid line) of entropy estimate H n for vMF p (e p , 1) distributions.

Figure 6 .
Figure 6.|Bias| (dashed line) and standard deviation (solid line) of cross entropy estimate H n,m for uniform distributions.

Figure 7 .
Figure 7. |Bias| (dashed line) and standard deviation (solid line) of cross entropy estimate H n,m for f = vMF p (e p , 1) and g = uniform distributions.

Figure 8 .
Figure 8. |Bias| (dashed line) and standard deviation (solid line) of KL-divergence estimate KL n,m for uniform distributions.

Figure 9 .
Figure 9. |Bias| (dashed line) and standard deviation (solid line) of KL-divergence estimate KL n,m for f = vMF p (e p , 1) and g = uniform distributions.

Figure 10 .
Figure 10.Convergence of estimates with sample size n using the first nearest neighbor.For vMF p , κ = 1.
It is reasonable to consider the Poisson approximation of Y n,i with mean λ n,i = np n,i = ke − Sc i p n,i .Thus, the limiting distribution of Y n,i is a Poisson distribution with mean:

Table 1 .
Comparison of knn and moment methods by simulations for spherical distributions.