Nearest Neighbor Estimates of Entropy for Multivariate Circular Distributions

Author to whom correspondence should be addressed; E-Mail: neeraj@iitk.ac.in.Received: 26 February 2010 / Accepted: 29 April 2010 / Published: 6 May 2010Abstract: In molecular sciences, the estimation of entropies of molecules is importantfor the understanding of many chemical and biological processes. Motivated by theseapplications, we consider the problem of estimating the entropies of circular randomvectors and introduce non-parametric estimators based on circular distances between nsample points and their k th nearest neighbors (NN), where k (n 1) is a ﬁxedpositive integer. The proposed NN estimators are based on two different circular distances,and are proven to be asymptotically unbiased and consistent. The performance of oneof the circular-distance estimators is investigated and compared with that of the alreadyestablished Euclidean-distance NN estimator using Monte Carlo samples from an analyticdistribution of six circular variables of an exactly known entropy and a large sampleof seven internal-rotation angles in the molecule of tartaric acid, obtained by a realisticmolecular-dynamics simulation.Keywords: circular random variables; differential entropy; non-parametric estimation;nearest neighbor; circular distance; molecular simulation


Introduction
Estimation of entropies of molecules is an important problem in molecular sciences.Internal configurational entropy of a molecule is the entropy of the joint distribution of the internal molecular coordinates (bond lengths, bond angles, and dihedral angles), and as such it is a measure of random fluctuations in these coordinates.Most significant contribution to the internal configurational entropy of a molecule comes from the fluctuations in dihedral angles (also called internal-rotation angles).Many important properties of complex molecules, such as their stability and adopted conformation, depend on random fluctuations in their internal coordinates.Estimation of the internal configurational entropy of molecules is therefore important for understanding many chemical and biological processes, such as the spontaneity of a chemical reaction, protein folding, intermolecular protein-protein interactions, and protein-ligand interactions.It is also a key in the design of drugs that can stabilize the normally folded molecular structure or correct a misfolded structure, since protein misfolding is a cause of several diseases such as Alzheimer disease, mad cow disease, cystic fibrosis, and some types of cancer.
Estimation of the internal entropy of macromolecules, such as proteins, is a challenging problem because of the large number of correlated internal molecular coordinates.A commonly used method of estimating the internal entropy of a molecule, known as the quasi-harmonic approach, is based on the assumption of a multivariate normal distribution for the internal molecular coordinates [1].Misra et al. [2] discussed the decision theoretic estimation of the entropy of a multivariate normal distribution and obtained improvements over the best affine equivariant estimator under the squared error loss function.However, the assumption of a multivariate normal distribution for the internal coordinates of a molecule is appropriate only at low temperatures, when the fluctuations in its internal coordinates are small.At higher temperatures, the dihedral angles of a complex molecule exhibit multimodes and skewness in their distributions, and the multivariate normal distribution becomes inadequate.
Demchuk and Singh [3] discussed a circular probability approach for modeling the dihedral angles of a molecule in the estimation of internal rotational entropy.As an illustration, they modeled the torsional angle of the methanol molecule by a trimodal von Mises distribution and derived a bath-tub-shaped distribution for the torsional potential energy of the molecule.Singh et al. [4] introduced a torus version of a bivariate normal distribution for modeling two dihedral angles.The marginal distributions of the model are symmetric unimodal or symmetric bimodal depending on the configurations of the parameters.A multivariate generalization of this bivariate model has been proposed by Mardia et al. [5].Hnizdo et al. [6] and Darian et al. [7] used a Fourier series expansion approach for modeling univariate and bivariate distributions of molecular dihedral angles.Complex molecules, however, have many significantly correlated dihedral angles, whose joint distribution can take an arbitrary form.For this reason, a non-parametric approach for estimating the entropy of a circular random vector of arbitrary dimensions is desirable.
Several non-parametric estimators of the entropy of an m-dimensional random variable X have been discussed in the literature.A common approach is to replace the probability density function (pdf) f (•) in the definition of the differential entropy, by its non-parametric kernel or histogram density estimator [8,9].However, in most practical situations, implementation of such estimates in higher dimensions becomes difficult.In one dimension (m = 1), several authors have proposed estimates of entropy in the context of testing goodness of fit [10,11].Singh et al. [12] proposed the following asymptotically unbiased and consistent nearest-neighbor (NN) estimator of the entropy H(f ): Here, R i,k,n is the Euclidean distance of a point is the usual gamma function.For k = 1, the estimator Ĥk,n reduces to the NN estimator proposed by Kozachenko and Leonenko [13].Results similar to those of [12] have been also reported by Goria et al. [14].For the purpose of the estimation of the information-theoretic quantity of mutual information, Kraskov et al. [15] generalized the first-nearest-neighbor estimator of [13] in terms of k th nearest-neighbor distances in a general metric, giving, however, explicit expressions only for the maximum and Euclidean metrics and without providing formal proofs of asymptotic unbiasedness and consistency.For m = k = 1, Tsybakov and van der Meulen [16] established the mean-square-root-n consistency of a truncated version of Ĥk,n .Earlier, Loftsgaarden and Quesenberry [17] had used NN distances to construct non-parametric estimates of a multivariate pdf.Recently, Mnatsakanov et al. [18] studied k-NN estimators of entropy in which the parameter k is assumed to be a function of the sample size.
The NN entropy estimator (2) uses Euclidean distances between the sample points.However, when the random variable X is circular, it is natural to base an NN estimate of entropy on a circular distance rather than the Euclidean distance.A circular observation can be regarded as a point on a circle of unit radius.Once an initial direction and an orientation of the circle have been chosen, each circular observation can be specified by the angle from the initial direction to the point on the circle corresponding to the observation.In this paper, we construct estimates of the entropy of an m-dimensional circular random vector Θ ∈ (0, 2π] m based on two different definitions of circular distances.Let φ = (φ 1 , . . ., φ m ) ∈ [0, 2π) m and ψ = (ψ 1 , . . ., ψ m ) ∈ [0, 2π) m be two observations on an m-dimensional circular random vector Θ.We define two circular distance functions d 1 (•, •) and d 2 (•, •) as follows: and Note that π − |π − |φ i − ψ i ||, i = 1, . . ., m, is the arc length between the points (cos φ i , sin φ i ) and (cos ψ i , sin ψ i ) on the unit circle S 1 .On the other hand, [2(1 − cos(φ i − ψ i ))] 1/2 is the Euclidean distance between the points (cos φ i , sin φ i ) and (cos ψ i , sin ψ i ) on the unit circle S 1 .
In Section 2 and Section 3, we propose explicit expressions for NN estimators of entropy based on the circular distance functions d 1 (•, •) and d 2 (•, •), respectively, and prove their asymptotic unbiasedness and consistency (with some mathematical details given in an Appendix).
In Section 4, we compare the performance of the estimator based on the circular distance d 1 with that of the Euclidean-distance estimator (2) using Monte Carlo simulations from an analytic 6-dimensional circular distribution, where the exact joint entropy is known.We there also apply the d 1 -distance estimator to the problem of estimating the entropy of a 7-dimensional joint distribution of the internal-rotation angles in the molecule of tartaric acid, using a large sample of these angles obtained by a realistic molecular-dynamics simulation.

Nearest Neighbor Estimates of Entropy Based on Circular Distance d 1
For constructing nearest neighbor estimates of the entropy of a circular random vector based on the distance function (3), we first derive the expression for the volume of a ball ) m and let V r be the volume of the ball N r (ψ), defined by (5).Then where A m (•) denotes the cumulative distribution function of the sum of m independent and identically distributed random variables each having a beta distribution with parameters α = 1 2 , β = 1.Proof.Without loss of generality, we may take ψ = (0, . . ., 0).Then . .dθ m and U 1 , . . ., U m are independent and identically distributed uniform random variables over the interval (0, 2π).
is the pdf of a beta random variable with parameters α = 1/2 and β = 1.
The circular distance (3) becomes the Euclidean distance , where θ ∈ (0, 2π) m and lim q→∞ r q = 0, thus tend to the corresponding Euclidean-distance balls as q → ∞.We can therefore apply the Lebesgue differentiation theorem [19] to the probability density function f (θ) of a circular random variable Θ ∈ [0, 2π) m in the form where V rq is given by (6).Equation (7) suggests that, given a sufficiently large random sample Θ 1 , . . ., Θ n from the distribution of Θ, the probability density function where | N r (θ) | denotes the cardinality of the set {i : Θ i ∈ N r (θ)} and r is sufficiently small.Guided by this insight, we will now construct nearest neighbor estimates of entropy for an m-variate circular random vector Θ, having a probability density function f (•).Let Θ 1 , . . ., Θ n be a random sample from the distribution of Θ and let k ∈ {1, . . ., n − 1} be a given positive integer.For i ∈ {1, . . ., n}, let d 1 (i, k, n) denote the circular distance of Θ i from its k th closest neighbor, with respect to the circular distance d 1 (•, •), i.e., Assume that the sample size n is sufficiently large, so that the distances d 1 (i, k, n) are small, on the average.Then, based on approximation (8), a reasonable estimator of f ) and thus a reasonable estimator of the entropy In the following theorem, we derive the expression for the asymptotic mean of the estimator Ĝ(1) k,n .Apart from the arguments for the interchange of the limit and the integral signs, the proof is similar to that of Theorem 8 of Singh et al. [12], who in their proof interchange the limit and the integral signs without mentioning the conditions under which it is allowed.Theorem 2.1.Suppose that there exists an > 0, such that and Then, for a fixed k ∈ {1, 2, . . ., } (not depending on n) Then T 1,k,n , . . ., T n,k,n are identically distributed random variables.Therefore, where, for a given θ ∈ [0, 2π) m , S θ,k,n is a random variable having the same distribution as that of the conditional distribution of where A −1 m (•) denotes the inverse function of A m (•).Using standard arguments, we get where N r (•) is defined by (5).For a fixed u ∈ (−∞, ∞), k ∈ {1, 2, . ..} and for almost all values of θ ∈ [0, 2π) m , using Lemma 2.1, we have Therefore, using the Poisson approximation to the binomial distribution, we get for almost all values of θ ∈ [0, 2π) m .For a fixed θ ∈ [0, 2π) m , let S θ,k be a random variable having the pdf Then, in view of (13), where d −→ stands for the convergence in distribution.For each fixed θ ∈ [0, 2π) m , it can be verified that Under the condition (10), it can be shown (for details, see the Appendix) that, for almost all values of θ ∈ [0, 2π) m , there exists a constant C (not depending on n) such that for all sufficiently large values of n Then, in view of ( 14) and the moment convergence theorem, it follows that for almost all values of θ ∈ [0, 2π) m Using Fatou's lemma, we get by (10).Therefore, Now the result follows from ( 12) and ( 16).Since the estimator Ĝ(1) k,n is not asymptotically unbiased, we propose the following (asymptotic) bias corrected estimator for estimating the entropy H(f ): Thus, we have the following corollary to Theorem 2.1.Corollary 2.1.Under the assumptions of Theorem 2.1, the estimator Ĥ(1) k,n is asymptotically unbiased for estimating the entropy H(f ).
The following theorem provides conditions under which the estimator Ĥ(1) k,n is consistent for estimating the entropy H(f ).Theorem 2.2.Suppose that there exists an > 0, such that and Then, for a fixed k ∈ {1, 2, . ..} (not depending on n), and thus Ĥ(1) k,n is a consistent estimator of the entropy H(f ).Under conditions ( 18) and (19), in the proof of Theorem 2.2, the steps involved in justifying the interchange of the limit and the integral sign are tedious but virtually identical to the arguments used in the proof of Theorem 2.1.The remaining part of the proof is identical to the proof of Theorem 11 of Singh et al. [12].We therefore omit the proof of Theorem 2.2.
Remark 2.2.For small values of m ≥ 4, the function A m (•) involved in the evaluation of estimate Ĥ (1) k,n can be computed using numerical integration.For moderate and large values of m, which is the case with many molecules encountered in molecular sciences, using the central limit theorem one can get a reasonable approximation of A m (•) by the cumulative distribution function of a normal distribution having mean m/3 and variance 4m/45.Proof.We have

Nearest Neighbor Estimates of Entropy Based on Circular Distance
where U 1 , . . ., U m are independent and identically distributed uniform random variables over the interval [0, 2π).
Then, D 1 , . . ., D m are independent and identically distributed beta random variables, having parameters α = 1 2 β = 1 2 .Hence the result follows.Remark 3.1.(i) For m = 1 and x ∈ (0, 1), (ii) B m (x) satisfies a similar recursion relation as that satisfied by A m (x) and given in Remark 2.1 (iv).For i ∈ {1, . . ., n}, let d 2 (i, k, n) denote the circular distance of Θ i from its k th closest neighbor with respect to the circular distance d 2 (•, •), defined by (4).Assume that the sample size n is sufficiently large, so that on average the distances d 2 (i, k, n) are small.Then, based on approximation (7), a reasonable estimator of f 2 (i, k, n) and thus a reasonable estimator of the entropy The proof of the following theorem is identical to the proof of Theorem 2.1 and therefore it is omitted.Theorem 3.1.Suppose that there exists an > 0 such that (10) holds and Then, for a fixed k ∈ {1, 2, . . ., } (not depending on n), where Ĝ(2) k,n is defined by (21).Since the estimator Ĝ(2) k,n is not asymptotically unbiased, we propose the following (asymptotic) bias corrected estimator for estimating the entropy H(f ): Thus, we have the following corollary to Theorem 3.1.Corollary 3.1.Under the assumptions of Theorem 3.1, the estimator Ĥ(2) k,n is asymptotically unbiased for estimating the entropy H(f ).
The following theorem provides conditions under which the estimator Ĥ(2) k,n is consistent for estimating the entropy H(f ).The proof of the theorem follows using the arguments similar to the one given for the proof of Theorem 2.2.
(ii) For small values of m ≥ 2, the function B m (•) involved in the evaluation of estimate Ĥ(2) k,n can be computed using numerical integration.For moderate and large values of m, which is the case with many molecules encountered in molecular sciences, using the central limit theorem one can get a reasonable approximation of B m (•) by the cumulative distribution function of a normal distribution having mean m/2 and variance m/8.
With k ∈ {1, 2, . ..}, (17) and ( 23) define two classes of estimators for the entropy H(f ).The biases and variances of these estimators depend on k, the sample size n, and the pdf f (•) and its dimensions m.It would be useful to have the knowledge of the biases and variances as functions of k, n, m, and some characteristic of f (•), such as µ = E(h(f (Θ))), where h(•) is a function such that a reliable estimate of µ can be obtained using the available data on Θ.We have not been able to derive any meaningful expressions for the biases and variances of the proposed estimators, and this problem is under further investigation.

Monte Carlo Results and a Molecular Entropy Example
The performance of an entropy estimator can be investigated rigorously by using Monte Carlo samples from a distribution for which the entropy is known exactly.While analytic distributions of more than two correlated circular variables with exactly calculable entropic attributes do not seem available, one may construct a distribution of higher dimensionality as a product of a suitable number of bivariate distributions.To test the performance of the circular-distance estimator (17), we used an analytic 6-dimensional circular distribution given as the product of three bivariate circular distributions, each of the form [4] f which, as a circular analogue of the bivariate normal distribution, can be called the bivariate von Mises distribution.Details pertaining to the 6-dimensional distribution used and the Monte Carlo sampling are given in [20], where the same circular distribution was used in an investigation of the combined mutual-information-expansion and Euclidean-distance-NN method of entropy estimation.k,n , k = 1, . . ., 5 obtained from samples of sizes in the range n = 5 × 10 4 -1 × 10 7 , together with the corresponding Euclidean-distance estimates Ĥk,n .Figure 1 displays the estimates Ĥ(1) k=1,n and Ĥk=1,n as functions of the sample size n.Noting that the exact entropy value here is, to 4 decimal places, H = 1.8334, we observe that as n increases the circular-distance estimates initially "undershoot" the exact value and then start to approach it slowly from below.In contrast, the Euclidean-distance estimates approach the exact value monotonically from above.Interestingly, the biases of the two kinds of estimates at sample sizes n 1 million are approximately equal in absolute value.The behavior of the circular-distance estimates at k = 2, . . ., 5 is similar to that at k = 1, and the estimate values at different k's become very close at n 1 million.

H ^1, , n
To investigate the usefulness of circular-distance NN estimators in the problem of evaluating the configurational entropy of internal rotations in molecules, we used the circular-distance estimator Ĥ (1) k,n to estimate the entropy of the joint distribution of internal-rotation angles in the molecule of tartaric acid, where the number of variables is m = 7.Samples of size n up to 14.4 million of the internal-rotation angles were obtained from a molecular dynamics simulation of the (R,S) stereoisomer of this molecule [21].Figure 2 shows marginal histograms, smoothed using a Gaussian kernel, of the seven internal-rotation angles of tartaric acid; note that these marginals display markedly non-Gaussian features.The code ANN [22] (with our modification for the circular distance d 1 ), which utilizes a k-d tree algorithm [23], was used for finding the k-th NN distances between sample points.Figure 3 presents the estimates Ĥ(1) k,n , k = 1, ..., 5 as functions of the sample size n.The values of Ĥ(1) k,n decrease as n increases, while, at a fixed value of n 7 million, they increase as k increases; at greater values of n, the estimates at different k's become quite close in value.Figure 4 compares the circular-distance estimates Ĥ(1) 1,n and Ĥ(1) 5,n with the corresponding Euclidean-distance estimates Ĥk,n .We note that an n → ∞ extrapolated Euclidean-distance estimate Ĥ = 5.04 ± 0.01 was obtained for this entropy in [21].Again, as in the case of the analytic circular distribution, this value approximately equals the arithmetic mean of the circular and Euclidean distance estimates at sample sizes n 1 million.
q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q Figure 4. Circular-distance nearest-neighbor estimates Ĥ(1) 1,n and Ĥ(1) 5,n of the internal-rotation entropy of tartaric acid as functions of the sample size n compared with the Euclidean-distance nearest-neighbor estimates Ĥ1,n and Ĥ5,n .An n → ∞ extrapolated estimate is Ĥ = 5.04 ± 0.01 [21].

H ^5, n
Perhaps surprisingly, the results of both the analytic-distribution and molecular-simulation studies undertaken here indicate that only when relatively small data samples are available, the use of a circular-distance estimator has some advantage over the Euclidean-distance estimator.On samples of large size, needed for sufficient convergence of an NN estimate of the entropy of a multivariate distribution, the circular-distance estimates obtained did not have a significantly smaller bias than the Euclidean-distance estimates.In view of such findings, one may question whether the additional computational complexity of a circular-distance estimate is worth the effort.However, we observed that as the sample size increased, the circular NN distances in the sample became quickly so small that the circular-distance estimator Ĥ(1) k,n coincided in value with the simpler Euclidean-distance estimator Ĥk,n in which the same NN-distance values were used.This is explained by the fact that when the circular NN distances d 1 (i, k, n) ≤ π, the estimator Ĥ(1) k,n can be replaced with the estimator Ĥk,n in which the NN distances d 1 (i, k, n) are substituted for the Euclidean NN distances R(i, k, n); this fact follows directly from Remark 2.1 (iii).The only extra computational effort is then expended in finding the circular, instead of Euclidean, NN distances in a given sample.

Figure 1 .
Figure 1.Plots of the circular-and Euclidean-distance estimates Ĥ(1)1,n and Ĥ1,n , respectively, as functions of the sample size n, for the analytic distribution of 6 circular variables; the exact entropy value is (to 4 decimals) H = 1.8334.

Table 1 .
Circular-and Euclidean-distance estimates Ĥ(1)k,n and Ĥk,n , respectively, from samples of size n of the analytic distribution of 6 circular variables; the exact entropy value is (to 4 decimals) H = 1.8334.
1,n and Ĥ1,n , respectively, as functions of the sample size n, for the analytic distribution of 6 circular variables; the exact entropy value is (to 4 decimals) H = 1.8334.