Open Access
This article is

- freely available
- re-usable

*Entropy*
**2010**,
*12*(5),
1125-1144;
doi:10.3390/e12051125

Article

Nearest Neighbor Estimates of Entropy for Multivariate Circular Distributions

^{1}

Department of Statistics, West Virginia University, Morgantown, West Virginia 26506, USA

^{2}

National Institute for Occupational Safety and Health, Morgantown, West Virginia 26505, USA

^{3}

Department of Mathematics and Statistics, Indian Institute of Technology Kanpur, Kanpur 208 016, India

^{}

Deceased.

^{*}

Author to whom correspondence should be addressed.

Received: 26 February 2010 / Accepted: 29 April 2010 / Published: 6 May 2010

## Abstract

**:**

In molecular sciences, the estimation of entropies of molecules is important for the understanding of many chemical and biological processes. Motivated by these applications, we consider the problem of estimating the entropies of circular random vectors and introduce non-parametric estimators based on circular distances between n sample points and their k th nearest neighbors (NN), where k ($\le n-1$) is a fixed positive integer. The proposed NN estimators are based on two different circular distances, and are proven to be asymptotically unbiased and consistent. The performance of one of the circular-distance estimators is investigated and compared with that of the already established Euclidean-distance NN estimator using Monte Carlo samples from an analytic distribution of six circular variables of an exactly known entropy and a large sample of seven internal-rotation angles in the molecule of tartaric acid, obtained by a realistic molecular-dynamics simulation.

Keywords:

circular random variables; differential entropy; non-parametric estimation; nearest neighbor; circular distance; molecular simulation## 1. Introduction

Estimation of entropies of molecules is an important problem in molecular sciences. Internal configurational entropy of a molecule is the entropy of the joint distribution of the internal molecular coordinates (bond lengths, bond angles, and dihedral angles), and as such it is a measure of random fluctuations in these coordinates. Most significant contribution to the internal configurational entropy of a molecule comes from the fluctuations in dihedral angles (also called internal-rotation angles). Many important properties of complex molecules, such as their stability and adopted conformation, depend on random fluctuations in their internal coordinates. Estimation of the internal configurational entropy of molecules is therefore important for understanding many chemical and biological processes, such as the spontaneity of a chemical reaction, protein folding, intermolecular protein-protein interactions, and protein-ligand interactions. It is also a key in the design of drugs that can stabilize the normally folded molecular structure or correct a misfolded structure, since protein misfolding is a cause of several diseases such as Alzheimer disease, mad cow disease, cystic fibrosis, and some types of cancer.

Estimation of the internal entropy of macromolecules, such as proteins, is a challenging problem because of the large number of correlated internal molecular coordinates. A commonly used method of estimating the internal entropy of a molecule, known as the quasi-harmonic approach, is based on the assumption of a multivariate normal distribution for the internal molecular coordinates [1]. Misra et al. [2] discussed the decision theoretic estimation of the entropy of a multivariate normal distribution and obtained improvements over the best affine equivariant estimator under the squared error loss function. However, the assumption of a multivariate normal distribution for the internal coordinates of a molecule is appropriate only at low temperatures, when the fluctuations in its internal coordinates are small. At higher temperatures, the dihedral angles of a complex molecule exhibit multimodes and skewness in their distributions, and the multivariate normal distribution becomes inadequate.

Demchuk and Singh [3] discussed a circular probability approach for modeling the dihedral angles of a molecule in the estimation of internal rotational entropy. As an illustration, they modeled the torsional angle of the methanol molecule by a trimodal von Mises distribution and derived a bath-tub-shaped distribution for the torsional potential energy of the molecule. Singh et al. [4] introduced a torus version of a bivariate normal distribution for modeling two dihedral angles. The marginal distributions of the model are symmetric unimodal or symmetric bimodal depending on the configurations of the parameters. A multivariate generalization of this bivariate model has been proposed by Mardia et al. [5]. Hnizdo et al. [6] and Darian et al. [7] used a Fourier series expansion approach for modeling univariate and bivariate distributions of molecular dihedral angles. Complex molecules, however, have many significantly correlated dihedral angles, whose joint distribution can take an arbitrary form. For this reason, a non-parametric approach for estimating the entropy of a circular random vector of arbitrary dimensions is desirable.

Several non-parametric estimators of the entropy of an m-dimensional random variable X have been discussed in the literature. A common approach is to replace the probability density function (pdf) $f(\xb7)$ in the definition of the differential entropy,
by its non-parametric kernel or histogram density estimator [8,9]. However, in most practical situations, implementation of such estimates in higher dimensions becomes difficult. In one dimension ($m=1$), several authors have proposed estimates of entropy in the context of testing goodness of fit [10,11]. Singh et al. [12] proposed the following asymptotically unbiased and consistent nearest-neighbor (NN) estimator of the entropy $H(f)$:
Here, ${R}_{i,k,n}$ is the Euclidean distance of a point ${X}_{i}$ to its ${k}^{\mathrm{th}}$, $k\le n-1$ nearest (in the Euclidean-distance sense) neighbor in a random sample ${X}_{1},{X}_{2},...,{X}_{n}$ from the distribution $f(\xb7)$; $\gamma =0.5772\cdots $ is Euler’s constant, ${L}_{0}=0$, ${L}_{j}={\sum}_{i=1}^{j}1/i,\phantom{\rule{0.166667em}{0ex}}j=1,2,\cdots $, and $\Gamma (\xb7)$ is the usual gamma function. For $k=1$, the estimator ${\widehat{H}}_{k,n}$ reduces to the NN estimator proposed by Kozachenko and Leonenko [13]. Results similar to those of [12] have been also reported by Goria et al. [14]. For the purpose of the estimation of the information-theoretic quantity of mutual information, Kraskov et al. [15] generalized the first-nearest-neighbor estimator of [13] in terms of ${k}^{\mathrm{th}}$ nearest-neighbor distances in a general metric, giving, however, explicit expressions only for the maximum and Euclidean metrics and without providing formal proofs of asymptotic unbiasedness and consistency. For $m=k=1$, Tsybakov and van der Meulen [16] established the mean-square-root-n consistency of a truncated version of ${\widehat{H}}_{k,n}$. Earlier, Loftsgaarden and Quesenberry [17] had used NN distances to construct non-parametric estimates of a multivariate pdf. Recently, Mnatsakanov et al. [18] studied k-NN estimators of entropy in which the parameter k is assumed to be a function of the sample size.

$$H(f)={E}_{f}(-lnf(X))$$

$${\widehat{H}}_{k,n}=\frac{m}{n}\sum _{i=1}^{n}ln{R}_{i,k,n}+ln\frac{{\pi}^{m/2}}{\Gamma (m/2+1)}+\gamma -{L}_{k-1}+lnn$$

The NN entropy estimator (2) uses Euclidean distances between the sample points. However, when the random variable X is circular, it is natural to base an NN estimate of entropy on a circular distance rather than the Euclidean distance. A circular observation can be regarded as a point on a circle of unit radius. Once an initial direction and an orientation of the circle have been chosen, each circular observation can be specified by the angle from the initial direction to the point on the circle corresponding to the observation. In this paper, we construct estimates of the entropy of an m-dimensional circular random vector $\Theta \in {(0,2\pi ]}^{m}$ based on two different definitions of circular distances. Let $\varphi =({\varphi}_{1},\dots ,{\varphi}_{m})\in {[0,2\pi )}^{m}$ and $\psi =({\psi}_{1},\dots ,{\psi}_{m})\in {[0,2\pi )}^{m}$ be two observations on an m-dimensional circular random vector Θ. We define two circular distance functions ${d}_{1}(\xb7,\xb7)$ and ${d}_{2}(\xb7,\xb7)$ as follows:
and
Note that $\pi -|\pi -|{\varphi}_{i}-{\psi}_{i}\left|\right|,\phantom{\rule{4pt}{0ex}}i=1,\dots ,m$, is the arc length between the points $(cos{\varphi}_{i},sin{\varphi}_{i})$ and $(cos{\psi}_{i},sin{\psi}_{i})$ on the unit circle ${S}_{1}$. On the other hand, ${[2(1-cos({\varphi}_{i}-{\psi}_{i}))]}^{1/2}$ is the Euclidean distance between the points $(cos{\varphi}_{i},sin{\varphi}_{i})$ and $(cos{\psi}_{i},sin{\psi}_{i})$ on the unit circle ${S}_{1}$.

$${d}_{1}(\varphi ,\psi )=\sqrt{\sum _{i=1}^{m}(\pi -|\pi -|{\varphi}_{i}-{\psi}_{i}{\left|\right|)}^{2}}$$

$${d}_{2}(\varphi ,\psi )=\sqrt{2\sum _{i=1}^{m}[1-cos({\varphi}_{i}-{\psi}_{i})]}$$

In Section 2 and Section 3, we propose explicit expressions for NN estimators of entropy based on the circular distance functions ${d}_{1}(\xb7,\xb7)$ and ${d}_{2}(\xb7,\xb7)$, respectively, and prove their asymptotic unbiasedness and consistency (with some mathematical details given in an Appendix). In Section 4, we compare the performance of the estimator based on the circular distance ${d}_{1}$ with that of the Euclidean-distance estimator (2) using Monte Carlo simulations from an analytic 6-dimensional circular distribution, where the exact joint entropy is known. We there also apply the ${d}_{1}$-distance estimator to the problem of estimating the entropy of a 7-dimensional joint distribution of the internal-rotation angles in the molecule of tartaric acid, using a large sample of these angles obtained by a realistic molecular-dynamics simulation.

## 2. Nearest Neighbor Estimates of Entropy Based on Circular Distance ${d}_{1}$

For constructing nearest neighbor estimates of the entropy of a circular random vector based on the distance function (3), we first derive the expression for the volume of a ball
centered at $\psi \in {[0,2\pi )}^{m}$ and having a radius $r\in [0,\sqrt{m}\pi ]$, $\sqrt{m}\pi $ being the maximum value of ${d}_{1}(\xb7,\xb7)$.

$${N}_{r}(\psi )=\{\theta \in {[0,2\pi )}^{m}:{d}_{1}(\theta ,\psi )<r\}$$

**Lemma 2.1.**Let $r\in [0,\sqrt{m}\pi ]$, $\psi \in {[0,2\pi )}^{m}$ and let ${V}_{r}$ be the volume of the ball ${N}_{r}(\psi )$, defined by (5). Then

$${V}_{r}={\left(2\pi \right)}^{m}{A}_{m}\left(\frac{{r}^{2}}{{\pi}^{2}}\right)$$

**Proof.**Without loss of generality, we may take $\psi =(0,\dots ,0)$. Then

$${V}_{r}={\int}_{R}d\theta ={\left(2\pi \right)}^{m}\mathrm{Pr}\left(\sum _{i=1}^{m}{\left(\pi -\mid \pi -{U}_{i}\mid \right)}^{2}<{r}^{2}\right)$$

**Remark 2.1.**(i) For $m=1$ and $x\in [0,1]$, we have ${A}_{1}(x)=\sqrt{x}$. For $m=2$ and $x\in [0,2]$, it can be verified that

$${A}_{2}(x)=\left\{\begin{array}{cc}\frac{\pi x}{4},\hfill & \text{if}\phantom{\rule{4.pt}{0ex}}0\le x\le 1\hfill \\ \sqrt{x-1}+\frac{x}{2}(2arcsin\sqrt{\frac{1}{x}}-\frac{\pi}{2}),\hfill & \text{if}\phantom{\rule{4.pt}{0ex}}1<x\le 2\hfill \end{array}\right.$$

$${A}_{3}(x)=\left\{\begin{array}{cc}\frac{\pi {x}^{3/2}}{6},\hfill & \text{if}\phantom{\rule{4.pt}{0ex}}0\le x\le 1\hfill \\ \frac{\pi}{12}(-3+9x-4{x}^{3/2}),\hfill & \text{if}\phantom{\rule{4.pt}{0ex}}1\le x\le 2\hfill \\ \sqrt{x-2}+\frac{1}{12}\left\{(1-3x)(\pi -4arcsin\frac{1}{\sqrt{x-1}})+(4-12x)arctan\frac{x-3}{2\sqrt{x-2}}\right\}\hfill & \\ +\frac{1}{3}{x}^{3/2}\left\{arctan\sqrt{x(x-2)}+arctan\frac{x-\sqrt{x}-1}{\sqrt{x-2}}-arctan\frac{x+\sqrt{x}-1}{\sqrt{x-2}}\right\},\hfill & \text{if}\phantom{\rule{4.pt}{0ex}}2<x\le 3\hfill \end{array}\right.$$

$${A}_{m}(x)=\frac{{(\pi x)}^{\frac{m}{2}}}{{2}^{m}\Gamma (1+\frac{m}{2})}$$

$${A}_{m}(x)=\left\{\begin{array}{cc}{\int}_{0}^{1}{A}_{m-1}(x-s){f}_{C}(s)ds,\hfill & \text{if}\phantom{\rule{4.pt}{0ex}}1\le x\le m-1\hfill \\ {A}_{1}(x-m+1)+{\int}_{x-m+1}^{1}{A}_{m-1}(x-s){f}_{C}(s)ds,\hfill & \text{if}\phantom{\rule{4.pt}{0ex}}m-1\le x\le m\hfill \end{array}\right.$$

The circular distance (3) becomes the Euclidean distance ${d}_{E}(\varphi ,\psi )={[{\sum}_{i=1}^{m}{({\varphi}_{i}-{\psi}_{i})}^{2}]}^{1/2}$ when $|{\varphi}_{i}-{\psi}_{i}|\le \pi ,\phantom{\rule{0.166667em}{0ex}}i=1,2,\cdots ,m$. Circular-distance balls ${N}_{{r}_{q}}(\theta )$, where $\theta \in {(0,2\pi )}^{m}$ and ${lim}_{q\to \infty}{r}_{q}=0$, thus tend to the corresponding Euclidean-distance balls as $q\to \infty $. We can therefore apply the Lebesgue differentiation theorem [19] to the probability density function $f(\theta )$ of a circular random variable $\Theta \in {[0,2\pi )}^{m}$ in the form
where ${V}_{{r}_{q}}$ is given by (6). Equation (7) suggests that, given a sufficiently large random sample ${\Theta}_{1},\cdots ,{\Theta}_{n}$ from the distribution of Θ, the probability density function $f(\theta )$ can be approximated at almost all $\theta \in {[0,2\pi )}^{m}$ by
where $\mid {N}_{r}(\theta )\mid $ denotes the cardinality of the set $\{i:{\Theta}_{i}\in {N}_{r}(\theta )\}$ and r is sufficiently small.

$$\underset{q\to \infty}{lim}\frac{1}{{V}_{{r}_{q}}}{\int}_{{N}_{{r}_{q}}(\theta )}f(\mu )\phantom{\rule{0.166667em}{0ex}}d\mu =f(\theta ),\phantom{\rule{1.em}{0ex}}\mathrm{at}\phantom{\rule{0.277778em}{0ex}}\mathrm{almost}\phantom{\rule{0.277778em}{0ex}}\mathrm{all}\phantom{\rule{0.277778em}{0ex}}\theta \in {[0,2\pi )}^{m}$$

$${\widehat{f}}_{n}^{(1)}(\theta )=\frac{\mid {N}_{r}(\theta )\mid}{n{V}_{r}}=\frac{\mid {N}_{r}(\theta )\mid}{n{(2\pi )}^{m}{A}_{m}(\frac{{r}^{2}}{{\pi}^{2}})}$$

Guided by this insight, we will now construct nearest neighbor estimates of entropy for an m-variate circular random vector Θ, having a probability density function $f(\xb7)$. Let ${\Theta}_{1},\dots ,{\Theta}_{n}$ be a random sample from the distribution of Θ and let $k\in \{1,\dots ,n-1\}$ be a given positive integer. For $i\in \{1,\dots ,n\}$, let ${d}_{1}(i,k,n)$ denote the circular distance of ${\Theta}_{i}$ from its ${k}^{\mathrm{th}}$ closest neighbor, with respect to the circular distance ${d}_{1}(\xb7,\xb7)$, i.e.,
Assume that the sample size n is sufficiently large, so that the distances ${d}_{1}(i,k,n)$ are small, on the average. Then, based on approximation (8), a reasonable estimator of $f({\Theta}_{i})$ is
and thus a reasonable estimator of the entropy $H(f)=E(-lnf(\Theta ))$ is
In the following theorem, we derive the expression for the asymptotic mean of the estimator ${\widehat{G}}_{k,n}^{(1)}$. Apart from the arguments for the interchange of the limit and the integral signs, the proof is similar to that of Theorem 8 of Singh et al. [12], who in their proof interchange the limit and the integral signs without mentioning the conditions under which it is allowed.

$${d}_{1}(i,k,n)={k}^{\mathrm{th}}\phantom{\rule{4pt}{0ex}}\text{smallest of}\phantom{\rule{4pt}{0ex}}\{{d}_{1}({\Theta}_{j},{\Theta}_{i})\phantom{\rule{4pt}{0ex}}j=1,\dots ,n,\phantom{\rule{4pt}{0ex}}j\ne i\},\phantom{\rule{4pt}{0ex}}i=1,\dots ,n$$

$${\widehat{f}}_{k,n}^{(1)}({\Theta}_{i})=\frac{k}{n{(2\pi )}^{m}{A}_{m}(\frac{{d}_{1}^{2}(i,k,n)}{{\pi}^{2}})}$$

$${\widehat{G}}_{k,n}^{(1)}=-\frac{1}{n}\sum _{i=1}^{n}ln{\widehat{f}}_{k,n}^{(1)}({\Theta}_{i})=\frac{1}{n}\sum _{i=1}^{n}ln\left(\frac{n}{k}\phantom{\rule{0.166667em}{0ex}}{(2\pi )}^{m}{A}_{m}\left(\frac{{d}_{1}^{2}(i,k,n)}{{\pi}^{2}}\right)\right)$$

**Theorem 2.1.**Suppose that there exists an $\u03f5>0$, such that

$${\int}_{{[0,2\pi )}^{m}}\mid lnf(\theta ){\mid}^{1+\u03f5}f(\theta )d\theta <\infty $$

$${\int}_{{[0,2\pi )}^{m}}{\int}_{{[0,2\pi )}^{m}}\mid ln{d}_{1}(\theta ,\mu ){\mid}^{1+\u03f5}f(\theta )f(\mu )d\theta d\mu <\infty $$

$$\underset{n\to \infty}{lim}{E}_{f}\left({\widehat{G}}_{k,n}^{(1)}\right)={L}_{k-1}-\gamma -lnk+H(f)$$

**Proof.**Let

$${T}_{i,k,n}=ln\left(\frac{n}{k}\phantom{\rule{0.166667em}{0ex}}{(2\pi )}^{m}{A}_{m}\left(\frac{{d}_{1}^{2}(i,k,n)}{{\pi}^{2}}\right)\right),\phantom{\rule{4pt}{0ex}}i=1,\dots ,n$$

$${E}_{f}\left({\widehat{G}}_{k,n}^{(1)}\right)={E}_{f}\left({T}_{1,k,n}\right)={\int}_{{[0,2\pi )}^{m}}{E}_{f}\left({S}_{\theta ,k,n}\right)f(\theta )d\theta $$

$${\rho}_{k,n}(u)=\pi \sqrt{{A}_{m}^{-1}\left(\frac{k{e}^{u}}{n{(2\pi )}^{m}}\right)}$$

$$\begin{array}{ccc}\hfill {P}_{f}\left({S}_{\theta ,k,n}\le u\right)& =& 1-{P}_{f}\left({d}_{1}(1,k,n)>{\rho}_{k,n}(u)\mid {\Theta}_{1}=\theta \right)\hfill \\ & =& 1-\sum _{j=0}^{k-1}\left(\genfrac{}{}{0pt}{}{n-1}{j}\right){\left({P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)\right)}^{j}{\left(1-{P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)\right)}^{n-1-j}\hfill \end{array}$$

$$\underset{n\to \infty}{lim}\left(n{P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)\right)=k{e}^{u}\underset{n\to \infty}{lim}\left(\frac{1}{{V}_{{\rho}_{k,n}(u)}}{\int}_{{N}_{{\rho}_{k,n}(u)}(\theta )}f(\mu )d\mu \right)=k{e}^{u}f(\theta )$$

$$\underset{n\to \infty}{lim}{P}_{f}\left({S}_{\theta ,k,n}\le u\right)=1-\sum _{j=0}^{k-1}{e}^{-kf(\theta ){e}^{u}}\frac{{\left(kf(\theta ){e}^{u}\right)}^{j}}{j!}=\frac{1}{\Gamma (k)}{\int}_{0}^{kf(\theta ){e}^{u}}{e}^{-t}{t}^{k-1}dt$$

$${g}_{\theta ,k}(u)=\frac{{e}^{-kf(\theta ){e}^{u}}{\left(kf(\theta ){e}^{u}\right)}^{k}}{\Gamma (k)},\phantom{\rule{4pt}{0ex}}-\infty <u<\infty $$

$${S}_{\theta ,k,n}\phantom{\rule{4pt}{0ex}}\phantom{\rule{4pt}{0ex}}\stackrel{d}{\u27f6}\phantom{\rule{4pt}{0ex}}\phantom{\rule{4pt}{0ex}}{S}_{\theta ,k},\phantom{\rule{4pt}{0ex}}\mathrm{as}\phantom{\rule{4pt}{0ex}}n\to \infty $$

$${E}_{f}\left({S}_{\theta ,k}\right)=\frac{1}{\Gamma (k)}{\int}_{0}^{\infty}(lnt){e}^{-t}{t}^{k-1}dt-lnk-lnf(\theta )={L}_{k-1}-\gamma -lnk-lnf(\theta )$$

$${E}_{f}\left(\mid {S}_{\theta ,k,n}{\mid}^{1+\u03f5}\right)<C$$

$$\underset{n\to \infty}{lim}{E}_{f}\left({S}_{\theta ,k,n}\right)={E}_{f}\left({S}_{\theta ,k}\right)={L}_{k-1}-\gamma -lnk-lnf(\theta )$$

$$\begin{array}{cc}& \underset{n\to \infty}{lim\; sup}{\int}_{{[0,2\pi )}^{m}}\mid {E}_{f}\left({S}_{\theta ,k,n}\right){\mid}^{1+\u03f5}f(\theta )d\theta \hfill \\ & \phantom{\rule{1.em}{0ex}}\phantom{\rule{1.em}{0ex}}\le {\int}_{{[0,2\pi )}^{m}}\underset{n\to \infty}{lim\; sup}\mid {E}_{f}\left({S}_{\theta ,k,n}\right){\mid}^{1+\u03f5}f(\theta )d\theta \hfill \\ & \phantom{\rule{1.em}{0ex}}\phantom{\rule{1.em}{0ex}}={\int}_{{[0,2\pi )}^{m}}\mid {L}_{k-1}-\gamma -lnk-lnf(\theta ){\mid}^{1+\u03f5}f(\theta )d\theta \hfill \\ & \phantom{\rule{1.em}{0ex}}\phantom{\rule{1.em}{0ex}}\le {2}^{\u03f5-1}\left(\mid {L}_{k-1}-\gamma -lnk{\mid}^{1+\u03f5}+{\int}_{{[0,2\pi )}^{m}}\mid lnf(\theta ){\mid}^{1+\u03f5}f(\theta )d\theta \right)<\infty \hfill \end{array}$$

$$\underset{n\to \infty}{lim}{\int}_{{[0,2\pi )}^{m}}{E}_{f}\left({S}_{\theta ,k,n}\right)f(\theta )d\theta ={\int}_{{[0,2\pi )}^{m}}\underset{n\to \infty}{lim}{E}_{f}\left({S}_{\theta ,k,n}\right)f(\theta )d\theta ={\int}_{{[0,2\pi )}^{m}}{E}_{f}\left({S}_{\theta ,k}\right)f(\theta )d\theta $$

Since the estimator ${\widehat{G}}_{k,n}^{(1)}$ is not asymptotically unbiased, we propose the following (asymptotic) bias corrected estimator for estimating the entropy $H(f)$:
Thus, we have the following corollary to Theorem 2.1.

$${\widehat{H}}_{k,n}^{(1)}=\frac{1}{n}\sum _{i=1}^{n}ln\left({A}_{m}\left(\frac{{d}_{1}^{2}(i,k,n)}{{\pi}^{2}}\right)\right)+ln\left(n{(2\pi )}^{m}\right)-{L}_{k-1}+\gamma $$

**Corollary 2.1.**Under the assumptions of Theorem 2.1, the estimator ${\widehat{H}}_{k,n}^{(1)}$ is asymptotically unbiased for estimating the entropy $H(f)$.

The following theorem provides conditions under which the estimator ${\widehat{H}}_{k,n}^{(1)}$ is consistent for estimating the entropy $H(f)$.

**Theorem 2.2.**Suppose that there exists an $\u03f5>0$, such that

$${\int}_{{[0,2\pi )}^{m}}\mid lnf(\theta ){\mid}^{2+\u03f5}f(\theta )d\theta <\infty $$

$${\int}_{{[0,2\pi )}^{m}}{\int}_{{[0,2\pi )}^{m}}\mid ln{d}_{1}(\theta ,\mu ){\mid}^{2+\u03f5}f(\theta )f(\mu )d\theta d\mu <\infty $$

$$\underset{n\to \infty}{lim}{\mathrm{Var}}_{f}\left({\widehat{H}}_{k,n}^{(1)}\right)=0$$

Under conditions (18) and (19), in the proof of Theorem 2.2, the steps involved in justifying the interchange of the limit and the integral sign are tedious but virtually identical to the arguments used in the proof of Theorem 2.1. The remaining part of the proof is identical to the proof of Theorem 11 of Singh et al. [12]. We therefore omit the proof of Theorem 2.2.

**Remark 2.2.**For small values of $m\ge 4$, the function ${A}_{m}(\xb7)$ involved in the evaluation of estimate ${\widehat{H}}_{k,n}^{(1)}$ can be computed using numerical integration. For moderate and large values of m, which is the case with many molecules encountered in molecular sciences, using the central limit theorem one can get a reasonable approximation of ${A}_{m}(\xb7)$ by the cumulative distribution function of a normal distribution having mean $m/3$ and variance $4m/45$.

## 3. Nearest Neighbor Estimates of Entropy Based on Circular Distance ${d}_{2}$

Let $\psi \in {[0,2\pi )}^{m}$. In order to construct nearest neighbor estimates of entropy based on the distance function ${d}_{2}(\xb7,\xb7)$, defined by (4), we require the volume of the ball to be
centered at $\psi \in {[0,2\pi )}^{m}$ and having radius $r\in [0,2\sqrt{m}]$.

$${S}_{r}(\psi )=\{\theta \in {[0,2\pi )}^{m}:{d}_{2}(\theta ,\psi )<r\}$$

**Lemma 3.1.**Let $r\in [0,2\sqrt{m}]$, $\psi \in {[0,2\pi )}^{m}$ and let ${W}_{r}$ be the volume of the ball ${S}_{r}(\psi )$, defined by (20). Then

$${W}_{r}={(2\pi )}^{m}{B}_{m}\left(\frac{{r}^{2}}{4}\right)$$

**Proof.**We have

$${W}_{r}={\int}_{\{\theta \in {[0,2\pi )}^{m},\phantom{\rule{3.33333pt}{0ex}}{\sum}_{i=1}^{m}(1-cos({\theta}_{i}))<\frac{{r}^{2}}{2}\}}d\theta ={(2\pi )}^{m}\mathrm{Pr}\left(\sum _{i=1}^{m}\frac{1-cos{U}_{i}}{2}<\frac{{r}^{2}}{4}\right)$$

Define ${D}_{i}=(1-cos{U}_{i})/2,\phantom{\rule{4pt}{0ex}}i=1,\dots ,m$. Then, ${D}_{1},\dots ,{D}_{m}$ are independent and identically distributed beta random variables, having parameters $\alpha =\frac{1}{2}$ $\beta =\frac{1}{2}$. Hence the result follows.

**Remark 3.1.**(i) For $m=1$ and $x\in (0,1)$,

$${B}_{1}(x)=\frac{2}{\pi}arcsin(\sqrt{x})=\frac{1}{\pi}\phantom{\rule{0.166667em}{0ex}}arccos(1-2x)$$

For $i\in \{1,\dots ,n\}$, let ${d}_{2}(i,k,n)$ denote the circular distance of ${\Theta}_{i}$ from its ${k}^{\mathrm{th}}$ closest neighbor with respect to the circular distance ${d}_{2}(\xb7,\xb7)$, defined by (4). Assume that the sample size n is sufficiently large, so that on average the distances ${d}_{2}(i,k,n)$ are small. Then, based on approximation (7), a reasonable estimator of $f({\Theta}_{i})$ is
and thus a reasonable estimator of the entropy $H(f)=E(-lnf(\Theta ))$ is
The proof of the following theorem is identical to the proof of Theorem 2.1 and therefore it is omitted.

$${\widehat{f}}_{k,n}^{(2)}({\Theta}_{i})=\frac{k}{n{(2\pi )}^{m}{B}_{m}\left({\textstyle \frac{1}{4}}\phantom{\rule{0.166667em}{0ex}}{d}_{2}^{2}(i,k,n)\right)}$$

$${\widehat{G}}_{k,n}^{(2)}=-\frac{1}{n}\sum _{i=1}^{n}ln{\widehat{f}}_{k,n}^{(2)}({\Theta}_{i})=\frac{1}{n}\sum _{i=1}^{n}ln\left(\frac{n{(2\pi )}^{m}}{k}{B}_{m}\left({\textstyle \frac{1}{4}}\phantom{\rule{0.166667em}{0ex}}{d}_{2}^{2}(i,k,n)\right)\right)$$

**Theorem 3.1.**Suppose that there exists an $\u03f5>0$ such that (10) holds and

$${\int}_{{[0,2\pi )}^{m}}{\int}_{{[0,2\pi )}^{m}}\mid ln{d}_{2}(\theta ,\mu ){\mid}^{1+\u03f5}f(\theta )f(\mu )d\theta d\mu <\infty $$

$$\underset{n\to \infty}{lim}{E}_{f}\left({\widehat{G}}_{k,n}^{(2)}\right)={L}_{k-1}-\gamma -lnk+H(f)$$

Since the estimator ${\widehat{G}}_{k,n}^{(2)}$ is not asymptotically unbiased, we propose the following (asymptotic) bias corrected estimator for estimating the entropy $H(f)$:
Thus, we have the following corollary to Theorem 3.1.

$${\widehat{H}}_{k,n}^{(2)}=\frac{1}{n}\sum _{i=1}^{n}ln\left({B}_{m}\left({\textstyle \frac{1}{4}}\phantom{\rule{0.166667em}{0ex}}{d}_{2}^{2}(i,k,n)\right)\right)+ln\left(n{(2\pi )}^{m}\right)-{L}_{k-1}+\gamma $$

**Corollary 3.1.**Under the assumptions of Theorem 3.1, the estimator ${\widehat{H}}_{k,n}^{(2)}$ is asymptotically unbiased for estimating the entropy $H(f)$.

The following theorem provides conditions under which the estimator ${\widehat{H}}_{k,n}^{(2)}$ is consistent for estimating the entropy $H(f)$. The proof of the theorem follows using the arguments similar to the one given for the proof of Theorem 2.2.

**Theorem 3.2.**Suppose that there exists an $\u03f5>0$ such that (18) holds and

$${\int}_{{[0,2\pi )}^{m}}{\int}_{{[0,2\pi )}^{m}}\mid ln{d}_{2}(\theta ,\mu ){\mid}^{2+\u03f5}f(\theta )f(\mu )d\theta d\mu <\infty $$

$$\underset{n\to \infty}{lim}{\mathrm{Var}}_{f}\left({\widehat{H}}_{k,n}^{(2)}\right)=0$$

**Remark 3.2.**(i) Using Remark 3.1 (i) and the fact that $arccos(x)\in [0,\pi ]$ is a decreasing function of $x\in [-1,1]$, for $m=1$ and $i\in \{1,\dots ,n\}$, we have

$$\begin{array}{ccc}\hfill {B}_{1}\left({\textstyle \frac{1}{4}}\phantom{\rule{0.166667em}{0ex}}{d}_{2}^{2}(i,k,n)\right)& =& \frac{1}{\pi}arccos\left(1-\frac{{d}_{2}^{2}(i,k,n)}{2}\right)\hfill \\ & =& \frac{1}{\pi}\times {k}^{\mathrm{th}}\phantom{\rule{4pt}{0ex}}\text{smallest of}\phantom{\rule{4pt}{0ex}}\left\{arccos\left(cos\left({\Theta}_{j}-{\Theta}_{i}\right)\right),\phantom{\rule{4pt}{0ex}}j=1,\dots ,n,\phantom{\rule{4pt}{0ex}}j\ne i\right\}\hfill \\ & =& \frac{1}{\pi}\times {k}^{\mathrm{th}}\phantom{\rule{4pt}{0ex}}\text{smallest of}\phantom{\rule{4pt}{0ex}}\left\{\pi -\mid \pi -\mid {\Theta}_{j}-{\Theta}_{i}\mid \mid ,\phantom{\rule{4pt}{0ex}}j=1,\dots ,n,\phantom{\rule{4pt}{0ex}}j\ne i\right\}\hfill \end{array}$$

Therefore, for $m=1$ and $i\in \{1,\dots ,n\}$, we have
Thus for $m=1$, estimators ${\widehat{H}}_{k,n}^{(1)}$ and ${\widehat{H}}_{k,n}^{(2)}$, based on circular distance functions ${d}_{1}(\xb7,\xb7)$ and ${d}_{2}(\xb7,\xb7)$ respectively, are identical.

$${B}_{1}\left({\textstyle \frac{1}{4}}\phantom{\rule{0.166667em}{0ex}}{d}_{2}^{2}(i,k,n)\right)=\frac{{d}_{1}(i,k,n)}{\pi}\phantom{\rule{4pt}{0ex}}\Rightarrow \phantom{\rule{4pt}{0ex}}{\widehat{G}}_{k,n}^{(1)}={\widehat{G}}_{k,n}^{(2)}\phantom{\rule{4pt}{0ex}}\Rightarrow \phantom{\rule{4pt}{0ex}}{\widehat{H}}_{k,n}^{(1)}={\widehat{H}}_{k,n}^{(2)}$$

(ii) For small values of $m\ge 2$, the function ${B}_{m}(\xb7)$ involved in the evaluation of estimate ${\widehat{H}}_{k,n}^{(2)}$ can be computed using numerical integration. For moderate and large values of m, which is the case with many molecules encountered in molecular sciences, using the central limit theorem one can get a reasonable approximation of ${B}_{m}(\xb7)$ by the cumulative distribution function of a normal distribution having mean $m/2$ and variance $m/8$.

With $k\in \{1,2,\dots \}$, (17) and (23) define two classes of estimators for the entropy $H(f)$. The biases and variances of these estimators depend on k, the sample size n, and the pdf $f(\xb7)$ and its dimensions m. It would be useful to have the knowledge of the biases and variances as functions of k, n, m, and some characteristic of $f(\xb7)$, such as $\mu =E(h(f(\Theta )))$, where $h(\xb7)$ is a function such that a reliable estimate of μ can be obtained using the available data on Θ. We have not been able to derive any meaningful expressions for the biases and variances of the proposed estimators, and this problem is under further investigation.

## 4. Monte Carlo Results and a Molecular Entropy Example

The performance of an entropy estimator can be investigated rigorously by using Monte Carlo samples from a distribution for which the entropy is known exactly. While analytic distributions of more than two correlated circular variables with exactly calculable entropic attributes do not seem available, one may construct a distribution of higher dimensionality as a product of a suitable number of bivariate distributions. To test the performance of the circular-distance estimator (17), we used an analytic 6-dimensional circular distribution given as the product of three bivariate circular distributions, each of the form [4]
which, as a circular analogue of the bivariate normal distribution, can be called the bivariate von Mises distribution. Details pertaining to the 6-dimensional distribution used and the Monte Carlo sampling are given in [20], where the same circular distribution was used in an investigation of the combined mutual-information-expansion and Euclidean-distance-NN method of entropy estimation.

$$f({\theta}_{1},{\theta}_{2})=C{e}^{{\kappa}_{1}cos({\theta}_{1}-{\mu}_{1})+{\kappa}_{2}cos({\theta}_{2}-{\mu}_{2})+\lambda sin({\theta}_{1}-{\mu}_{1})sin({\theta}_{2}-{\mu}_{2})}$$

**Table 1.**Circular- and Euclidean-distance estimates ${\widehat{H}}_{k,n}^{(1)}$ and ${\widehat{H}}_{k,n}$, respectively, from samples of size n of the analytic distribution of 6 circular variables; the exact entropy value is (to 4 decimals) $H=1.8334$.

$n\times {10}^{-6}$ | ${\widehat{H}}_{1,n}^{(1)}/{\widehat{H}}_{1,n}$ | ${\widehat{H}}_{2,n}^{(1)}/{\widehat{H}}_{2,n}$ | ${\widehat{H}}_{3,n}^{(1)}/{\widehat{H}}_{3,n}$ | ${\widehat{H}}_{4,n}^{(1)}/{\widehat{H}}_{4,n}$ | ${\widehat{H}}_{5,n}^{(1)}/{\widehat{H}}_{5,n}$ |
---|---|---|---|---|---|

0.05 | 1.86851 | 1.92019 | 1.96379 | 2.00395 | 2.03653 |

2.01225 | 2.09929 | 2.16540 | 2.22380 | 2.27390 | |

0.10 | 1.81503 | 1.86968 | 1.90481 | 1.92899 | 1.95164 |

1.95640 | 2.01952 | 2.07339 | 2.11255 | 2.14865 | |

0.20 | 1.80070 | 1.83713 | 1.85672 | 1.87498 | 1.89122 |

1.92305 | 1.96691 | 2.00110 | 2.03086 | 2.05650 | |

0.40 | 1.8007 | 1.81582 | 1.82840 | 1.83904 | 1.84862 |

1.89363 | 1.92652 | 1.95001 | 1.96939 | 1.98668 | |

0.60 | 1.80066 | 1.80693 | 1.81496 | 1.82296 | 1.83097 |

1.88696 | 1.90952 | 1.92764 | 1.94358 | 1.95783 | |

0.80 | 1.79837 | 1.80297 | 1.81063 | 1.81660 | 1.82222 |

1.87936 | 1.89971 | 1.91649 | 1.92946 | 1.94125 | |

1.00 | 1.79539 | 1.80017 | 1.80566 | 1.81030 | 1.81566 |

1.87317 | 1.89238 | 1.90694 | 1.91850 | 1.92915 | |

2.00 | 1.79660 | 1.79533 | 1.79736 | 1.79970 | 1.80266 |

1.86471 | 1.87555 | 1.88493 | 1.89298 | 1.90073 | |

4.00 | 1.79673 | 1.79383 | 1.79404 | 1.79480 | 1.79613 |

1.85696 | 1.86477 | 1.87136 | 1.87702 | 1.88211 | |

6.00 | 1.79795 | 1.79491 | 1.79385 | 1.79419 | 1.79458 |

1.85403 | 1.86071 | 1.86552 | 1.87029 | 1.87414 | |

8.00 | 1.79893 | 1.79484 | 1.79373 | 1.79322 | 1.79337 |

1.85240 | 1.85745 | 1.86197 | 1.86545 | 1.86891 | |

10.00 | 1.80036 | 1.79562 | 1.79426 | 1.79350 | 1.79329 |

1.85170 | 1.85578 | 1.85969 | 1.86287 | 1.86583 |

Table 1 presents the circular-distance estimates ${\widehat{H}}_{k,n}^{(1)}$, $k=1,\cdots ,5$ obtained from samples of sizes in the range $n=5\times {10}^{4}$–$1\times {10}^{7}$, together with the corresponding Euclidean-distance estimates ${\widehat{H}}_{k,n}$. Figure 1 displays the estimates ${\widehat{H}}_{k=1,n}^{(1)}$ and ${\widehat{H}}_{k=1,n}$ as functions of the sample size n. Noting that the exact entropy value here is, to 4 decimal places, $H=1.8334$, we observe that as n increases the circular-distance estimates initially “undershoot” the exact value and then start to approach it slowly from below. In contrast, the Euclidean-distance estimates approach the exact value monotonically from above. Interestingly, the biases of the two kinds of estimates at sample sizes $n\gtrsim 1$ million are approximately equal in absolute value. The behavior of the circular-distance estimates at $k=2,\cdots ,5$ is similar to that at $k=1$, and the estimate values at different k’s become very close at $n\gtrsim 1$ million.

**Figure 1.**Plots of the circular- and Euclidean-distance estimates ${\widehat{H}}_{1,n}^{(1)}$ and ${\widehat{H}}_{1,n}$, respectively, as functions of the sample size n, for the analytic distribution of 6 circular variables; the exact entropy value is (to 4 decimals) $H=1.8334$.

To investigate the usefulness of circular-distance NN estimators in the problem of evaluating the configurational entropy of internal rotations in molecules, we used the circular-distance estimator ${\widehat{H}}_{k,n}^{(1)}$ to estimate the entropy of the joint distribution of internal-rotation angles in the molecule of tartaric acid, where the number of variables is $m=7$. Samples of size n up to 14.4 million of the internal-rotation angles were obtained from a molecular dynamics simulation of the (R,S) stereoisomer of this molecule [21]. Figure 2 shows marginal histograms, smoothed using a Gaussian kernel, of the seven internal-rotation angles of tartaric acid; note that these marginals display markedly non-Gaussian features. The code ANN [22] (with our modification for the circular distance ${d}_{1}$), which utilizes a k-d tree algorithm [23], was used for finding the k-th NN distances between sample points. Figure 3 presents the estimates ${\widehat{H}}_{k,n}^{(1)}$, $k=1,...,5$ as functions of the sample size n. The values of ${\widehat{H}}_{k,n}^{(1)}$ decrease as n increases, while, at a fixed value of $n\lesssim 7$ million, they increase as k increases; at greater values of n, the estimates at different k’s become quite close in value. Figure 4 compares the circular-distance estimates ${\widehat{H}}_{1,n}^{(1)}$ and ${\widehat{H}}_{5,n}^{(1)}$ with the corresponding Euclidean-distance estimates ${\widehat{H}}_{k,n}$. We note that an $n\to \infty $ extrapolated Euclidean-distance estimate $\widehat{H}=5.04\pm 0.01$ was obtained for this entropy in [21]. Again, as in the case of the analytic circular distribution, this value approximately equals the arithmetic mean of the circular and Euclidean distance estimates at sample sizes $n\gtrsim 1$ million.

**Figure 2.**Smoothed marginal histograms of the internal-rotation angles ${\varphi}_{i}$, $i=1,\cdots ,7$ of the (R,S) isomer of tartaric acid obtained by molecular dynamics simulations.

**Figure 3.**Circular-distance nearest-neighbor estimates ${\widehat{H}}_{k,n}^{(1)}$, $k=1,\cdots ,5$ of the entropy of the 7-dimensional joint distribution of internal-rotation angles in the (R,S) isomer of tartaric acid as functions of the sample size n. The estimates ${\widehat{H}}_{k,n}^{(1)}$ at a fixed $n\lesssim 7$ million increase in value as k increases.

**Figure 4.**Circular-distance nearest-neighbor estimates ${\widehat{H}}_{1,n}^{(1)}$ and ${\widehat{H}}_{5,n}^{(1)}$ of the internal-rotation entropy of tartaric acid as functions of the sample size n compared with the Euclidean-distance nearest-neighbor estimates ${\widehat{H}}_{1,n}$ and ${\widehat{H}}_{5,n}$. An $n\to \infty $ extrapolated estimate is $\widehat{H}=5.04\pm 0.01$ [21].

Perhaps surprisingly, the results of both the analytic-distribution and molecular-simulation studies undertaken here indicate that only when relatively small data samples are available, the use of a circular-distance estimator has some advantage over the Euclidean-distance estimator. On samples of large size, needed for sufficient convergence of an NN estimate of the entropy of a multivariate distribution, the circular-distance estimates obtained did not have a significantly smaller bias than the Euclidean-distance estimates. In view of such findings, one may question whether the additional computational complexity of a circular-distance estimate is worth the effort. However, we observed that as the sample size increased, the circular NN distances in the sample became quickly so small that the circular-distance estimator ${\widehat{H}}_{k,n}^{(1)}$ coincided in value with the simpler Euclidean-distance estimator ${\widehat{H}}_{k,n}$ in which the same NN-distance values were used. This is explained by the fact that when the circular NN distances ${d}_{1}(i,k,n)\le \pi $, the estimator ${\widehat{H}}_{k,n}^{(1)}$ can be replaced with the estimator ${\widehat{H}}_{k,n}$ in which the NN distances ${d}_{1}(i,k,n)$ are substituted for the Euclidean NN distances $R(i,k,n)$; this fact follows directly from Remark 2.1 (iii). The only extra computational effort is then expended in finding the circular, instead of Euclidean, NN distances in a given sample.

## Acknowledgments

The authors are thankful to Jun Tan for carrying out the circular-distance NN calculations, and to E. James Harner, Cecil Burchfiel, Robert Mnatsakanov and Dan S. Sharp for helpful discussions. The findings and conclusions in this paper are those of the authors and do not necessarily represent the views of the National Institute for Occupational Safety and Health.

## Appendix

Here we provide the proof of (15). For a fixed $\theta \in {[0,2\pi )}^{m}$, let ${F}_{\theta ,k,n}(\xb7)$ be the distribution function of the random variable ${S}_{\theta ,k,n}$. To establish (15), we will first show that, for $n=2$ and $k=1$,
for almost all values of $\theta \in {[0,2\pi )}^{m}$. In order to establish (25), consider
Thus, to establish (25), it is enough to show that the second term in the above expression is finite. Note that, for $x\in [0,m]$,
and thus
In view of assumption (11), it follows that
for almost all values of $\theta \in {[0,2\pi )}^{m}$. Therefore, (25) is established.

$${E}_{f}\left(\mid {S}_{\theta ,1,2}{\mid}^{1+\u03f5}\right)={\int}_{-\infty}^{\infty}\mid u{\mid}^{1+\u03f5}d{F}_{\theta ,1,2}(u)<\infty $$

$$\begin{array}{ccc}\hfill {E}_{f}\left(\mid {S}_{\theta ,1,2}{\mid}^{1+\u03f5}\right)& =& {E}_{f}\left({\left|ln\left[2{(2\pi )}^{m}{A}_{m}\left({d}_{1}^{2}(1,1,2)/{\pi}^{2}\right)\right]\right|}^{1+\u03f5}|{\Theta}_{1}=\theta \right)\hfill \\ & =& {E}_{f}\left({\left|ln\left[2{(2\pi )}^{m}\right]+ln\left[{A}_{m}\left({d}_{1}^{2}(\theta ,{\Theta}_{2})/{\pi}^{2}\right)\right]\right|}^{1+\u03f5}\right)\hfill \\ & \le & {2}^{\u03f5}\left(\mid ln\left[2{(2\pi )}^{m}\right]{\mid}^{1+\u03f5}+{E}_{f}\left({\left|ln\left[{A}_{m}\left({d}_{1}^{2}(\theta ,{\Theta}_{2})/{\pi}^{2}\right)\right]\right|}^{1+\u03f5}\right)\right)\hfill \end{array}$$

$${A}_{m}(x)=\mathrm{Pr}\left(\sum _{i=1}^{m}{C}_{i}\le x\right)\ge \prod _{i=1}^{m}\mathrm{Pr}\left({C}_{i}\le \frac{x}{m}\right)={\left(\frac{x}{m}\right)}^{m/2}$$

$$\begin{array}{cc}\hfill {E}_{f}\left({\left|ln\left[{A}_{m}\left({d}_{1}^{2}(\theta ,{\Theta}_{2})/{\pi}^{2}\right)\right]\right|}^{1+\u03f5}\right)& \le {m}^{1+\u03f5}{E}_{f}\left({\left|ln[{d}_{1}(\theta ,{\Theta}_{2})/\pi \sqrt{m}]\right|}^{1+\u03f5}\right)\hfill \\ & \le {m}^{1+\u03f5}{2}^{\u03f5}\left({E}_{f}\left({\left|ln[{d}_{1}(\theta ,{\Theta}_{2})]\right|}^{1+\u03f5}\right)+{\left|ln(\pi \sqrt{m})\right|}^{1+\u03f5}\right)\hfill \\ & ={m}^{1+\u03f5}{2}^{\u03f5}\left({\int}_{{[0,2\pi )}^{m}}{\left|ln[{d}_{1}(\theta ,\mu )]\right|}^{1+\u03f5}f(\mu )d\mu +{\left|ln(\pi \sqrt{m})\right|}^{1+\u03f5}\right)\hfill \end{array}$$

$${\int}_{{[0,2\pi )}^{m}}\mid ln{d}_{1}(\theta ,\mu ){\mid}^{1+\u03f5}f(\mu )d\mu <\infty $$

Now we will establish (15). Consider
We can write
We have,
For $j\in \{0,1,\dots ,k-1\}$, we have $\left(\genfrac{}{}{0pt}{}{n-1}{j}\right)\le \frac{n-1}{n-k}\left(\genfrac{}{}{0pt}{}{n-2}{j}\right)$ Therefore,
where ${B}_{a,b}$ denotes the beta random variable with parameter $(a,b)$, $a>0,b>0$. For $u>ln\sqrt{n}$, we have ${P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)\ge {P}_{f}\left({N}_{{\rho}_{k,n}(ln\sqrt{n})}(\theta )\right)$ Therefore, (28) yields
Note that ${lim}_{n\to \infty}{\rho}_{k,n}(ln\sqrt{n})=0$. Thus, on using (7), we get
for almost all values of $\theta \in {[0,2\pi )}^{m}$.

$${E}_{f}\left(\mid {S}_{\theta ,k,n}{\mid}^{1+\u03f5}\right)={\int}_{-\infty}^{0}\mid u{\mid}^{1+\u03f5}d{F}_{\theta ,k,n}(u)+{\int}_{0}^{\infty}\mid u{\mid}^{1+\u03f5}d{F}_{\theta ,k,n}(u)$$

$$\begin{array}{ccc}\hfill {\int}_{0}^{\infty}\mid u{\mid}^{1+\u03f5}d{F}_{\theta ,k,n}(u)& =& (1+\u03f5)\left({\int}_{0}^{ln\sqrt{n}}{u}^{\u03f5}\left(1-{F}_{\theta ,k,n}(u)\right)du+{\int}_{ln\sqrt{n}}^{\infty}{u}^{\u03f5}\left(1-{F}_{\theta ,k,n}(u)\right)du\right)\hfill \\ & =& (1+\u03f5)\left({I}_{1}(n)+{I}_{2}(n)\right),\phantom{\rule{4pt}{0ex}}\mathrm{say}.\hfill \end{array}$$

$${I}_{2}(n)={\int}_{ln\sqrt{n}}^{\infty}{u}^{\u03f5}\left(\sum _{j=0}^{k-1}\left(\genfrac{}{}{0pt}{}{n-1}{j}\right){\left({P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)\right)}^{j}{\left(1-{P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)\right)}^{n-1-j}\right)du$$

$$\begin{array}{ccc}\hfill {I}_{2}(n)& \le & \frac{n-1}{n-k}{\int}_{ln\sqrt{n}}^{\infty}{u}^{\u03f5}\left(1-{P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)\right)\hfill \\ & & \phantom{\rule{1.em}{0ex}}\times \left(\sum _{j=0}^{k-1}\left(\genfrac{}{}{0pt}{}{n-2}{j}\right){\left({P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)\right)}^{j}{\left(1-{P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)\right)}^{n-2-j}\right)du\hfill \\ & =& \frac{n-1}{n-k}{\int}_{ln\sqrt{n}}^{\infty}{u}^{\u03f5}\left(1-{P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)\right)P\left({B}_{k,n-k-1}\ge {P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)\right)du\hfill \end{array}$$

$${I}_{2}(n)\le \frac{n-1}{n-k}P\left({B}_{k,n-k-1}\ge {P}_{f}\left({N}_{{\rho}_{k,n}(ln\sqrt{n})}(\theta )\right)\right){\int}_{ln\sqrt{n}}^{\infty}{u}^{\u03f5}\left(1-{P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)\right)du$$

$$\underset{n\to \infty}{lim}\left(\frac{\sqrt{n}}{k}{P}_{f}\left({N}_{{\rho}_{k,n}(ln\sqrt{n})}(\theta )\right)\right)=f(\theta )$$

For a $\theta \in {[0,2\pi )}^{m}$, for which $f(\theta )>0$, choose $\delta \in (0,f(\theta ))$. Then, for sufficiently large values of n,
and therefore, for sufficiently large values of n,
Therefore, for sufficiently large values of n and for almost all values of $\theta \in {[0,2\pi )}^{m}$, (29) yields
On making the change of variable $z=ln(2k/n)+u$ in the integral in (30), we get
We also have
and
where ${C}_{\u03f5}=max(1,{2}^{\u03f5-1})$.

$${P}_{f}\left({N}_{{\rho}_{k,n}(ln\sqrt{n})}(\theta )\right)>\frac{k}{\sqrt{n}}(f(\theta )-\delta )$$

$$\begin{array}{ccc}& & P\left({B}_{k,n-k-1}\ge {P}_{f}\left({N}_{{\rho}_{k,n}(ln\sqrt{n})}(\theta )\right)\right)\le P\left({B}_{k,n-k-1}\ge \frac{k}{\sqrt{n}}(f(\theta )-\delta )\right)\le \frac{E\left({B}_{k,n-k-1}^{2}\right)}{{\left(\frac{k}{\sqrt{n}}(f(\theta )-\delta )\right)}^{2}}\hfill \\ & & =\frac{k+1}{(n-1)k{(f(\theta )-\delta )}^{2}}\hfill \end{array}$$

$${I}_{2}(n)\le \frac{k+1}{k(n-k){(f(\theta )-\delta )}^{2}}{\int}_{ln\sqrt{n}}^{\infty}{u}^{\u03f5}\left(1-{P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)\right)du$$

$$\begin{array}{ccc}& & {\int}_{ln\sqrt{n}}^{\infty}{u}^{\u03f5}\left(1-{P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)\right)du={\int}_{ln\frac{2k}{\sqrt{n}}}^{\infty}{\left(u+ln\frac{n}{2k}\right)}^{\u03f5}\left(1-{P}_{f}\left({N}_{{\rho}_{k,n}(u+ln(\frac{n}{2k}))}(\theta )\right)\right)du\hfill \\ & & ={\int}_{ln\frac{2k}{\sqrt{n}}}^{\infty}{\left(u+ln\frac{n}{2k}\right)}^{\u03f5}\left(1-{P}_{f}\left({N}_{{\rho}_{1,2}(u)}(\theta )\right)\right)du={\int}_{ln\frac{2k}{\sqrt{n}}}^{0}{\left(u+ln\frac{n}{2k}\right)}^{\u03f5}\left(1-{P}_{f}\left({N}_{{\rho}_{1,2}(u)}(\theta )\right)\right)du\hfill \\ & & \phantom{\rule{1.em}{0ex}}+{\int}_{0}^{\infty}{\left(u+ln\frac{n}{2k}\right)}^{\u03f5}\left(1-{P}_{f}\left({N}_{{\rho}_{1,2}(u)}(\theta )\right)\right)du\hfill \end{array}$$

$$\begin{array}{ccc}& & {\int}_{ln\frac{2k}{\sqrt{n}}}^{0}{\left(u+ln\frac{n}{2k}\right)}^{\u03f5}\left(1-{P}_{f}\left({N}_{{\rho}_{1,2}(u)}(\theta )\right)\right)du\le {\left(ln\frac{n}{2k}\right)}^{\u03f5}{\int}_{ln\frac{2k}{\sqrt{n}}}^{0}\left(1-{P}_{f}\left({N}_{{\rho}_{1,2}(u)}(\theta )\right)\right)du\hfill \\ & & ={\left(ln\frac{n}{2k}\right)}^{\u03f5}{\int}_{\frac{2k}{\sqrt{n}}}^{1}\frac{1}{u}\left(1-{P}_{f}\left({N}_{{\rho}_{1,2}(lnu)}(\theta )\right)\right)du\le \frac{\sqrt{n}}{2k}{\left(ln\frac{n}{2k}\right)}^{\u03f5}{\int}_{\frac{2k}{\sqrt{n}}}^{1}\left(1-{P}_{f}\left({N}_{{\rho}_{1,2}(lnu)}(\theta )\right)\right)du\hfill \\ & & \le \frac{\sqrt{n}}{2k}{\left(ln\frac{n}{2k}\right)}^{\u03f5}\hfill \end{array}$$

$$\begin{array}{ccc}\hfill {\int}_{0}^{\infty}{\left(u+ln\frac{n}{2k}\right)}^{\u03f5}\left(1-{P}_{f}\left({N}_{{\rho}_{1,2}(u)}(\theta )\right)\right)du& \le & {C}_{\u03f5}\left({\left(ln\frac{n}{2k}\right)}^{\u03f5}{\int}_{0}^{\infty}\left(1-{P}_{f}\left({N}_{{\rho}_{1,2}(u)}(\theta )\right)\right)du\right.\hfill \\ & & \phantom{\rule{1.em}{0ex}}\left.+{\int}_{0}^{\infty}{u}^{\u03f5}\left(1-{P}_{f}\left({N}_{{\rho}_{1,2}(u)}(\theta )\right)\right)du\right)\hfill \end{array}$$

Note that, for $u\in (-\infty ,\infty )$,
Therefore (33) yields
In view of (25), we have
Therefore, using (30)-(35), we conclude that
for almost all values of θ.

$$1-{P}_{f}\left({N}_{{\rho}_{1,2}(u)}(\theta )\right)=1-{P}_{f}\left({S}_{\theta ,1,2}\le u\right)=1-{F}_{\theta ,1,2}(u)$$

$$\begin{array}{ccc}\hfill {\int}_{0}^{\infty}{\left(u+ln\frac{n}{2k}\right)}^{\u03f5}\left(1-{P}_{f}\left({N}_{{\rho}_{1,2}(u)}(\theta )\right)\right)du& \le & {C}_{\u03f5}\left({\left(ln\frac{n}{2k}\right)}^{\u03f5}{\int}_{0}^{\infty}\left(1-{F}_{\theta ,1,2}(u)\right)du\right.\hfill \\ & & \phantom{\rule{1.em}{0ex}}\left.+{\int}_{0}^{\infty}{u}^{\u03f5}\left(1-{F}_{\theta ,1,2}(u)\right)du\right)\hfill \end{array}$$

$${\int}_{0}^{\infty}\left(1-{F}_{\theta ,1,2}(u)\right)du<\infty \phantom{\rule{4pt}{0ex}}\phantom{\rule{4pt}{0ex}}\mathrm{and}\phantom{\rule{4pt}{0ex}}\phantom{\rule{4pt}{0ex}}{\int}_{0}^{\infty}{u}^{\u03f5}\left(1-{F}_{\theta ,1,2}(u)\right)du<\infty $$

$$\underset{n\to \infty}{lim}{I}_{2}(n)=0$$

Now consider
For $u<ln\sqrt{n}$, we have
Therefore, for $u\in (-\infty ,\infty )$ and for almost all values of $\theta \in {[0,2\pi )}^{m}$, we have
uniformly in u.

$$\begin{array}{ccc}\hfill {I}_{1}(n)& =& {\int}_{0}^{ln\sqrt{n}}{u}^{\u03f5}\left(1-{F}_{\theta ,k,n}(u)\right)du\hfill \\ & =& {\int}_{0}^{ln\sqrt{n}}{u}^{\u03f5}\left(\sum _{j=0}^{k-1}\left(\genfrac{}{}{0pt}{}{n-1}{j}\right){\left({P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)\right)}^{j}{\left(1-{P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)\right)}^{n-1-j}\right)du\hfill \end{array}$$

$$\begin{array}{ccc}& =& {\int}_{0}^{ln\sqrt{n}}{u}^{\u03f5}P\left({B}_{k,n-k}\ge {P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)\right)du\hfill \end{array}$$

$$0\le {\rho}_{k,n}(u)\le \pi \sqrt{{A}_{m}^{-1}\left(\frac{k}{\sqrt{n}{(2\pi )}^{m}}\right)}\u27f60,\phantom{\rule{4pt}{0ex}}\phantom{\rule{4pt}{0ex}}as\phantom{\rule{4pt}{0ex}}\phantom{\rule{4pt}{0ex}}n\to \infty $$

$$\underset{n\to \infty}{lim}\left(\frac{n}{k{e}^{u}}{P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)\right)=f(\theta )$$

For $f(\theta )>0$, let $\delta \in (0,f(\theta ))$. Then, for sufficiently large n, $u<ln\sqrt{n}$ and for almost all values of $\theta \in {[0,2\pi )}^{m}$, we have
and therefore
Using (39) in (37) we conclude that, for sufficiently large values of n, and for almost all values of θ
Using (36) and (40) in (27), we conclude further that there exists a constant ${D}_{1}$ such that, for sufficiently large values of n,
for almost all values of $\theta \in {[0,2\pi )}^{m}$.

$${P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)>\frac{k}{n}\left(f(\theta )-\delta \right){e}^{u}$$

$$\begin{array}{ccc}\hfill P\left({B}_{k,n-k}\ge {P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)\right)& \le & P\left({B}_{k,n-k}\ge \frac{k}{n}(f(\theta )-\delta ){e}^{u}\right)\hfill \\ & \le & \frac{E\left({B}_{k,n-k}\right)}{\frac{k}{n}(f(\theta )-\delta ){e}^{u}}=\frac{{e}^{-u}}{f(\theta )-\delta}\hfill \end{array}$$

$${I}_{1}(n)\le \frac{1}{f(\theta )-\delta}{\int}_{0}^{ln\sqrt{n}}{u}^{\u03f5}{e}^{-u}du\le \frac{1}{f(\theta )-\delta}{\int}_{0}^{\infty}{u}^{\u03f5}{e}^{-u}du\phantom{\rule{4pt}{0ex}}<\phantom{\rule{4pt}{0ex}}\infty $$

$${\int}_{0}^{\infty}\mid u{\mid}^{1+\u03f5}d{F}_{\theta ,k,n}(u)du<{D}_{1}$$

Now consider
Note that, for each $u\in (-\infty ,0)$ and for almost all values of $\theta \in {[0,2\pi )}^{m}$,
$$\underset{n\to \infty}{lim}\left(\frac{n}{k{e}^{u}}{P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)\right)=f(\theta )$$
uniformly in $u<0$, i.e., for almost all values of $\theta \in {[0,2\pi )}^{m}$, $u\in (-\infty ,0)$ and every $\delta >0$
for sufficiently large values of n. Therefore,
Using (43) in (42), we conclude that, for sufficiently large values of n and for almost all values of $\theta \in {[0,2\pi )}^{m}$,
Finally, on using (41) and (44) in (26), we conclude (15).

$${\int}_{-\infty}^{0}\mid u{\mid}^{1+\u03f5}d{F}_{\theta ,k,n}(u)={\int}_{-\infty}^{0}{(-u)}^{1+\u03f5}d{F}_{\theta ,k,n}(u)=(1+\u03f5){\int}_{-\infty}^{0}{(-u)}^{\u03f5}{F}_{\theta ,k,n}(u)du$$

$${P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)<\frac{k}{n}(f(\theta )+\delta ){e}^{u}$$

$$\begin{array}{ccc}\hfill {F}_{\theta ,k,n}(u)& =& P\left({B}_{k,n-k}\le {P}_{f}\left({N}_{{\rho}_{k,n}(u)}(\theta )\right)\right)\hfill \\ & \le & P\left({B}_{k,n-k}\le \frac{k}{n}(f(\theta )+\delta ){e}^{u}\right)\hfill \\ & =& \sum _{j=k}^{n-1}\left(\genfrac{}{}{0pt}{}{n-1}{j}\right){\left(\frac{k}{n}\left(f(\theta )+\delta \right){e}^{u}\right)}^{j}{\left(1-\frac{k}{n}\left(f(\theta )+\delta \right){e}^{u}\right)}^{n-1-j}\hfill \\ & \le & \frac{n-1}{n}\phantom{\rule{0.166667em}{0ex}}(f(\theta )+\delta ){e}^{u}\le \left(f(\theta )+\delta \right){e}^{u}\hfill \end{array}$$

$$\begin{array}{ccc}\hfill {\int}_{-\infty}^{0}\mid u{\mid}^{1+\u03f5}d{F}_{\theta ,k,n}(u)& =& (1+\u03f5){\int}_{-\infty}^{0}{(-u)}^{\u03f5}{F}_{\theta ,k,n}(u)du\hfill \\ & \le & (1+\u03f5)\left(f(\theta )+\delta \right){\int}_{-\infty}^{0}{(-u)}^{\u03f5}{e}^{u}du\phantom{\rule{4pt}{0ex}}<\phantom{\rule{4pt}{0ex}}\infty \hfill \end{array}$$

## References

- Karplus, M.; Kushick, J.N. Method for estimating the configurational entropy of macromolecules. Macromolecules
**1981**, 14, 325–332. [Google Scholar] [CrossRef] - Misra, N.; Singh, H.; Demchuk, E. Estimation of the entropy of a multivariate normal distribution. J. Multivar. Anal.
**2005**, 92, 324–342. [Google Scholar] [CrossRef] - Demchuk, E.; Singh, H. Statistical thermodynamics of hindered rotation from computer simulations. Mol. Phys.
**2001**, 99, 627–636. [Google Scholar] [CrossRef] - Singh, H.; Hnizdo, V.; Demchuk, E. Probabilistic modeling of two dependent circular variables. Biometrika
**2002**, 89, 719–723. [Google Scholar] [CrossRef] - Mardia, K.V.; Hughes, G.; Taylor, C.C.; Singh, H. A multivariate von Mises distribution with applications to bioinformatics. Can. J. Stat.
**2008**, 36, 99–109. [Google Scholar] [CrossRef] - Hnizdo, V.; Fedorowicz, A.; Singh, H.; Demchuk, E. Statistical thermodynamics of internal rotation in a hindering potential of mean force obtained from computer simulations. J. Comput. Chem.
**2003**, 24, 1172–1183. [Google Scholar] [CrossRef] [PubMed] - Darian, E.; Hnizdo, V.; Fedorowicz, A.; Singh, H.; Demchuk, E. Estimation of the absolute internal-rotation entropy of molecules with two torsional degrees of freedom from stochastic simulations. J. Comput. Chem.
**2005**, 26, 651–660. [Google Scholar] [CrossRef] [PubMed] - Beirlant, J.; Dudewicz, E.J.; Gyorfi, L.; van der Meulen, E.C. Nonparametric estimation of entropy: An overview. Internat. J. Math. Stat.
**1997**, 6, 17–39. [Google Scholar] - Scott, D. Multivariate Density Estimation: Theory, Practice and Visualization; Wiley: New York, NY, USA, 1992. [Google Scholar]
- Vasicek, O. On a test for normality based on sample entropy. J. R. Stat. Soc. Series B
**1976**, 38, 54–59. [Google Scholar] - Dudewicz, E.J.; van der Meulen, E.C. Entropy-based tests of uniformity. J. Am. Stat. Assoc.
**1981**, 76, 967–974. [Google Scholar] [CrossRef] - Singh, H.; Misra, N.; Hnizdo, V.; Fedorowicz, E.; Demchuk, E. Nearest neighbor estimates of entropy. Am. J. Math. Manag. Sci.
**2003**, 23, 301–321. [Google Scholar] [CrossRef] - Kozachenko, L.F.; Leonenko, N.N. Sample estimates of entropy of a random vector. Prob. Inf. Trans.
**1987**, 23, 95–101. [Google Scholar] - Goria, M.N.; Leonenko, N.N.; Novi Inveradi, P.L. A new class of random vector entropy estimators and its applications. Nonparam. Stat.
**2005**, 17, 277–297. [Google Scholar] [CrossRef] - Kraskov, A.; Stögbauer, H.; Grassberger, P. Estimating mutual information. Phys. Rev. E
**2004**, 69, 066138-1–066138-16. [Google Scholar] [CrossRef] - Tsybakov, A.B.; van der Meulen, E.C. Root-n consistent estimators of entropy for densities with unbounded support. Scan. J. Stat.
**1996**, 23, 75–83. [Google Scholar] - Loftsgaarden, D.O.; Quesenberry, C.P. A non-parametric estimate of a multivariate density function. Ann. Math. Stat.
**1965**, 36, 1049–1051. [Google Scholar] [CrossRef] - Mnatsakanov, R.M.; Misra, N.; Li, Sh.; Harner, E.J. k
_{n}-Nearest neighbor estimators of entropy. Math. Meth. Stat.**2008**, 17, 261–277. [Google Scholar] [CrossRef] - Lebesgue, H. Sur l’intégration des fonctions discontinues. Ann. Ecole Norm.
**1910**, 27, 361–450. [Google Scholar] - Hnizdo, V.; Tan, J.; Killian, B.J.; Gilson, M.K. Efficient calculation of configurational entropy from molecular simulations by combining the mutual-information expansion and nearest-neighbor methods. J. Comput. Chem.
**2008**, 29, 1605–1614. [Google Scholar] [CrossRef] [PubMed] - Hnizdo, V.; Darian, E.; Fedorowicz, A.; Demchuk, E.; Li, S.; Singh, H. Nearest-neighbor nonparametric method for estimating the configurational entropy of complex molecules. J. Comput. Chem.
**2006**, 28, 655–668. [Google Scholar] [CrossRef] [PubMed] - Arya, S.; Mount, D.M. Approximate nearest neighbor searching. In the Proceedings of the Fourth Annual ACM-SIAM Symposium on Discrete Algorithms, 25–27 January 1993; p. 271. Available online: http:// www.cs.umd.edu/∼mount/ANN/ (accessed on 5 May 2010).
- Friedman, J.H.; Bentley, J.L.; Finkel, R.A. An algorithm for finding best matches in logarithmic expected time. ACM Trans. Math. Software
**1977**, 3, 209–226. [Google Scholar] [CrossRef]

© 2010 by the authors; licensee MDPI, Basel, Switzerland. This article is an open-access article distributed under the terms and conditions of the Creative Commons Attribution license http://creativecommons.org/licenses/by/3.0/.