Open Access
This article is

- freely available
- re-usable

*Entropy*
**2013**,
*15*(12),
5154-5177;
doi:10.3390/e15125154

Article

Non–Parametric Estimation of Mutual Information through the Entropy of the Linkage

Department of Mathematics, University of Torino, Via Carlo Alberto 10, Torino 10123, Italy

^{*}

Authors to whom correspondence should be addressed.

Received: 25 September 2013; in revised form: 30 October 2013 / Accepted: 11 November 2013 / Published: 26 November 2013

## Abstract

**:**

A new, non–parametric and binless estimator for the mutual information of a d–dimensional random vector is proposed. First of all, an equation that links the mutual information to the entropy of a suitable random vector with uniformly distributed components is deduced. When $d=2$ this equation reduces to the well known connection between mutual information and entropy of the copula function associated to the original random variables. Hence, the problem of estimating the mutual information of the original random vector is reduced to the estimation of the entropy of a random vector obtained through a multidimensional transformation. The estimator we propose is a two–step method: first estimate the transformation and obtain the transformed sample, then estimate its entropy. The properties of the new estimator are discussed through simulation examples and its performances are compared to those of the best estimators in the literature. The precision of the estimator converges to values of the same order of magnitude of the best estimator tested. However, the new estimator is unbiased even for larger dimensions and smaller sample sizes, while the other tested estimators show a bias in these cases.

Keywords:

information measures; mutual information; entropy; copula function; linkage function; kernel method; binless estimatorMSC Classification:

62G05; 94A17## 1. Introduction

The measure of multivariate association, that is, of the association between groups of components of a general d-dimensional random vector $X=({X}_{1},...,{X}_{d})$, is a topic of increasing interest in a series of application contexts. Among the information measures, the mutual information, a special case of relative entropy or Kullback-Leibler distance [1], is a quantity that measures the mutual dependence between the variables considered. Unlike the measures of linear dependency between random variables, such as the correlation coefficient, the mutual information is particularly interesting as it is sensitive also to dependencies that are not codified in the covariance. Although this is one of the most popular dependency measures, it is only one of the many other existing ones, see for example [2].

The extension of mutual information to d–dimensional random vectors is not trivial [3,4,5]. Considering the distance between the joint distribution of the random vector and the joint distribution of the random vector with independent univariate components gives the so called total mutual information. However, in many instances, it is more interesting to study the distance between groups of components of the random vector, which are again multivariate random vectors of different dimensions. Hence the mutual information, in its general d–dimensional definition, is a distance of the joint distribution of the random vector to the joint distribution of groups of components considered independent from each other. We consider here this general framework with the aim of introducing a good estimator for the multivariate mutual information.

The problem of statistical estimation of mutual information has been considered by various authors. Optimized estimators that use adaptive bin sizes have been developed in [6,7]. A very good estimator based on k–nearest neighbor statistics is proposed in [8]. A computationally efficient modification of this method appeared recently in [9]. The estimation of mutual information sets in the broader context of the estimation of information–type measures such as entropy, Kullback–Leibler distance, divergence functionals, Rényi entropy. The k–nearest neighbor statistics are used to build an estimator for entropy in [10] and for the more general Rényi entropy in [11,12]. An overview of nonparametric entropy estimators can be found in [13]. For the case of discrete random variables, see for example [14]. A variational characterization of divergences allows the estimation of Kullback–Leibler divergence (and more generally any f–divergence) to turn into a convex optimization problem [15,16,17]. An interesting application of information–type quantities to devise a measure of statistical dispersion can be found in [18].

We propose here a new and simple estimator for the mutual information in its general multidimensional definition. To accomplish this aim, we first deduce an equation that links the mutual information between groups of components of a d-dimensional random vector to the entropy of the so called linkage function [19], that reduces to the copula function [20] in dimension $d=2$. In this way the problem of estimating mutual information is reduced to the estimation of the entropy of a suitably transformed sample of the same dimensions as the original random vector. This topic is hence closely related to the estimation of the Shannon entropy.

The structure of the paper is as follows: Notation and mathematical background are introduced in Section 2, where a brief survey on copula function and linkage function is also provided. In Section 3 we expose the method proposed to estimate the mutual information, providing also some details on the resulting algorithm and on the implementation care required. Section 4 contains some examples where the estimator is applied to simulated data and its performances are compared with some other estimators in literature. Conclusions are drawn in Section 5.

## 2. Notations and Mathematical Background

Given a $d-$dimensional random vector $X=({X}_{1},\cdots ,{X}_{d})$, let us denote ${F}_{1,\cdots ,d}({x}_{1},\cdots ,{x}_{d})=P({X}_{1}\le {x}_{1},\cdots ,{X}_{d}\le {x}_{d})$, for $({x}_{1},\cdots ,{x}_{d})\in {\mathbf{R}}^{d}$, the joint cumulative distribution function (c.d.f.) and ${F}_{i}\left(x\right)=P({X}_{i}\le x)$, for $i=1,\cdots ,d$ and $x\in \mathbf{R}$, the marginal c.d.f. of the $i-$th component. When X is absolutely continuous, ${f}_{1,\cdots ,d}({x}_{1},\cdots ,{x}_{d})$ for $({x}_{1},\cdots ,{x}_{d})\in {\mathbf{R}}^{d}$ is the joint probability density function (p.d.f.) and ${f}_{i}\left(x\right)$ for $i=1,\cdots ,d$ and $x\in \mathbf{R}$ the corresponding marginal p.d.f. of the $i-$th component. Furthermore, ${F}_{{i}_{1}|{i}_{2},\cdots ,{i}_{l}}\left({x}_{{i}_{1}}\right|{x}_{{i}_{2},\cdots ,{x}_{{i}_{l}}})$ and ${f}_{{i}_{1}|{i}_{2},\cdots ,{i}_{l}}\left({x}_{{i}_{1}}\right|{x}_{{i}_{2}},\cdots ,{x}_{{i}_{l}})$, where each ${i}_{1},\cdots ,{i}_{l}$ assumes a different value ranging from 1 to d and $l=2,\cdots ,d$, denote respectively the conditional c.d.f. and the conditional p.d.f. of the variable ${X}_{{i}_{1}}$ with respect to the variables ${X}_{{i}_{2}},\cdots ,{X}_{{i}_{d}}$. For the sake of brevity, when $\alpha =({\alpha}_{1},\cdots ,{\alpha}_{n})$ is any multi–index of length n we use the following notation ${F}_{\alpha}\left({x}_{\alpha}\right)={F}_{{\alpha}_{1},\cdots ,{\alpha}_{n}}({x}_{{\alpha}_{1}},\cdots ,{x}_{{\alpha}_{n})}$ denoting the c.d.f. of the random vector ${X}_{\alpha}=({X}_{{\alpha}_{1}},\cdots ,{X}_{{\alpha}_{n}})$ and ${f}_{\alpha}\left({x}_{\alpha}\right)={f}_{{\alpha}_{1},\cdots ,{\alpha}_{n}}({x}_{{\alpha}_{1}},\cdots ,{x}_{{\alpha}_{n})}$ denoting the p.d.f. of ${X}_{\alpha}$. Furthermore, when not necessary, we omit the argument of the functions, i.e., we may write ${f}_{\alpha}$ in spite of ${f}_{\alpha}\left({x}_{\alpha}\right)$.

The mutual information (MI) of a 2-dimensional random vector $X=({X}_{1},{X}_{2})$ is given by
If ${X}_{1}$ and ${X}_{2}$ are independent $MI({X}_{1},{X}_{2})=0$.

$$MI({X}_{1},{X}_{2})={\int}_{{\mathbf{R}}^{2}}{f}_{1,2}({x}_{1},{x}_{2}){log}_{2}\left[\frac{{f}_{1,2}({x}_{1},{x}_{2})}{{f}_{1}\left({x}_{1}\right){f}_{2}\left({x}_{2}\right)}\right]d{x}_{1}d{x}_{2}$$

The relative entropy (or Kullback-Leibler distance) D between the p.d.f.’s f and g is defined by
Hence, MI can be interpreted as the Kullback-Leibler distance of the joint p.d.f. ${f}_{1,2}$ from the p.d.f. ${f}_{1}\xb7{f}_{2}$ of a vector with independent components, i.e., it is a measure of the distance between $X=({X}_{1},{X}_{2})$ and the random vector with the same marginal distributions but independent components.

$$D\left(f\u2225g\right)={\int}_{{\mathbf{R}}^{2}}f({x}_{1},{x}_{2})log\frac{f({x}_{1},{x}_{2})}{g({x}_{1},{x}_{2})}d{x}_{1}d{x}_{2}$$

The differential entropy of the absolutely continuous random vector $X=({X}_{1},\cdots ,{X}_{d})$ is defined as
MI and entropy in the case $d=2$ are related through the well known equation [1]

$$H({X}_{1},\cdots ,{X}_{d})=-{\int}_{{\mathbf{R}}^{d}}{f}_{1,\cdots ,d}({x}_{1},\cdots ,{x}_{d}){log}_{2}{f}_{1,\cdots ,d}({x}_{1},\cdots ,{x}_{d})d{x}_{1}\cdots d{x}_{d}$$

$$MI({X}_{1},{X}_{2})=H\left({X}_{1}\right)+H\left({X}_{2}\right)-H({X}_{1},{X}_{2})$$

The generalization of MI to more than two random variables is not unique [3]. Indeed different definitions can be given according to different grouping of the components of the random vector $X=({X}_{1},\cdots ,{X}_{d})$. More precisely, for any couple of multi-indices $(\alpha ,\beta )$ of dimensions h and k respectively, with $h+k=d$ and partitioning the set of indices $\left\{1,2,\cdots ,d\right\}$, the $MI({X}_{1},\cdots ,{X}_{d})$ can be defined as the $MI({X}_{\alpha},{X}_{\beta})$, where ${X}_{\alpha}=({X}_{{\alpha}_{1}},\cdots ,{X}_{{\alpha}_{h}})$ and ${X}_{\beta}=({X}_{{\beta}_{1}},\cdots ,{X}_{{\beta}_{k}})$. Therefore we have
More generally, for any n multi-indices $({\alpha}^{1},\cdots ,{\alpha}^{n})$ of dimensions ${h}_{1},\cdots ,{h}_{n}$ respectively, such that ${h}_{1}+\cdots +{h}_{n}=d$ and partitioning the set of indices $\left\{1,2,\cdots ,d\right\}$ the following quantities
are all d–dimensional extensions of Equation (1). In the particular case $n=d$ where all the multi-indices have dimension 1, Equation (6) gives the so called total MI, a measure of the distance of the distribution of the given random vector to the one with the same marginal distributions but mutually independent components.

$$\begin{array}{cc}MI({X}_{\alpha},{X}_{\beta})& =\int {f}_{\alpha ,\beta}{log}_{2}\frac{{f}_{\alpha ,\beta}}{{f}_{\alpha}{f}_{\beta}}\hfill \\ & ={\int}_{{\mathbf{R}}^{d}}{f}_{1,\cdots ,d}({x}_{1},\cdots ,{x}_{d}){log}_{2}\left[\frac{{f}_{1,\cdots ,d}({x}_{1},\cdots ,{x}_{d})}{{f}_{{\alpha}_{1},\cdots ,{\alpha}_{h}}({x}_{{\alpha}_{1}},\cdots ,{x}_{{\alpha}_{h}}){f}_{{\beta}_{1},\cdots ,{\beta}_{k}}({x}_{{\beta}_{1}},\cdots ,{x}_{{\beta}_{k}})}\right]d{x}_{1}\cdots d{x}_{d}\hfill \end{array}$$

$$\begin{array}{cc}\hfill MI({X}_{{\alpha}^{1}},\cdots ,{X}_{{\alpha}^{n}})& ={\int}_{{\mathbf{R}}^{d}}{f}_{{\alpha}^{1},\cdots ,{\alpha}^{n}}{log}_{2}\frac{{f}_{{\alpha}^{1},\cdots ,{\alpha}^{n}}}{{f}_{{\alpha}^{1}}\cdots {f}_{{\alpha}^{n}}}\hfill \\ & ={\int}_{{\mathbf{R}}^{d}}{f}_{1,\cdots ,d}({x}_{1},\cdots ,{x}_{d})\times \hfill \\ & {log}_{2}\left[\frac{{f}_{1,\cdots ,d}({x}_{1},\cdots ,{x}_{d})}{{f}_{{\alpha}_{1}^{1},\cdots ,{\alpha}_{{h}_{1}}^{1}}({x}_{{\alpha}_{1}^{1}},\cdots ,{x}_{{\alpha}_{{h}_{1}}^{1}})\cdots {f}_{{\alpha}_{1}^{n},\cdots ,{\alpha}_{{h}_{n}}^{n}}({x}_{{\alpha}_{1}^{n}},\cdots ,{x}_{{\alpha}_{{h}_{n}}^{n}})}\right]d{x}_{1}\cdots d{x}_{d}\hfill \end{array}$$

Equation (6) can be rewritten as
and then, integrating each term, it is is easy to prove the following generalization to the d–dimensional case of Equation (4)

$$\begin{array}{cc}\hfill MI({X}_{{\alpha}^{1}},\cdots ,{X}_{{\alpha}^{n}})& =\int {f}_{{\alpha}^{1},\cdots ,{\alpha}^{n}}{log}_{2}{f}_{{\alpha}^{1},\cdots ,{\alpha}^{n}}\hfill \\ & -\int {f}_{{\alpha}^{1},\cdots ,{\alpha}^{n}}{log}_{2}{f}_{{\alpha}^{1}}-\cdots -\int {f}_{{\alpha}^{1},\cdots ,{\alpha}^{n}}{log}_{2}{f}_{{\alpha}^{n}}\hfill \end{array}$$

$$MI({X}_{{\alpha}^{1}},\cdots ,{X}_{{\alpha}^{n}})=H\left({X}_{{\alpha}^{1}}\right)+\cdots +H\left({X}_{{\alpha}^{n}}\right)-H({X}_{1},\cdots ,{X}_{d})$$

Another approach to the study of dependencies between random variables is given by copula functions (for example [20]). A $d-$dimensional copula (or $d-$copula) is a function $C:{[0,1]}^{d}\to [0,1]$ with the following properties:

- (1)
- for every $u=({u}_{1},\cdots ,{u}_{d})\in {[0,1]}^{d}$, $C\left(u\right)=0$ if at least one coordinate is null and $C\left(u\right)={u}_{k}$ if all coordinates are 1 except ${u}_{k}$;
- (2)
- for every $a=({a}_{1},\cdots ,{a}_{d})$ and $b=({b}_{1},\cdots ,{b}_{d})\in {[0,1]}^{d}$ such that $a\le b$, ${V}_{C}\left([a,b]\right)\ge 0$.

Here ${V}_{C}$ is the so called $C-$volume of $[a,b]$, i.e., the $n-$th order difference of C on $[a,b]$
where
Hence, a copula function is a non–decreasing function in each argument.

$${V}_{C}\left([a,b]\right)={\Delta}_{{a}_{d}}^{{b}_{d}}{\Delta}_{{a}_{d-1}}^{{b}_{d-1}}\cdots {\Delta}_{{a}_{1}}^{{b}_{1}}C\left(u\right)$$

$${\Delta}_{{a}_{k}}^{{b}_{k}}C\left(u\right)=C({u}_{1},\cdots ,{u}_{k-1},{b}_{k},{u}_{k+1},\cdots ,{h}_{d})-C({u}_{1},\cdots ,{u}_{k-1},{a}_{k},{u}_{k+1},\cdots ,{h}_{d})$$

A central result in the theory of copulas is the so called Sklar’s theorem, see [20]. It captures the role that copulas play in the relationship between joint c.d.f.’s and their corresponding marginal univariate distributions. In particular it states that for any $d-$dimensional c.d.f. ${F}_{1,\cdots ,d}$ of the random vector $X=({X}_{1},\cdots ,{X}_{d})$ there exists a $d-$copula C such that for all $x=({x}_{1},\cdots ,{x}_{d})\in {\mathbf{R}}^{d}$
where ${F}_{i}$ are the univariate margins. If the margins are continuous, then the copula C is uniquely determined. Otherwise, C is uniquely determined over Ran${F}_{1}\times \cdots \times $Ran${F}_{d}$, where Ran${F}_{i}$ is the range of the function ${F}_{i}$. Conversely, if C is a copula and ${F}_{i}$, $i=1,\cdots ,d$ are one-dimensional distribution functions, then the function ${F}_{1,\cdots ,d}({x}_{1},\cdots ,{x}_{d})$ defined in Equation (11) is a d-dimensional distribution function with margins ${F}_{i}$, $i=1,\cdots ,d$.

$${F}_{1,\cdots ,d}({x}_{1},\cdots ,{x}_{d})=C({F}_{1}\left({x}_{1}\right),\cdots ,{F}_{d}\left({x}_{d}\right))$$

In particular, a copula C can be interpreted as the joint c.d.f. of the random vector $U=({U}_{1},\cdots ,{U}_{d})$, where each component is obtained through the transformation
Indeed
Moreover, as each component is obtained with Equation (12), the univariate marginal distributions of the random vector U are all uniform on $[0,1]$. An important property that is tightly bound to MI is that copula functions are invariant under strictly increasing transformations of the margins, see [20].

$${U}_{i}={F}_{i}\left({X}_{i}\right),\phantom{\rule{2.em}{0ex}}i=1,\cdots ,d$$

$$\begin{array}{ccc}\hfill P({U}_{1}\le {u}_{1},\cdots ,{U}_{d}\le {u}_{d})& =& P({F}_{1}\left({X}_{1}\right)\le {u}_{1},\cdots ,{F}_{d}\left({X}_{d}\right)\le {u}_{d})\hfill \\ & =& P({X}_{1}\le {F}_{1}^{-1}\left({u}_{1}\right),\cdots ,{X}_{d}\le {F}_{d}^{-1}\left({u}_{d}\right))\hfill \\ & =& {F}_{1,\cdots ,d}({F}_{1}^{-1}\left({u}_{1}\right),\cdots ,{F}_{d}^{-1}\left({u}_{d}\right))\hfill \\ \mathrm{(13)}& & =& C({u}_{1},\cdots ,{u}_{d})\hfill \end{array}$$

If a copula C is differentiable, the joint p.d.f. of the random vector X can be written as
where
is the so called density of the copula C.

$${f}_{1,\cdots ,d}({x}_{1},\cdots ,{x}_{d})=\prod _{i=1}^{d}{f}_{i}\left({x}_{i}\right)\xb7c({F}_{1}\left({x}_{1}\right),\cdots ,{F}_{d}\left({x}_{d}\right))$$

$$c({u}_{1},\cdots ,{u}_{d})=\frac{{\partial}^{d}}{\partial {u}_{1}\cdots \partial {u}_{d}}C({u}_{1},\cdots ,{u}_{d}),\phantom{\rule{0.166667em}{0ex}}u\in {[0,1]}^{d}$$

As illustrated in [21], it is not possible to use copula functions to handle multivariate distribution with given marginal distributions of general dimensions, since the only copula that is compatible with any assigned multidimensional marginal distributions is the independent one. To study dependencies between multidimensional random variables we resort to a generalization of copula notion. It relies on the use of the so-called linkage functions introduced in [19].

Let us consider the d-dimensional random vector X and any n multi-indices $({\alpha}^{1},\cdots ,{\alpha}^{n})$ of dimensions $({h}_{1},\cdots ,{h}_{n})$ respectively, such that ${h}_{1}+\cdots +{h}_{n}=d$ and partitioning the set $\{1,2,\cdots ,d\}$. For $i=1,\cdots ,n$, let ${F}_{{\alpha}^{i}}$ be the ${h}_{i}$-dimensional c.d.f. of ${X}_{{\alpha}^{i}}=({X}_{{\alpha}_{1}^{i}},\cdots ,{X}_{{\alpha}_{{h}_{i}}^{i}})$ and let ${F}_{{\alpha}^{1},\cdots ,{\alpha}^{n}}$ be the joint c.d.f. of ${X}_{{\alpha}^{1}},\cdots ,{X}_{{\alpha}^{n}}$, which is of dimension d as the multi–indices are partitioning the set $\{1,2,\cdots ,d\}$.

Define the transformation ${\Psi}_{{\alpha}^{i}}:{\mathbf{R}}^{{h}_{i}}\to {[0,1]}^{{h}_{i}}$, $i=1,\cdots ,n$ as
for all ${x}_{{\alpha}^{i}}$ in the range of ${X}_{{\alpha}^{i}}$.

$${\Psi}_{{\alpha}^{i}}({x}_{{\alpha}_{1}^{i}},\cdots ,{x}_{{\alpha}_{{h}_{i}}^{i}})=({F}_{{\alpha}_{1}^{i}}\left({x}_{{\alpha}_{1}^{i}}\right),{F}_{{\alpha}_{2}^{i}|{\alpha}_{1}^{i}}\left({x}_{{\alpha}_{2}^{i}}\right|{x}_{{\alpha}_{1}^{i}}),\cdots ,{F}_{{\alpha}_{{h}_{i}}^{i}|{\alpha}_{1}^{i},\cdots ,{\alpha}_{{h}_{i}-1}^{i}}\left({x}_{{\alpha}_{{h}_{i}}^{i}}\right|{x}_{{\alpha}_{1}^{i}},{x}_{{\alpha}_{2}^{i}},\cdots ,{x}_{{\alpha}_{{h}_{i}-1}^{i}}))$$

Then the vectors
are ${h}_{i}$-dimensional vectors of independent uniform $[0,1]$ random variables, see [19].

$${U}_{{\alpha}^{i}}=({U}_{{\alpha}_{1}^{i}},\cdots {U}_{{\alpha}_{{h}_{i}}^{i}})={\Psi}_{{\alpha}^{i}}({X}_{{\alpha}_{1}^{i}},\cdots ,{X}_{{\alpha}_{{h}_{i}}^{i}})={\Psi}_{{\alpha}^{i}}\left({X}_{{\alpha}^{i}}\right)$$

The linkage function corresponding to the d-dimensional random vector $({X}_{{\alpha}^{1}},\cdots ,{X}_{{\alpha}^{n}})$ is defined as the joint p.d.f. L of the vector

$$({U}_{{\alpha}^{1}},\cdots ,{U}_{{\alpha}^{n}})=({U}_{{\alpha}_{1}^{1}},\cdots ,{U}_{{\alpha}_{{h}_{1}}^{1}},\cdots ,{U}_{{\alpha}_{1}^{n}},\cdots ,{U}_{{\alpha}_{{h}_{n}}^{n}})=\left({\Psi}_{{\alpha}^{1}}\left({X}_{{\alpha}^{1}}\right),\cdots ,{\Psi}_{{\alpha}^{n}}\left({X}_{{\alpha}^{n}}\right)\right)$$

Notice that for $d=2$ the linkage function reduces to the copula function. Analogously to the two–dimensional case, linkages are invariant under strictly increasing functions, that is a d–dimensional function with components strictly increasing real univariate functions, see Theorem 3.4 in [19].

The information regarding the dependence properties between the multivariate components of the random vector $({X}_{{\alpha}^{1}},\cdots ,{X}_{{\alpha}^{n}})$ is contained in the linkage function, that is independent from the marginal c.d.f.’s. Linkage functions can then be successfully employed when one is interested in studying the dependence properties between the random vectors $({X}_{{\alpha}^{1}},\cdots ,{X}_{{\alpha}^{n}})$ disregarding their single components. On the other hand, linkage functions allow to study the interrelationships not only between all the components of a random vector, but also between given chosen sets of not overlapping random components of the vector. They contain the information regarding the dependence structure among such marginal vectors, while the dependence structure within the marginal vectors is not considered explicitly. It must be underlined that different multivariate c.d.f.’s ${F}_{{\alpha}^{1},\cdots ,{\alpha}^{n}}$ can have the same linkage function but different marginal distributions.

In the next Section we will discuss the use of linkage functions for the computation of the MI between random vectors.

## 3. The Method

We propose here a method to estimate the MI of a d-dimensional random vector defined in Equation (6) by means of a random sample drawn from the corresponding d-dimensional joint distribution. We assume that neither the joint c.d.f. nor the marginal c.d.f.’s of the components of the random vector are known. The estimation approach proposed is then completely non parametric.

#### 3.1. MI of a d–Dimensional Random Vector and Entropy of the Linkage

The method we are presenting is based on the equation between the MI of a d–dimensional random vector and its entropy deduced in the next theorem.

**Theorem 1.**Let $X=({X}_{1},\cdots ,{X}_{d})$ be a d–dimensional random vector. For any n multi-indices $({\alpha}^{1},\cdots ,{\alpha}^{n})$ of dimensions $({h}_{1},\cdots ,{h}_{n})$ respectively, such that ${h}_{1}+\cdots +{h}_{n}=d$ and partitioning the set of indices $\left\{1,2,\cdots ,d\right\}$, it holds

$$MI({X}_{{\alpha}^{1}},\cdots ,{X}_{{\alpha}^{n}})=-H({U}_{{\alpha}^{1}},\cdots ,{U}_{{\alpha}^{n}})$$

Proof. According to Equation (16) the random vector $({U}_{{\alpha}^{1}},\cdots ,{U}_{{\alpha}^{n}})$ is obtained from $({X}_{1},\cdots ,{X}_{d})$ by means of the following transformations:
where the elements of the vector $({X}_{1},\cdots ,{X}_{d})$ are grouped according to the multi–indices $({\alpha}^{1},\cdots ,{\alpha}^{n})$. The corresponding Jacobian matrix is given as follows
It is evident that the Jacobian matrix is a block diagonal matrix having as main diagonal blocks lower triangular matrices. Its determinant can be calculated as the product of the determinants of the main diagonal blocks, and hence
Recalling the transformations given in Equation (20) we have
where the second to the last equality is obtained using iteratively the definition of conditional probability density function, i.e.,
As the random variables $({U}_{{\alpha}^{1}},\cdots ,{U}_{{\alpha}^{n}})$ are the transformation of the vector $({X}_{1},\cdots ,{X}_{d})$ given in Equation (20), their joint p.d.f. can be deduced as
where ${f}_{1,\cdots ,d}$ is the joint p.d.f. of $({X}_{1},\cdots ,{X}_{d})$ and $\left|J\right|$ is given in Equation (22). The last equation can be further simplified by renaming $({U}_{{{\alpha}^{1}}_{1}},\cdots ,{U}_{{{\alpha}^{1}}_{{h}_{1}}},{U}_{{{\alpha}^{2}}_{1}},\cdots ,{U}_{{{\alpha}^{2}}_{{h}_{2}}},{U}_{{\alpha}_{1}^{n},\cdots ,{\alpha}_{{h}_{n}}^{n}})=({U}_{1},\cdots ,{U}_{d})$, and we get

$$\left\{\begin{array}{ccc}{U}_{{\alpha}_{1}^{1}}\hfill & =& {F}_{{\alpha}_{1}^{1}}\left({X}_{{\alpha}_{1}^{1}}\right)\hfill \\ {U}_{{\alpha}_{2}^{1}}\hfill & =& {F}_{{\alpha}_{2}^{1}|{\alpha}_{1}^{1}}\left({X}_{{\alpha}_{2}^{1}}\right|{X}_{{\alpha}_{1}^{1}})\hfill \\ & \vdots & \\ {U}_{{\alpha}_{{h}_{1}}^{1}}\hfill & =& {F}_{{\alpha}_{{h}_{1}}^{1}|{\alpha}_{1}^{1},{\alpha}_{2}^{1},\cdots ,{\alpha}_{{h}_{1}-1}^{1}}\left({X}_{{\alpha}_{{h}_{1}}^{1}}\right|{X}_{{\alpha}_{1}^{1}},{X}_{{\alpha}_{2}^{1}},\cdots ,{X}_{{\alpha}_{{h}_{1}-1}^{1}})\hfill \\ {U}_{{\alpha}_{1}^{2}}\hfill & =& {F}_{{\alpha}_{1}^{2}}\left({X}_{{\alpha}_{1}^{2}}\right)\hfill \\ {U}_{{\alpha}_{2}^{2}}\hfill & =& {F}_{{\alpha}_{2}^{2}|{\alpha}_{1}^{2}}\left({X}_{{\alpha}_{2}^{2}}\right|{X}_{{\alpha}_{1}^{2}})\hfill \\ & \vdots & \\ {U}_{{\alpha}_{1}^{n}}\hfill & =& {F}_{{\alpha}_{1}^{n}}\left({X}_{{\alpha}_{1}^{n}}\right)\hfill \\ & \vdots & \\ {U}_{{\alpha}_{{h}_{n}}^{n}}\hfill & =& {F}_{{\alpha}_{{h}_{n}}^{n}|{\alpha}_{1}^{n},\cdots ,{\alpha}_{{h}_{n}-1}^{n}}\left({X}_{{\alpha}_{{h}_{n}}^{n}}\right|{X}_{{\alpha}_{{h}_{1}}^{n}},\cdots ,{X}_{{\alpha}_{{h}_{n}-1}^{n}})\hfill \end{array}\right.$$

$$J=\left(\begin{array}{ccccccccccccc}\frac{\partial {u}_{{\alpha}_{1}^{1}}}{\partial {x}_{{\alpha}_{1}^{1}}}& 0& \cdots & 0& 0& 0& \cdots & 0& \cdots & 0& 0& \cdots & 0\\ \frac{\partial {u}_{{\alpha}_{2}^{1}}}{\partial {x}_{{\alpha}_{1}^{1}}}& \frac{\partial {u}_{{\alpha}_{2}^{1}}}{\partial {x}_{{\alpha}_{2}^{1}}}& \cdots & 0& 0& 0& \cdots & 0& \cdots & 0& 0& \cdots & 0\\ \vdots & \vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots & \cdots & \vdots & \vdots & \ddots & \vdots \\ \frac{\partial {u}_{{\alpha}_{{h}_{1}}^{1}}}{\partial {x}_{{\alpha}_{1}^{1}}}& \frac{\partial {u}_{{\alpha}_{{h}_{1}}^{1}}}{\partial {x}_{{\alpha}_{2}^{1}}}& \cdots & \frac{\partial {u}_{{\alpha}_{{h}_{1}}^{1}}}{\partial {x}_{{\alpha}_{{h}_{1}}^{1}}}& 0& 0& \cdots & 0& \cdots & 0& 0& \cdots & 0\\ 0& 0& \cdots & 0& \frac{\partial {u}_{{\alpha}_{1}^{2}}}{\partial {x}_{{\alpha}_{1}^{2}}}& 0& \cdots & 0& \cdots & 0& 0& \cdots & 0\\ 0& 0& \cdots & 0& \frac{\partial {u}_{{\alpha}_{2}^{2}}}{\partial {x}_{{\alpha}_{1}^{2}}}& \frac{\partial {u}_{{\alpha}_{2}^{2}}}{\partial {x}_{{\alpha}_{2}^{2}}}& \cdots & 0& \cdots & 0& 0& \cdots & 0\\ \vdots & \vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots & \cdots & \vdots & \vdots & \ddots & \vdots \\ 0& 0& \cdots & 0& \frac{\partial {u}_{{\alpha}_{{h}_{2}}^{2}}}{\partial {x}_{{\alpha}_{1}^{2}}}& \frac{\partial {u}_{{\alpha}_{{h}_{2}}^{2}}}{\partial {x}_{{\alpha}_{2}^{2}}}& \cdots & \frac{\partial {u}_{{\alpha}_{{h}_{2}}^{2}}}{\partial {x}_{{\alpha}_{{h}_{2}}^{2}}}& \cdots & 0& 0& \cdots & 0\\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \vdots & \vdots \\ 0& 0& \cdots & 0& 0& 0& \cdots & 0& \cdots & \frac{\partial {u}_{{\alpha}_{1}^{n}}}{\partial {x}_{{\alpha}_{1}^{n}}}& 0& \cdots & 0\\ 0& 0& \cdots & 0& 0& 0& \cdots & 0& \cdots & \frac{\partial {u}_{{\alpha}_{2}^{n}}}{\partial {x}_{{\alpha}_{1}^{n}}}& \frac{\partial {u}_{{\alpha}_{2}^{n}}}{\partial {x}_{{\alpha}_{2}^{n}}}& \cdots & 0\\ \vdots & \vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots & \cdots & \vdots & \vdots & \ddots & \vdots \\ 0& 0& \cdots & 0& 0& 0& \cdots & 0& \cdots & \frac{\partial {u}_{{\alpha}_{{h}_{n}}^{n}}}{\partial {x}_{{\alpha}_{1}^{n}}}& \frac{\partial {u}_{{\alpha}_{{h}_{n}}^{n}}}{\partial {x}_{{\alpha}_{2}^{n}}}& \cdots & \frac{\partial {u}_{{\alpha}_{{h}_{n}}^{n}}}{\partial {x}_{{\alpha}_{{h}_{n}}^{n}}}\end{array}\right).$$

$$\left|J\right|=\frac{\partial {u}_{{\alpha}_{1}^{1}}}{\partial {x}_{{\alpha}_{1}^{1}}}\cdots \frac{\partial {u}_{{\alpha}_{{h}_{1}}^{1}}}{\partial {x}_{{\alpha}_{{h}_{1}}^{1}}}\frac{\partial {u}_{{\alpha}_{1}^{2}}}{\partial {x}_{{\alpha}_{1}^{2}}}\cdots \frac{\partial {u}_{{\alpha}_{{h}_{2}}^{2}}}{\partial {x}_{{\alpha}_{{h}_{2}}^{2}}}\frac{\partial {u}_{{\alpha}_{1}^{n}}}{\partial {x}_{{\alpha}_{1}^{n}}}\cdots \frac{\partial {u}_{{\alpha}_{{h}_{n}}^{n}}}{\partial {x}_{{\alpha}_{{h}_{n}}^{n}}}$$

$$\begin{array}{ccc}\hfill \left|J\right|& =& \frac{\partial {u}_{{\alpha}_{1}^{1}}}{\partial {x}_{{\alpha}_{1}^{1}}}\cdots \frac{\partial {u}_{{\alpha}_{{h}_{1}}^{1}}}{\partial {x}_{{\alpha}_{{h}_{1}}^{1}}}\frac{\partial {u}_{{\alpha}_{1}^{2}}}{\partial {x}_{{\alpha}_{1}^{2}}}\cdots \frac{\partial {u}_{{\alpha}_{{h}_{2}}^{2}}}{\partial {x}_{{\alpha}_{{h}_{2}}^{2}}}\frac{\partial {u}_{{\alpha}_{1}^{n}}}{\partial {x}_{{\alpha}_{1}^{n}}}\cdots \frac{\partial {u}_{{\alpha}_{{h}_{n}}^{n}}}{\partial {x}_{{\alpha}_{{h}_{n}}^{n}}}\hfill \\ & =& \prod _{j=1}^{n}\left(\frac{\partial {F}_{{\alpha}_{1}^{j}}}{\partial {x}_{{\alpha}_{1}^{j}}}\prod _{i=2}^{{h}_{j}}\frac{\partial {F}_{{\alpha}_{i}^{j}|{\alpha}_{1}^{j},\cdots ,{\alpha}_{i-1}^{j}}}{\partial {x}_{{\alpha}_{i}^{j}}}\right)\hfill \\ & =& \prod _{j=1}^{n}\left({f}_{{\alpha}_{1}^{j}}\prod _{i=2}^{{h}_{j}}{f}_{{\alpha}_{i}^{j}|{\alpha}_{1}^{j},\cdots ,{\alpha}_{i-1}^{j}}\right)\hfill \\ & =& \prod _{j=1}^{n}{f}_{{\alpha}_{1}^{j},\cdots ,{\alpha}_{{h}_{j}}^{j}}\hfill \\ \mathrm{(22)}& & =& {f}_{{\alpha}^{1}}\cdots {f}_{{\alpha}^{n}}\hfill \end{array}$$

$${f}_{{\alpha}_{1}^{j}|{\alpha}_{2}^{j}}=\frac{{f}_{{\alpha}_{1}^{j},{\alpha}_{2}^{j}}}{{f}_{{\alpha}_{2}^{j}}},\phantom{\rule{1.em}{0ex}}{f}_{{\alpha}_{3}^{j}|{\alpha}_{1}^{j},{\alpha}_{2}^{j}}=\frac{{f}_{{\alpha}_{1}^{j},{\alpha}_{2}^{j},{\alpha}_{3}^{j}}}{{f}_{{\alpha}_{1}^{j},{\alpha}_{2}^{j}}},\phantom{\rule{1.em}{0ex}}\cdots \phantom{\rule{1.em}{0ex}},{f}_{{\alpha}_{{h}_{j}}^{j}|{\alpha}_{1}^{j},\cdots ,{\alpha}_{{h}_{j}-1}^{j}}=\frac{{f}_{{\alpha}_{1}^{j},\cdots ,{\alpha}_{{h}_{j}}^{j}}}{{f}_{{\alpha}_{1}^{j},\cdots ,{\alpha}_{{h}_{j}-1}^{j}}}$$

$${f}_{{U}_{{\alpha}^{1}},\cdots ,{U}_{{\alpha}^{n}}}({u}_{{\alpha}^{1}},\cdots ,{u}_{{\alpha}^{n}})=\frac{{f}_{1,\cdots ,d}({x}_{1},\cdots ,{x}_{d})}{\left|J\right|}$$

$${f}_{{U}_{1},\cdots ,{U}_{d}}({u}_{1},\cdots ,{u}_{d})=\frac{{f}_{1,\cdots ,d}({x}_{1},\cdots ,{x}_{d})}{\left|J\right|}$$

From the d–dimensional definition of mutual information given in Equation (6) and applying the change of variables given in Equation (20) we have
Just rewriting the last line of the previous equation using Equation (3), we get the thesis
$$MI({X}_{{\alpha}^{1}},\cdots ,{X}_{{\alpha}^{n}})=-H({U}_{{\alpha}^{1}},\cdots ,{U}_{{\alpha}^{n}})$$
□

$$\begin{array}{ccc}\hfill MI({X}_{{\alpha}^{1}},\cdots ,{X}_{{\alpha}^{n}})& =& {\int}_{{\mathbf{R}}^{d}}{f}_{1,\cdots ,d}({x}_{1},\cdots ,{x}_{d}){log}_{2}\frac{{f}_{1,\cdots ,d}({x}_{1},\cdots ,{x}_{d})}{{f}_{{\alpha}^{1}}\left({x}_{{\alpha}^{1}}\right)\cdots {f}_{{\alpha}^{n}}\left({x}_{{\alpha}^{n}}\right)}d{x}_{1}\cdots d{x}_{d}\hfill \\ \hfill & =& {\int}_{{\mathbf{R}}^{d}}{f}_{{U}_{1},\cdots ,{U}_{d}}({u}_{1}({x}_{1},\cdots ,{x}_{d}),\cdots ,{u}_{d}({x}_{1},\cdots ,{x}_{d}))\left|J\right|\hfill \\ \hfill & \times & {log}_{2}\left[{f}_{{U}_{1},\cdots ,{U}_{d}}({u}_{1}({x}_{1},\cdots ,{x}_{d}),\cdots ,{u}_{d}({x}_{1},\cdots ,{x}_{d}))\right]d{x}_{1}\cdots d{x}_{d}\hfill \\ \hfill & =& {\int}_{{[0,1]}^{d}}{f}_{{U}_{1},\cdots ,{U}_{d}}({u}_{1},\cdots ,{u}_{d}){log}_{2}{f}_{{U}_{1},\cdots ,{U}_{d}}({u}_{1},\cdots ,{u}_{d})d{u}_{1}\cdots d{u}_{d}\hfill \end{array}$$

Theorem 1 proves that the MI of a random vector can be computed without resorting to the marginal distributions, but rather transforming the components by means of Equation (20) into one-dimensional uniform $[0,1]$ random variables and then computing the Shannon entropy of the resulting vector.

**Remark:**Linkage functions do not change for monotone transformations. This property reflects the well know invariance property of MI under reparametrization.

In the particular case $d=2$ the definition of mutual information is simpler, see Equation (1) and the linkage functions reduce to the well known copula functions, as shown in the following

**Corollary 1.**The MI of the 2-dimensional random vector $X=({X}_{1},{X}_{2})$ can be obtained as

$$MI({X}_{1},{X}_{2})=-H\left({U}_{1},{U}_{2}\right)$$

Proof. For $d=2$ the only non trivial choice of multi–indices partitioning the set $\{1,2\}$ is ${\alpha}^{1}=1$ and ${\alpha}^{2}=2$, hence ${h}_{1}=1$ and ${h}_{2}=1$. The transformations given in Equation (20) reduce to Equation (12) for $d=2$
Let us notice that the joint c.d.f. of the couple $({U}_{1},{U}_{2})$ is the copula function, see Equation (13). Hence their p.d.f. is obtained as
where $c({u}_{1},{u}_{2})$ is the copula density defined in Equation (15). In this special case Equation (22) reduces to
and the definition given in Equation (1) can be rewritten as

$$\left\{\begin{array}{ccc}{U}_{1}\hfill & =& {F}_{1}\left({X}_{1}\right)\hfill \\ {U}_{2}\hfill & =& {F}_{2}\left({X}_{2}\right)\hfill \end{array}\right.$$

$${f}_{1,2}({x}_{1},{x}_{2})=\frac{{\partial}^{2}}{\partial {x}_{1}\partial {x}_{2}}C({F}_{1}\left({x}_{1}\right),{F}_{2}\left({x}_{2}\right))=c({F}_{1}\left({x}_{1}\right),{F}_{2}\left({x}_{2}\right)){f}_{1}\left({x}_{1}\right){f}_{2}\left({x}_{2}\right)$$

$$\left|J\right|={f}_{1}\left({x}_{1}\right){f}_{2}\left({x}_{2}\right)$$

$$\begin{array}{ccc}\hfill MI({X}_{1},{X}_{2})& =& {\int}_{{\mathbf{R}}^{2}}{f}_{1}\left({x}_{1}\right){f}_{2}\left({x}_{2}\right)c\left({F}_{1}\left({x}_{1}\right),{F}_{2}\left({x}_{2}\right)\right){log}_{2}c\left({F}_{1}\left({x}_{1}\right),{F}_{2}\left({x}_{2}\right)\right)d{x}_{1}d{x}_{2}\hfill \\ & =& {\int}_{{[0,1]}^{2}}c({u}_{1},{u}_{2}){log}_{2}c({u}_{1},{u}_{2})d{u}_{1}d{u}_{2}\hfill \\ & =& -H\left({U}_{1},{U}_{2}\right)\hfill \end{array}$$

Hence, the MI between two one-dimensional random variables can be computed without resorting to the marginal distributions of the two variables, but transforming them into the corresponding one-dimensional uniform $[0,1]$ variables according to Equation (12) and then computing their Shannon entropy. Equation (26) can be equivalently rewritten as
where ${E}_{C}$ denotes the expectation with respect to the copula distribution. Note that the result concerning $d=2$ has already been obtained in [22,23] and one of the more impactful studies that exploits this equality is [24].

$$MI({X}_{1},{X}_{2})={E}_{C}\left[{log}_{2}c({U}_{1},{U}_{2})\right]$$

**Example.**In the particular case where $X=({X}_{1},{X}_{2},{X}_{3})$ and the multi–indices are chosen such that ${X}_{{\alpha}^{1}}={X}_{1}$ and ${X}_{{\alpha}^{2}}=({X}_{2},{X}_{3})$ we have

$$MI({X}_{{\alpha}^{1}},{X}_{{\alpha}^{2}})=-H({U}_{1},{U}_{2},{U}_{3})$$

$$\left\{\begin{array}{c}{U}_{1}={F}_{1}\left({X}_{1}\right)\hfill \\ {U}_{2}={F}_{2}\left({X}_{2}\right)\hfill \\ {U}_{3}={F}_{3|2}\left({X}_{3}\right|{X}_{2})\hfill \end{array}\right.$$

#### 3.2. The Estimation Procedure

Let $({X}^{1},\cdots ,{X}^{N})$ be a random sample of size N drawn from the multivariate distribution ${F}_{1,\cdots ,d}$ of the random vector X. Using Theorem 1, we propose here a method to estimate the mutual information of the random vector X in its general definition, i.e., the $MI({X}_{{\alpha}^{1}},\cdots ,{X}_{{\alpha}^{n}})$ for any n multi-indices $({\alpha}^{1},\cdots ,{\alpha}^{n})$ of dimensions $({h}_{1},\cdots ,{h}_{n})$ respectively, such that ${h}_{1}+\cdots +{h}_{n}=d$ and partitioning the set $1,2,\cdots ,d$.

Let us proceed as follows:

- estimate the conditional c.d.f.’s in Equation (20). Denote these functions as ${\widehat{\Psi}}_{{\alpha}^{i}}=({\widehat{F}}_{{\alpha}_{1}^{i}},{\widehat{F}}_{{\alpha}_{2}^{i}|{\alpha}_{2}^{i}},\cdots ,{\widehat{F}}_{{\alpha}_{{h}_{i}}^{i}|{\alpha}_{{h}_{i}-1}^{i}})$, for $i=1,\cdots ,n$;
- for $k=1,\cdots ,N$ calculate ${U}^{k}=({U}_{{\alpha}^{1}}^{k},\cdots ,{U}_{{\alpha}^{n}}^{k})$, where ${U}_{{\alpha}^{i}}^{k}=({\widehat{\Psi}}_{{\alpha}^{1}}\left({X}_{{\alpha}^{1}}^{k}\right),\cdots ,{\widehat{\Psi}}_{{\alpha}^{n}}\left({X}_{{\alpha}^{n}}^{k}\right))$, for $i=1,\cdots ,n$;
- estimate the $MI({X}_{{\alpha}^{1}},\cdots ,{X}_{{\alpha}^{n}})$ as the Shannon entropy in Equation (19) of the transformed sample $({U}^{1},\cdots ,{U}^{N})$.

For the particular case when $d=2$ the procedure becomes the following:

- estimate the c.d.f.’s in Equation (27). Denote the estimated functions as $({\widehat{F}}_{1},{\widehat{F}}_{2})$;
- calculate ${U}^{k}=({\widehat{F}}_{1}\left({X}_{1}^{k}\right),{\widehat{F}}_{2}\left({X}_{2}^{k}\right))$, for $k=1,\cdots ,N$;
- estimate $MI({X}_{1},{X}_{2})$ as the Shannon entropy in Equation (26) of the transformed sample $({U}^{1},\cdots ,{U}^{N})$.

Let us now enter the details of the above illustrated procedure. The performances of the proposed method strongly depend on the estimators used for the c.d.f.’s and conditional c.d.f.’s at step 1 and the entropy at step 3 of the above proposed algorithm.

**Step 1 of the procedure: estimating c.d.f’s.**Let us compare the performances of the estimation algorithm for three different choices of the estimator for the c.d.f.’s, namely the empirical distribution function, the kernel density estimator and the k–nearest neighbor density estimator.

The most natural estimator for the c.d.f is the empirical distribution function, that for $({X}^{1},\cdots ,{X}^{N})$ a univariate random sample of size N is defined as
Despite its good asymptotic properties, the empirical distribution function exhibits a slow rate of convergence, see for example [25] where it is shown that the empirical distribution function is a deficient estimator with respect to a certain class of kernel type estimators. It could be then necessary to resort to more reliable techniques to estimate the c.d.f.’s.

$${\widehat{F}}_{N}\left(t\right)=\frac{1}{N}\sum _{i=1}^{N}{\mathbf{1}}_{(-\infty ,t]}\left({X}^{i}\right)$$

As suggested in [25], we could consider kernel–based density estimators, both to estimate the univariate c.d.f.’s and the conditional c.d.f.’s, where the conditioning random variable could be multivariate, see Equation (20). Let us recall that, given ${X}^{1},\cdots ,{X}^{N}$ a random sample of size N drawn from a law with p.d.f. f, the univariate kernel density estimator for f is obtained as
where K is a suitable function called kernel, that is required to be a normalized probability density, and h is a positive number called the bandwidth, see [26,27]. One of the most commonly used kernel is the Gaussian kernel
However, depending on the properties of the density to be estimated, it can be convenient to consider different kernels. For example the rectangular kernel and the Gamma kernel can be recommended to estimate the uniform and the exponential distributions respectively. The bandwidth h has to be chosen carefully as too small values can give rise to spurious fine structures of the estimate, while too large values can hide all the details in the estimated density curve. The choice of a possible optimal bandwidth usually falls on the one that minimizes the mean integrated square error. When it is assumed that the underlying distribution is Gaussian, the optimal bandwidth ${h}_{opt}$ is given by
where σ denotes the standard deviation of the sample values, see [26]. Let us remark that having no parametric hypothesis on the distribution of the random vector considered, in our method the choice of the kernel function and of the bandwidth should be made by visual inspection of the histograms of the involved distributions.

$${\widehat{f}}_{ker}\left(x\right)=\frac{1}{Nh}\sum _{i=1}^{N}K\left(\frac{x-{X}^{i}}{h}\right)$$

$${\widehat{f}}_{ker}\left(x\right)=\frac{1}{Nh}\sum _{i=1}^{N}\frac{1}{\sqrt{2\pi}}exp\left(-\frac{{(x-{X}^{i})}^{2}}{2{h}^{2}}\right)$$

$${h}_{opt}={\left(\frac{4}{3N}\right)}^{\frac{1}{5}}\sigma \cong 1.06\phantom{\rule{0.277778em}{0ex}}\sigma {N}^{-\frac{1}{5}}$$

A third choice to estimate the c.d.f.’s could rely on the nearest neighbor density estimator, see [27,28]. Let us recall that, given ${X}^{1},\cdots ,{X}^{N}$ a random sample of size N drawn from a law with p.d.f. f, the nearest neighbor density estimator for f is obtained as
where k is fixed and V is the volume of the sphere that contains precisely k data points, the $k-$nearest to x. A possible generalization of the nearest neighbor estimator provides an estimator of the kernel type again, as in Equation (34) but with bandwidth h equal to the distance between x and its $k-$nearest neighbor. As actually the generalized nearest neighbor is a kernel estimator with a suitable choice of the bandwidth parameter, we are calling nearest neighbor density estimator only the simple one given in Equation (37).

$${\widehat{f}}_{knn}\left(x\right)=\frac{k}{NV}$$

For the estimation of conditional densities, some papers propose methods improving the naive kernel estimator obtained as the ratio of the multidimensional kernel estimators for the joint p.d.f. and the joint p.d.f. of the conditioning variables, see [29,30,31]. In this paper we use the estimator proposed in [30] and successively improved in [31], based on a locally polynomial regression.

The performances of our estimator for the three different choices of c.d.f.’s estimations are compared in Figure 1. The samples are drawn from a bivariate gaussian distribution with standard marginal distributions and correlation coefficient $\rho =0.9$ for different sample sizes N. The accuracy of the estimation increases with increasing sample size, but the best performance is definitely obtained with the kernel density estimator (black).

**Figure 1.**95% confidence intervals of the estimated MI of a bivariate gaussian distribution as the sample size N increases. True value of MI = 1.1979 bits. Cumulative distribution functions are estimated by the empirical distribution function given in Equation (33) (blue), the kernel density estimator given in Equation (34) (black) and the $k-$nearest neighbor density estimator given in Equation (37) (red).

**Step 3 of the procedure: estimating entropy.**The second important point that requires some care, is the estimation of the differential entropy of the transformed sample at step 3 of our method. Also in this case the estimation can be performed using a kernel type estimator or a nearest neighbor one.

The unbinned nearest-neighbor method proposed in [10,32] seems to have good properties. It is based on nearest–neighbor Euclidean distances and it has been shown to be asymptotically unbiased and consistent, provided that the underlying p.d.f. obeys certain conditions that control the convergence of the integrals for the differential entropy, see [10]. Given a random sample of size N from the $d-$dimensional random vector $X=({X}_{1},\cdots ,{X}_{d})$, the estimator proposed for the differential entropy is given by
where $\gamma =-{\int}_{0}^{\infty}{e}^{-v}lnvdv\cong 0.5772156649$ is the Euler-Mascheroni constant, ${\lambda}_{j}$ is the Euclidean distance of each sample point to its nearest neighbor and
with Γ the gamma function is the area of a unit d-dimensional spherical surface (for example ${S}_{1}=2,$ ${S}_{2}=2\pi ,\phantom{\rule{0.277778em}{0ex}}{S}_{3}=4\pi ,\cdots )$.

$${\widehat{H}}_{knn}=\frac{d}{N}\sum _{j=1}^{N}{log}_{2}\left({\lambda}_{j}\right)+{log}_{2}\left[\frac{{S}_{d}(N-1)}{d}\right]+\frac{\gamma}{ln\left(2\right)}$$

$${S}_{d}=\frac{d{\pi}^{d/2}}{\Gamma (\frac{d}{2}+1)}$$

Another choice could be an estimator of the kernel type, see [13]. The so called resubstitution estimate proposed in [13] is of the form
where ${\widehat{f}}_{ker}$ is a kernel density estimate.

$${\widehat{H}}_{ker}=-\frac{1}{N}\sum _{i=1}^{N}log{\widehat{f}}_{ker}\left({X}_{i}\right)$$

**Figure 2.**95% confidence intervals of the estimated MI of a bivariate gaussian distribution as the sample size N increases. True value of MI = 1.1979 bits. Entropy is estimated by the $k-$ nearest neighbor given in Equation (38) (black) and the kernel type entropy estimator given in Equation (39) (green).

The performances of our estimator for the two different choices of entropy estimators are compared in Figure 2. Again, the samples are drawn from a bivariate gaussian distribution with standard marginal distributions and correlation coefficient $\rho =0.9$ for different sample sizes N. The best performance is definitely obtained with the $k-$ nearest neighbor entropy estimator (black).

Hence we come to the conclusion that the best results are obtained using a kernel type density estimator at the step 1 of the method and with a k–nearest neighbor entropy estimator at the step 3 of the method. This is the chosen procedure for the examples proposed in the next Section.

## 4. Examples and Simulation Results

In this Section we show the results obtained with the estimator proposed in Section 3. The examples cover the cases $d=2,3,4$. We choose cases for which the MI can be computed either in a closed form expression or numerically, so that we can easily compare our results with the exact ones.

Several methods have been already proposed in the literature to estimate the MI between multivariate random vectors from sample data. One of the most efficient seems to be the binless method exposed in [8], based on estimates from k–nearest neighbor statistics. We refer from now on to this method as “KSG” from the names of the authors. On the other hand, a straightforward way to estimate MI could be to estimate separately the entropy terms in Equation (8) by means of a suitable estimator as the one proposed in [10,32] and sum them up. Let us notice that this procedure is not recommended as the errors made in the individual estimates of each entropy term would presumably not cancel. We refer to this methodology as the “plain entropy” method. The results obtained with our estimator are here compared both to the KSG and to the “plain entropy”.

As mutual information is a particular case of Kullback–Leibler divergence, we could use the estimators proposed in [15,17]. However let us remark that our estimator is based on one single sample of size N drawn from ${f}_{1,\cdots ,d}$, the multivariate joint law of the random vector $({X}_{1},\cdots ,{X}_{d})$. On the other side, the estimators proposed in [15,17] and adapted to the case of mutual information are based on two samples: one drawn from the multivariate joint law and the other from ${f}_{{\alpha}^{1}}\cdots {f}_{{\alpha}^{n}}$, the product of the n marginal laws. We could build the missing sample from the given one, but it is not obvious that the estimators will keep their good properties. Indeed in our setting the samples are no more independent as some resampling procedure should be devised to make the method suitable for our particular case. From a preliminary numerical study it seems that the performances of the estimators proposed in [15,17] are not comparable to the performances of our estimator. Hence we skip this comparison.

The examples illustrated in the next two Subsections are chosen to test our approach on distributions that could be troublesome for our approach. Following the discussion introduced in the previous Section, the delicate point of the method is the estimation of the (conditional) cumulative distribution functions in Equation (20). Hence we consider an example where the density to be estimated has bounded support as for the Uniform r.v. (Example 2) and an example where the density is positive valued and with an early steep peak as for the Exponential r.v. (Example 3). To address the problem of tails in estimation of information–theoretic quantities, we present two examples of heavy-tailed distributions: the lognormal (Example 4) and the Levy (Example 5). Note that the lognormal distribution is heavy tailed but with finite all order moments, while the Levy is an alpha–stable distribution with power law tail probability and no finite moments.

We rely on simulated data, while we reserve to a future work the application of the estimator to data coming from real experiments. For each example we perform $m=100$ simulation batches of the random vectors involved for each of the following sample sizes $N=100,500,1000,2000,3000$.

#### 4.1. Two-Dimensional Vectors

Let us consider the following five cases of two-dimensional random vectors $X=({X}_{1},{X}_{2})$.

**Example 1.**Let $X=({X}_{1},{X}_{2})$ be a gaussian random vector with standard normal components and correlation coefficient $\rho =0.9$. Denoting as σ the covariance matrix of X, the MI between ${X}_{1}$ and ${X}_{2}$ is given as$$MI({X}_{1},{X}_{2})=-\frac{1}{2}{log}_{2}\left[det\left(\sigma \right)\right]=1.1980\phantom{\rule{0.277778em}{0ex}}bit$$

The numerical results are illustrated in Figure 3. The horizontal line indicates the exact value $MI=1.1980$. Both plots reveal that our estimator of the MI is centered around its true value. The interquartile range considerably decreases with increasing sample size, becoming minimal for $n=3000$. The comparison of the performances with the KSG and “plain entropy” estimators shows that our method performs comparably with KSG except for very small sample sizes ($n=100$), where it has a larger dispersion, see Figure 3—right panel—bottom right inset. It results even better for the highest sample sizes. However its variance quickly decreases to the values attained by the KSG estimator that are sensibly smaller that the variances of the “plain entropy” method. Actually, the “plain entropy” estimator leads to poorer performances such as larger interquartile range and less marked centering of the confidence interval around the true value.

**Figure 3.**(Example 1) Left panel: Boxplots of the estimated MI grouped according to different sample sizes. Right panel: mean error as a function of $1/$(Sample Size). Insets to the right panel: 95% confidence interval for the estimated MI (bottom left) and standard deviations (bottom right) versus Sample Size. Color map: black and white for the estimator we propose in Section 3.2, red for KSG and blue for “plain entropy”.

**Example 2.**Let ${X}_{1}$ and ${X}_{2}$ be linearly dependent random variables, related through the following equation$${X}_{2}=5{X}_{1}+{X}_{3}$$

The numerical results are illustrated in Figure 4. Similar observations as for the previous example can be deduced for this case. The dispersion is clearly highly reduced as the sample size increases, but the estimated MI is always centered with respect to the true value. The “plain entropy” method is again less performing with respect to the others both in terms of dispersion and of centering the true value. The standard deviations of the estimates obtained with our method appear always lower than the ones obtained with the “plain entropy” method and comparable with those obtained by KSG method except for very small sample sizes $N=100$.

**Figure 4.**(Example 2) Left panel: Boxplots of the estimated MI grouped according to different sample sizes. Right panel: mean error as a function of $1/$(Sample Size). Insets to the right panel: 95% confidence interval for the estimated MI (bottom left) and standard deviations (bottom right) versus Sample Size. Color map: black and white for the estimator we propose in Section 3.2, red for KSG and blue for “plain entropy”.

**Example 3.**Let ${X}_{1}$,${X}_{2}$ have joint c.d.f. given in the following equation$${F}_{1,2}({x}_{1},{x}_{2})=\left\{\begin{array}{c}\frac{({x}_{1}+1)({e}_{2}^{x}-1)}{{x}_{1}+2{e}_{2}^{x}-1}\phantom{\rule{0.277778em}{0ex}}({x}_{1},{x}_{2})\in [-1,1]\times [0,\infty ]\\ 1-{e}^{-{x}_{2}}\phantom{\rule{0.277778em}{0ex}}({x}_{1},{x}_{2})\in (1,\infty ]\times [0,\infty ]\end{array}\right.$$$$C({u}_{1},{u}_{2})=\frac{{u}_{1}{u}_{2}}{{u}_{1}+{u}_{2}-{u}_{1}{u}_{2}}$$$$c({u}_{1},{u}_{2})=\frac{2{u}_{1}{u}_{2}}{{({u}_{1}+{u}_{2}-{u}_{1}{u}_{2})}^{3}}$$

Both the boxplots and the confidence intervals in Figure 5 show that the values of MI obtained with our estimator are centered around the true value even for the smallest sample sizes $n=100$ and $n=500$. The comparison with KSG and “plain entropy” methods in Figure 5—right panel shows that the method proposed allows to obtain results that are comparable with the first one apart only for $n=100$ and is always better performing than the second one. This last method produces not centered estimates for small sample sizes such as $n=100$ and $n=500$. The standard deviations behave as in the previous example.

**Figure 5.**(Example 3) Left panel: Boxplots of the estimated MI grouped according to different sample sizes. Right panel: mean error as a function of $1/$(Sample Size). Insets to the right panel: 95% confidence interval for the estimated MI (bottom left) and standard deviations (bottom right) versus Sample Size. Color map: black and white for the estimator we propose in Section 3.2, red for KSG and blue for “plain entropy”.

**Example 4.**Let $({X}_{1},{X}_{2})$ be a bivariate lognormal random vector with zero mean and covariance matrix given by$$Cov({X}_{1},{X}_{2})=\frac{1-{e}^{-2\beta T}}{2\beta}\Sigma =\frac{1-{e}^{-2\beta T}}{2\beta}\left(\begin{array}{cc}1& 0.9\\ 0.9& 1\end{array}\right)$$

The results are illustrated in Figure 6. The methods are applied to samples drawn from the joint distribution of $({X}_{1},{X}_{2})$ with no pre–processing. Our approach gives good results both in terms of centered confidence intervals and standard deviations. Hence the method is robust to this kind of heavy–tailed distributions. The KSG estimator gives bad results for smaller sample sizes ($N=100,500,1000$) as the corresponding confidence intervals do not include the true value of MI. The “plain entropy” method seems to behave perfectly in this case.

**Figure 6.**(Example 4) Left panel: Boxplots of the estimated MI grouped according to different sample sizes. Right panel: mean error as a function of $1/$(Sample Size). Insets to the right panel: 95% confidence interval for the estimated MI (bottom left) and standard deviations (bottom right) versus Sample Size. Color map: black and white for the estimator we propose in Section 3.2, red for KSG and blue for “plain entropy”.

**Example 5.**Let $({X}_{1},{X}_{2})$ be a random vector with Levy distributed margins, i.e., with p.d.f. given by$$f(x,\mu ,c)=\sqrt{\frac{c}{2\pi}}\frac{{e}^{-\frac{c}{2(x-\mu )}}}{{(x-\mu )}^{3/2}}$$

All the three methods fail in estimating MI in this case. The results are totally unreliable. The errors are very large even for larger sample sizes. Concerning our method the problem relies in the estimation of the density of the margins when the tail has strongly power law behavior. In this very ill–served example, the tails are so heavy that it is frequent to have values very far from those corresponding to the largest part of the probability mass and the kernel estimate is poor as it would need too many points. The kernel method fails and so the estimation of MI.

However we propose here to exploit the property of MI of being invariant under reparametrization of the margins. This trick allows to get rid of the heavy tail problems at least in some cases. For example, when the random variables are positive valued, we could apply a sufficiently good transformation of the values in order to improve the estimates. Such a transformation should be strictly monotone and differentiable and able to lighten and shorten the tails of the distribution, as the logarithm function. In Figure 7 we illustrate the results obtained on samples drawn from the couple $({X}_{1},{X}_{2})$ with Levy margins and pre–processed applying a log–transformation to the values. Such a procedure will not affect the estimated value, as it can be seen in the figure. All the three methods now give correctly estimated values.

**Figure 7.**(Example 5) Left panel: Boxplots of the estimated MI grouped according to different sample sizes. Right panel: mean error as a function of $1/$(Sample Size). Insets to the right panel: 95% confidence interval for the estimated MI (bottom left) and standard deviations (bottom right) versus Sample Size. Color map: black and white for the estimator we propose in Section 3.2, red for KSG and blue for “plain entropy”.

Hence the method is shown to be only partly robust to heavy–tailed distributions, as KSG and “plain entropy”. In particular it succeeds when applied to distributions with tail behavior as in the log–normal case and fails on strong power law tailed distributions as the Levy one. However, in the latter case, our proposal is to take advantage of the invariance property of MI and to pre–process data in order to reduce the importance of the tails. This comment holds good for any dimension $d>2$, as the examples shown in the next section.

#### 4.2. Three-Dimensional Vectors

We tested our estimator also in the case $d=3$, considering the multi–indices given in Equation (31). Let $X=({X}_{1},{X}_{2},{X}_{3})$ be a gaussian random vector with standard normal components and pair correlation coefficients ${\rho}_{{X}_{1},{X}_{2}}={\rho}_{{X}_{2},{X}_{3}}={\rho}_{{X}_{1},{X}_{3}}=0.9$. The exact MI can be evaluated by means of the following equation
where ${\sigma}_{Y}$ denotes the covariance matrix of the random vector Y. In the specific case we get $MI=1.3812$ bit.

$$MI=\frac{1}{2}{log}_{2}\left(\frac{det\left({\sigma}_{({X}_{2},{X}_{3})}\right)}{det\left({\sigma}_{X}\right)}\right)$$

In Figure 8—left panel we show the boxplots of the estimated MI values as the sample size increases. From both plots the unbiasedness of the estimator even for small sample sizes can be clearly detected. Our estimator always leads to better results than the others both for unbiasedness and for the dispersion, cf. Figure 8—right panel. Both the KSG and the “plain entropy” estimator do not center the exact value of the MI for the largest sample sizes ($n=2000$ and $n=3000$), even though the standard deviations are comparable with the ones obtained with our method.

**Figure 8.**(Gaussian $d=3$) Left panel: Boxplots of the estimated MI grouped according to different sample sizes. Right panel: mean error as a function of $1/$(Sample Size). Insets to the right panel: 95% confidence interval for the estimated MI (bottom left) and standard deviations (bottom right) versus Sample Size. Color map: black and white for the estimator we propose in Section 3.2, red for KSG and blue for “plain entropy”.

#### 4.3. Four-Dimensional Vectors

For the case $d=4$ we considered a multivariate Gaussian random vector with mean and covariance matrix given as
We chose the following multi–indices to group the components: ${\alpha}^{1}=(1,2)$ and ${\alpha}^{2}=(3,4)$. The results are illustrated in Figure 9. As already noticed for the three dimensional Gaussian case, the KSG and “plain entropy” estimators seems biased while the estimator here presented centers the true value for all the explored sample sizes.

$$\begin{array}{c}\hfill \mu =(0,0,0,0)\phantom{\rule{1.em}{0ex}}\phantom{\rule{1.em}{0ex}}\Sigma =\left(\begin{array}{cccc}1& 0.9& 0.9& 0.9\\ 0.9& 1& 0.9& 0.9\\ 0.9& 0.9& 1& 0.9\\ 0.9& 0.9& 0.9& 1\end{array}\right)\end{array}$$

**Figure 9.**(Gaussian $d=4$) Left panel: Boxplots of the estimated MI grouped according to different sample sizes. Right panel: mean error as a function of $1/$(Sample Size). Insets to the right panel: 95% confidence interval for the estimated MI (bottom left) and standard deviations (bottom right) versus Sample Size. Color map: black and white for the estimator we propose in Section 3.2, red for KSG and blue for “plain entropy”.

## 5. Conclusions

In this paper, we have presented a new estimator for the mutual information between subsets of components of d-dimensional random vectors. It exploits the link between the mutual information and the entropy of the linkage function here proved. Hence the problem of estimating mutual information is reduced to the computation of the entropy of a suitably transformed sample of uniformly distributed random variables on the interval $[0,1]$, that can be easily performed by the k–nearest neighbor technique illustrated in [10].

The method gives very good performances in terms both of unbiasedness and variance of the estimates. For the 2–dimensional case, the results are comparable to the KSG and “plain entropy” estimates. However, for higher dimensions (we tested on examples up to dimension 4), our method is preferable as it keeps being centered while KSG and “plain entropy” both show a bias. All the tested estimators are shown to be robust for mild heavy tailed distributions, such as the lognormal distribution, but they fail on distributions with power law tail probabilities and no finite moments, such as the Levy distribution. However, we suggest to overcome the problem using the invariance property of MI under reparametrization of the margins and hence pre–processing the data with a suitable transformation of the univariate components before estimation.

The fact that the estimator gives unbiased results for small sample sizes and larger dimensions allows a wide use, also in applications where the availability of big data sets from real experiments is extremely rare.

## Acknowledgments

The authors would like to thank the two referees and Prof. Ilya Nemenman for their useful comments that helped improving this paper. The authors are grateful to András Horváth for his precious support concerning some numerical issue. RS is grateful to Michael I. Jordan for a short but valuable comment on KL–divergence estimators. Work partially supported by “AMALFI” project (ORTO119W8J) and by “Stochastic Processes and Applications 2012” project.

## Conflicts of Interest

The authors declare no conflict of interest.

## References

- Cover, T.M.; Thomas, J.A. Elements of Information Theory, 2nd ed.; Wiley-Interscience (John Wiley & Sons): Hoboken, NJ, USA, 2006. [Google Scholar]
- Ghahramani, Z.; Póczos, B.; Schneider, J.G. Copula-Based Kernel Dependency Measures. In Proceedings of the 29th International Conference on Machine Learning, (ICML-12), Edinburgh, UK, 26 June–1 July 2012; pp. 775–782.
- McGill, W.J. Multivariate information transmission. Psychometrika
**1954**, 19, 97–116. [Google Scholar] [CrossRef] - Schneidman, E.; Still, S.; Berry, M.J.; Bialek, W. Network information and connected correlations. Phys. Rev. Lett.
**2003**, 91, 238701. [Google Scholar] [CrossRef] [PubMed] - Margolin, A.A.; Wang, K.; Califano, A.; Nemenman, I. Multivariate dependence and genetic networks inference. IET Syst. Biol.
**2010**, 4, 428–440. [Google Scholar] [CrossRef] [PubMed] - Darbellay, G.A. An estimator of the mutual information based on a criterion for independence. Comput. Stat. Data Anal.
**1999**, 32, 1–17. [Google Scholar] [CrossRef] - Darbellay, G.A.; Vajda, I. Estimation of the information by an adaptive partitioning of the observation space. IEEE Trans. Inform. Theory
**1999**, 45, 1315–1321. [Google Scholar] [CrossRef] - Kraskov, A.; Stögbauer, H.; Grassberger, P. Estimating mutual information. Phys. Rev. E
**2004**, 69, 066138. [Google Scholar] [CrossRef] [PubMed] - Evans, D. A computationally efficient estimator for mutual information. Proc. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci.
**2008**, 464, 1203–1215. [Google Scholar] [CrossRef] - Kozachenko, L.F.; Leonenko, N.N. A statistical estimate for the entropy of a random vector. Probl. Peredachi Inform.
**1987**, 23, 9–16. [Google Scholar] - Leonenko, N.; Pronzato, L.; Savani, V. A class of Rényi information estimators for multidimensional densities. Ann. Stat.
**2008**, 36, 2153–2182. [Google Scholar] [CrossRef] - Pál, D.; Póczos, B.; Szepesvári, C. Estimation of Rényi Entropy and Mutual Information Based On Generalized Nearest-Neighbor Graphs.
- Beirlant, J.; Dudewicz, E.J.; Györfi, L.; van der Meulen, E.C. Nonparametric entropy estimation: An overview. Int. J. Math. Stat. Sci.
**1997**, 6, 17–40. [Google Scholar] - Nemenman, I. Coincidences and estimation of entropies of random variables with large cardinalities. Entropy
**2011**, 13, 2013–2023. [Google Scholar] [CrossRef] - Nguyen, X.; Wainwright, M.J.; Jordan, M.I. Estimating Divergence Functionals and the Likelihood Ratio by Penalized Convex Risk Minimization. In Advances in Neural Information Processing Systems; MIT Press: Cambridge, MA, USA, 2007; pp. 1089–1096. [Google Scholar]
- Sugiyama, M.; Nakajima, S.; Kashima, H.; Buenau, P.V.; Kawanabe, M. Direct Importance Estimation with Model Selection and Its Application to Covariate Shift Adaptation. In Advances in Neural Information Processing Systems; MIT Press: Cambridge, MA, USA, 2007; pp. 1433–1440. [Google Scholar]
- Nguyen, X.; Wainwright, M.J.; Jordan, M.I. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Trans. Inf. Theory
**2010**, 56, 5847–5861. [Google Scholar] [CrossRef] - Kostal, L.; Pokora, O. Nonparametric estimation of information-based measures of statistical dispersion. Entropy
**2012**, 14, 1221–1233. [Google Scholar] [CrossRef] - Li, H.; Scarsini, M.; Shaked, M. Linkages: A tool for the construction of multivariate distributions with given nonoverlapping multivariate marginals. J. Multivar. Anal.
**1996**, 56, 20–41. [Google Scholar] [CrossRef] - Nelsen, R.B. An Introduction to Copulas, Lecture Notes in Statistics; Springer-Verlag: New York, NY, USA, 1999; Volume 139. [Google Scholar]
- Genest, C.; Quesada Molina, J.J.; Rodríguez Lallena, J.A. De l’impossibilité de construire des lois à marges multidimensionnelles données à partir de copules. C. R. Acad. Sci. Paris Sér. I Math.
**1995**, 320, 723–726. [Google Scholar] - Jenison, R.L.; Reale, R.A. The shape of neural dependence. Neural Comput.
**2004**, 16, 665–672. [Google Scholar] [CrossRef] [PubMed] - Blumentritt, T.; Schmid, F. Mutual information as a measure of multivariate association: Analytical properties and statistical estimation. J. Stat. Comput. Simul.
**2012**, 82, 1257–1274. [Google Scholar] [CrossRef] - Margolin, A.A.; Nemenman, I.; Basso, K.; Wiggins, C.; Stolovitzky, G.; Favera, R.D.; Califano, A. ARACNE: An algorithm for the reconstruction of gene regulatory networks in a mammalian cellular context. BMC Bioinf.
**2006**, 7, S7. [Google Scholar] [CrossRef] [PubMed] - Reiss, R.D. Nonparametric estimation of smooth distribution functions. Scand. J. Statist.
**1981**, 8, 116–119. [Google Scholar] - Wand, M.P.; Jones, M.C. Kernel Smoothing; Chapman and Hall Ltd.: London, UK, 1995; Volume 60 Monographs on Statistics and Applied Probability. [Google Scholar]
- Silverman, B.W. Density Estimation for Statistics and Data Analysis; CRC Press: London, UK, 1986; Volume 26. [Google Scholar]
- Bishop, C.M. Neural Networks for Pattern Recognition; Oxford University Press: New York, NY, USA, 1995. [Google Scholar]
- Hyndman, R.J.; Bashtannyk, D.M.; Grunwald, G.K. Estimating and visualizing conditional densities. J. Comput. Graph. Statist.
**1996**, 5, 315–336. [Google Scholar] - Fan, J.; Yao, Q.; Tong, H. Estimation of conditional densities and sensitivity measures in nonlinear dynamical systems. Biometrika
**1996**, 83, 189–206. [Google Scholar] [CrossRef] - Hyndman, R.J.; Yao, Q. Nonparametric estimation and symmetry tests for conditional density functions. J. Nonparametr. Stat.
**2002**, 14, 259–278. [Google Scholar] [CrossRef] - Victor, J.D. Binless strategies for estimation of information from neural data. Phys. Rev. E
**2002**, 66, 051903. [Google Scholar] [CrossRef] [PubMed] - Marek, T.; Tichavsky, P. On the estimation of mutual information. In Proceedings of ROBUST 2008, Honolulu, USA, 2–5 November 2008; Antoch, J., Dohnal, G., Eds.; 2008. [Google Scholar]
- Platen, E.; Bruti-Liberati, N. Numerical Solution of Stochastic Differential Equations with Jumps in Finance; Springer: Berlin, Germany, 2010; Volume 64. [Google Scholar]

© 2013 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).